Cloud object store for archive storage of high performance computing data using decoupling middleware
View Patent ↗Cloud object storage is enabled for archived data, such as checkpoints and results, of high performance computing applications using a middleware process. A plurality of archived files, such as checkpoint files and results, generated by a plurality of processes in a parallel computing system are stored by obtaining the plurality of archived files from the parallel computing system; converting the plurality of archived files to objects using a log structured file system middleware process; and providing the objects for storage in a cloud object storage system. The plurality of processes may run, for example, on a plurality of compute nodes. The log structured file system middleware process may be embodied, for example, as a Parallel Log-Structured File System (PLFS). The log structured file system middleware process optionally executes on a burst buffer node.
1. A method for storing a plurality of archived files generated by a plurality of processes in a parallel computing system, comprising the steps of:
obtaining said plurality of archived files from said parallel computing system;
converting said plurality of archived files to objects using a log structured file system middleware process, wherein said log structured file system middleware process writes modifications sequentially in a log-like structure; and
providing said objects for storage in a cloud object storage system using a software interface module of said log structured file system middleware process, wherein said software interface module communicates with a cloud storage application programming interface (API) of said cloud object storage system, wherein said log structured file system middleware process is a middleware process between said parallel computing system and said cloud object storage system.
2. The method of claim 1 , wherein said plurality of processes are running on a plurality of compute nodes.
3. The method of claim 1 , wherein said plurality of archived files comprise one or more of checkpoint files and results generated by said plurality of compute nodes.
4. The method of claim 1 , wherein said log structured file system middleware process comprises a Parallel Log-Structured File System (PLFS).
5. The method of claim 1 , wherein said log structured file system middleware process executes on a burst buffer node.
6. The method of claim 1 , wherein said step of providing said objects for storage in said cloud object storage system further comprises the step of providing said objects to said cloud storage of said cloud object storage system.
7. The method of claim 6 , wherein said log structured file system middleware process comprises said software module for providing said objects for storage to said cloud storage API.
8. The method of claim 7 , wherein said software module embeds one or more directory paths into a name of each of said objects provided to said cloud storage API.
9. The method of claim 7 , wherein said log structured file system middleware process comprises a software module for generating a namespace of said plurality of files.
10. The method of claim 1 , wherein said step of converting said plurality of archived files to objects further comprises converting said plurality of archived files to a plurality of Get-Put operations on a set of said objects.
11. The method of claim 1 , wherein said log structured file system middleware process processes partial file input/output operations by buffering entire objects in one or more buffers and applying one or more of partial read and write commands using said one or more buffers.
12. A tangible machine-readable recordable storage medium for storing a plurality of archived files generated by a plurality of processes in a parallel computing system, wherein one or more software programs when executed by one or more processing devices implement the steps of the method of claim 1 .
13. An apparatus for storing a plurality of archived files generated by a plurality of processes in a parallel computing system, comprising:
a memory; and
at least one hardware device operatively coupled to the memory and configured to:
obtain said plurality of archived files;
convert said plurality of archived files to objects using a log structured file system middleware process, wherein said log structured file system middleware process writes modifications sequentially in a log-like structure; and
provide said objects for storage in a cloud object storage system using a software interface module of said log structured file system middleware process, wherein said software interface module communicates with a cloud storage application programming interface (API) of said cloud object storage system, wherein said log structured file system middleware process is a middleware process between said parallel computing system and said cloud object storage system.
14. The apparatus of claim 13 , wherein said plurality of processes are running on a plurality of compute nodes.
15. The apparatus of claim 13 , wherein said plurality of archived files comprise one or more of checkpoint files and results generated by said plurality of compute nodes.
16. The apparatus of claim 13 , wherein said log structured file system middleware process comprises a Parallel Log-Structured File System (PLFS).
17. The apparatus of claim 13 , wherein said log structured file system middleware process executes on a burst buffer node.
18. The apparatus of claim 13 , wherein said objects are provided for storage in said cloud object storage system by providing said objects to said cloud storage API of said cloud object storage system.
19. The apparatus of claim 13 , wherein said plurality of archived files are converted to objects by converting said plurality of files to a plurality of Get-Put operations on a set of said objects.
20. The apparatus of claim 13 , wherein said log structured file system middleware process processes partial file input/output operations by buffering entire objects in one or more buffers and applying one or more of partial read and write commands using said one or more buffers.
21. A data storage system for storing a plurality of archived files generated by a plurality of processes in a parallel computing system, comprising:
a processing unit for obtaining said plurality of archived files; converting said plurality of archived files to objects using a log structured file system middleware process, wherein said log structured file system middleware process writes modifications sequentially in a log-like structure; and providing said objects for storage in a cloud object storage system using a software interface module of said log structured file system middleware process, wherein said software interface module communicates with a cloud storage application programming interface (API) of said cloud object storage system, wherein said log structured file system middleware process is a middleware process between said parallel computing system and said cloud object storage system; and
said cloud object storage system for storing said objects.
22. The data storage system of claim 21 , wherein said plurality of processes are running on a plurality of compute nodes.
23. The data storage system of claim 21 , wherein said plurality of archived files comprise one or more checkpoint files and results generated by said plurality of compute nodes.
24. The data storage system of claim 21 , wherein said log structured file system middleware process comprises a Parallel Log-Structured File System (PLFS).
25. The data storage system of claim 21 , wherein said log structured file system middleware process executes on a burst buffer node.