Small file aggregation in a parallel computing system
View Patent ↗Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.
1. A method for storing a plurality of files generated by a single write process in a parallel computing system, comprising the steps of:
aggregating said plurality of files into a single aggregated file, wherein said plurality of files are generated by said single write process;
generating metadata for said single aggregated file, wherein said metadata comprises an offset and a length of each of said plurality of files in said single aggregated file; and
providing said single aggregated file to a file system for storage as a single file with said metadata, wherein said single aggregated file is stored in one level of a file directory comprised of a data file for each of said plurality of files.
2. The method of claim 1 , further comprising a plurality of said write processes running on a plurality of compute nodes, and wherein a single aggregated file is generated for each of said write processes.
3. The method of claim 1 , wherein said file system is a middleware virtual file system.
4. The method of claim 1 , wherein said file system is a parallel file system comprised of one or more disks.
5. The method of claim 1 , further comprising the step of unpacking one or more of said files from said single aggregated file using said metadata.
6. The method of claim 1 , wherein said metadata characterizes a layout of said files within said single aggregated file.
7. The method of claim 1 , wherein a plurality of said files that are in a same directory are aggregated into a single file aggregated container structure.
8. The method of claim 1 , further comprising the step of maintaining a name list file, said data file, and an index file for each of said files in said single aggregated file.
9. The method of claim 1 , wherein said single aggregated file comprises a number of bytes for said metadata plus data for said plurality of files.
10. The method of claim 1 , wherein said metadata further comprises one or more of a timestamp and an identifier of each small file.
11. A computer program product comprising a non-transitory processor-readable recordable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by a processor of a processing device implement the steps of the method of claim 1 .
12. An apparatus for storing a plurality of files generated by a single write process in a parallel computing system, comprising:
a memory; and
at least one hardware device operatively coupled to the memory and configured to:
aggregate said plurality of files into a single aggregated file, wherein said plurality of files are generated by said single write process;
generate metadata for said single aggregated file, wherein said metadata comprises an offset and a length of each of said plurality of files in said single aggregated file; and
provide said single aggregated file to a file system for storage as a single file with said metadata, wherein said single aggregated file is stored in one level of a file directory comprised of a data file for each of said plurality of files.
13. The apparatus of claim 12 , further comprising a plurality of said write processes running on a plurality of compute nodes, and wherein a single aggregated file is generated for each of said write processes.
14. The apparatus of claim 12 , wherein said file system is a middleware virtual file system.
15. The apparatus of claim 12 , wherein said file system is a parallel file system comprised of one or more disks.
16. The apparatus of claim 12 , wherein said at least one hardware device is further configured to unpack one or more of said files from said single aggregated file using said metadata.
17. The apparatus of claim 12 , wherein said at least one hardware device is further configured to maintain a name list file, said data file, and an index file for each of said files in said single aggregated file.
18. A data storage system for storing a plurality of files generated by a single write process plurality of processes in a parallel computing system, comprising:
a hardware processing unit for aggregating said plurality of files into a single aggregated file, wherein said plurality of files are generated by said single write process, and generating metadata for said single aggregated file, wherein said metadata comprises an offset and a length of each of said plurality of files in said single aggregated file; and
a storage medium for storing said single aggregated file as a single file with said metadata, wherein said single aggregated file is stored in one level of a file directory comprised of a data file for each of said plurality of files.
19. The data storage system of claim 18 , wherein said plurality of files are provided to a middleware virtual file system for storage.
20. The data storage system of claim 19 , wherein said middleware virtual file system comprises one or more of a parallel log file system and a burst buffer.
21. The data storage system of claim 18 , wherein said single aggregated file is stored on a parallel file system comprised of one or more disks.
22. The data storage system of claim 18 , wherein said at least one hardware device is further configured to maintain a name list file, said data file, and an index file for each of said files in said single aggregated file.