Cooperative storage of shared files in a parallel computing system with dynamic block size
View Patent ↗Improved techniques are provided for parallel writing of data to a shared object in a parallel computing system. A method is provided for storing data generated by a plurality of parallel processes to a shared object in a parallel computing system. The method is performed by at least one of the processes and comprises: dynamically determining a block size for storing the data; exchanging a determined amount of the data with at least one additional process to achieve a block of the data having the dynamically determined block size; and writing the block of the data having the dynamically determined block size to a file system. The determined block size comprises, e.g., a total amount of the data to be stored divided by the number of parallel processes. The file system comprises, for example, a log structured virtual parallel file system, such as a Parallel Log-Structured File System (PLFS).
1. A method performed at a compute node for storing data generated by a plurality of parallel processes to a shared object in a parallel computing system, wherein said method is performed by at least a first one of said parallel processes that generate said shared object, wherein said compute node hosts at least said first one of said parallel processes and wherein said method comprises the steps of:
dynamically determining a write block size for storing said data;
exchanging a determined amount of said data with at least a second one of said parallel processes to achieve a block of said data having said dynamically determined write block size, wherein said exchanging comprises said first one of said parallel processes one or more of (i) sending said determined amount of said data to, and (ii) receiving said determined amount of said data from, said at least second one of said parallel processes to achieve said block of said data having said dynamically determined write block size at said compute node hosting said first one of said parallel processes; and
writing said block of said data having said dynamically determined block size to a file system for storage.
2. The method of claim 1 , wherein said determined write block size comprises a total amount of said data to be stored by said plurality of parallel processes divided by a number of said plurality of parallel processes.
3. The method of claim 1 , further comprising the step of broadcasting a number of bytes to be written by each of said plurality of parallel processes.
4. The method of claim 1 , wherein said exchanging step comprises an inter-process message exchange.
5. The method of claim 1 , wherein said file system comprises a log structured virtual parallel file system.
6. The method of claim 1 , wherein said file system comprises a Parallel Log-Structured File System (PLFS).
7. A compute node apparatus for storing data generated by a plurality of parallel processes to a shared object in a parallel computing system, wherein said apparatus hosts at least a first one of said parallel processes, wherein said apparatus comprises:
a memory; and
at least one hardware device operatively coupled to the memory and configured to:
dynamically determine a write block size for storing said data;
exchange a determined amount of said data with at least a second one of said parallel processes to achieve a block of said data having said dynamically determined write block size, wherein said exchange comprises said first one of said parallel processes one or more of (i) sending said determined amount of said data to, and (ii) receiving said determined amount of said data from, said at least second one of said parallel processes to achieve said block of said data having said dynamically determined write block size at said compute node hosting said first one of said parallel processes; and
write said block of said data having said dynamically determined block size to a file system for storage.
8. The compute node apparatus of claim 7 , wherein said determined write block size comprises a total amount of said data to be stored by said plurality of parallel processes divided by a number of said plurality of parallel processes.
9. The compute node apparatus of claim 7 , further comprising broadcasting a number of bytes to be written by each of said plurality of parallel processes.
10. The compute node apparatus of claim 7 , wherein said exchange comprises an inter-process message exchange.
11. The compute node apparatus of claim 7 , wherein said file system comprises a log structured virtual parallel file system.
12. The compute node apparatus of claim 7 , wherein said file system comprises a Parallel Log-Structured File System (PLFS).
13. A data storage system of a compute node for storing data generated by a plurality of parallel processes to a shared object in a parallel computing system, wherein said compute node hosts at least a first one of said parallel processes, wherein said data storage system comprises:
a processing unit that dynamically determines a write block size for storing said data and for exchanging a determined amount of said data with at least a second one of said parallel processes to achieve a block of said data having said dynamically determined write block size, wherein said exchange comprises said first one of said parallel processes one or more of (i) sending said determined amount of said data to, and (ii) receiving said determined amount of said data from, said at least second one of said parallel processes to achieve said block of said data having said dynamically determined write block size at said compute node hosting said first one of said parallel processes; and
a storage medium that stores said block of said data having said dynamically determined write block size prior to writing said block of said data to a file system.
14. The data storage system of claim 13 , wherein said determined write block size comprises a total amount of said data to be stored by said plurality of parallel processes divided by a number of said plurality of parallel processes.
15. The data storage system of claim 13 , wherein said processing unit is further configured to broadcast a number of bytes to be written by each of said plurality of parallel processes.
16. The data storage system of claim 13 , wherein said exchange comprises an inter-process message exchange.
17. The data storage system of claim 13 , wherein said file system comprises a log structured virtual parallel file system.
18. The data storage system of claim 13 , wherein said file system comprises a Parallel Log-Structured File System (PLFS).
19. The method of claim 1 , wherein said step of exchanging a determined amount of said data with at least a second one of said parallel processes is performed using an interconnect of said parallel computing system.
20. The compute node apparatus of claim 7 , wherein said step of exchanging a determined amount of said data with at least a second one of said parallel processes is performed using an interconnect of said parallel computing system.