Parallel checksumming of data chunks of a shared data object using a log-structured file system
View Patent ↗Checksum values are generated and used to verify the data integrity. A client executing in a parallel computing system stores a data chunk to a shared data object on a storage node in the parallel computing system. The client determines a checksum value for the data chunk; and provides the checksum value with the data chunk to the storage node that stores the shared object. The data chunk can be stored on the storage node with the corresponding checksum value as part of the shared object. The storage node may be part of a Parallel Log-Structured File System (PLFS), and the client may comprise, for example, a Log-Structured File System client on a compute node or burst buffer. The checksum value can be evaluated when the data chunk is read from the storage node to verify the integrity of the data that is read.
1. A method performed by a Log-Structured File System client executing on one or more of a compute node and a burst buffer node in a parallel computing system for storing a data chunk generated by said parallel computing system to a shared data object on a storage node in said parallel computing system, comprising:
processing said data chunk generated by said parallel computing system using said Log-Structured File System client, wherein said Log-Structured File System client executes on one or more of said compute node and said burst buffer node in said parallel computing system;
determining, by said Log-Structured File System client, a checksum value for said data chunk as said data chunk is written by said Log-Structured File System client to said storage node, wherein a plurality of other Log-Structured File System clients determine a plurality of checksums for a corresponding plurality of other data chunks of said shared data object in parallel; and
providing, by said Log-Structured File System client, said checksum value with said data chunk to said storage node for storage as part of said shared object, wherein said storage node stores said data chunk with said corresponding checksum value as part of said shared object with said plurality of checksums for said corresponding plurality of other data chunks of said shared data object, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.
2. The method of claim 1 , wherein said storage node comprises a Log-Structured File server.
3. The method of claim 1 , wherein said storage node is part of a log structured virtual parallel file system.
4. The method of claim 1 , wherein said storage node is part of a Parallel Log-Structured File System (PLFS).
5. The method of claim 1 , further comprising the step of evaluating said checksum value when said data chunk is read from said storage node.
6. The method of claim 5 , further comprising the step of recomputing said checksum value when said data chunk is read from said storage node and comparing said recomputed checksum value to said stored checksum value.
7. The method of claim 6 , further comprising the step of providing a verified data chunk if said corresponding checksum values match.
8. The method of claim 6 , further comprising the step of correcting said data chunk if said corresponding checksum values do not match.
9. A tangible machine-readable recordable storage medium for storing a data chunk generated by a parallel computing system to a shared data object on a storage node in said parallel computing system, wherein one or more software programs when executed by one or more processing devices implement the steps of the method of claim 1 .
10. A compute node apparatus in a parallel computing system for executing a Log-Structured File System client that stores a data chunk generated by said parallel computing system to a shared data object on a storage node in said parallel computing system, said compute node apparatus comprising:
a memory; and
at least one hardware device operatively coupled to the memory and configured to:
process said data chunk generated by said parallel computing system using said Log-Structured File System client, wherein said Log-Structured File System client executes on one or more of said compute node and said burst buffer node in said parallel computing system;
determine, by said Log-Structured File System client, a checksum value for said data chunk as said data chunk is written by said Log-Structured File System client to said storage node, wherein a plurality of other Log-Structured File System clients determine a plurality of checksums for a corresponding plurality of other data chunks of said shared data object in parallel; and
provide, by said Log-Structured File System client, said checksum value with said data chunk to said storage node for storage as part of said shared object, wherein said storage node stores said data chunk with said corresponding checksum value as part of said shared object with said plurality of checksums for said corresponding plurality of other data chunks of said shared data object, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.
11. The apparatus of claim 10 , wherein said storage node comprises a Log-Structured File server.
12. The apparatus of claim 10 , wherein said apparatus comprises one or more of a compute node and a burst buffer node.
13. The apparatus of claim 10 , wherein said at least one hardware device is further configured to evaluate said checksum value when said data chunk is read from said storage node.
14. The apparatus of claim 13 , wherein said at least one hardware device is further configured to recompute said checksum value when said data chunk is read from said storage node and comparing said recomputed checksum value to said stored checksum value.
15. The apparatus of claim 14 , wherein said at least one hardware device is further configured to provide a verified data chunk if said corresponding checksum values match.
16. The apparatus of claim 14 , wherein said at least one hardware device is further configured to correct said data chunk if said corresponding checksum values do not match.
17. A method performed by a Log-Structured File System server node in a parallel computing system for storing a data chunk as a shared object, comprising:
receiving, by said Log-Structured File System server, said data chunk with a corresponding checksum value from a Log-Structured File System client on a compute node in said parallel computing system, wherein said data chunk is generated by said parallel computing system using said Log-Structured File System client, and wherein said corresponding checksum value is generated by said Log-Structured File System client as said data chunk is written by said Log-Structured File System Client, wherein a plurality of other Log-Structured File System Clients determine a plurality of checksums for a corresponding plurality of other data chunks of said shared data object in parallel; and
storing, by said Log-Structured File System server, said received data chunk and said corresponding checksum value to a shared data object on said storage node, wherein said storage node stores said data chunk with said corresponding checksum value as part of said shared object with said plurality of checksums for said corresponding plurality of other data chunks of said shared data object, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.
18. A tangible machine-readable recordable storage medium for storing a data chunk as a shared object in a parallel computing system, wherein one or more software programs when executed by one or more processing devices implement the steps of the method of claim 17 .
19. A Log-Structured File System server apparatus in a parallel computing system for storing a data chunk as a shared object, said storage node apparatus comprising:
a memory; and
at least one hardware device operatively coupled to the memory and configured to:
receive, by said Log-Structured File System server, said data chunk with a corresponding checksum value from a Log-Structured File System client on a compute node in said parallel computing system, wherein said data chunk is generated by said parallel computing system using Log-Structured File System client, and wherein said corresponding checksum value is generated by said Log-Structured File System client as said data chunk is written by said Log-Structured File System Client, wherein a plurality of other log-structured file system clients determine a plurality of checksums for a corresponding plurality of other data chunks of said shared data object in parallel; and
store, by said Log-Structured File System server, said received data chunk and said corresponding checksum value to a shared data object on said storage node, wherein said storage node stores said data chunk with said corresponding checksum value as part of said shared object with said plurality of checksums for said corresponding plurality of other data chunks of said shared data object, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.
20. The method of claim 17 , further comprising the step of providing said data chunk and said corresponding checksum value to a Log-Structured File System client when said data chunk is read.