Parallel compression of data chunks of a shared data object using a log-structured file system
View Patent ↗Techniques are provided for parallel compression of data chunks being written to a shared object. A client executing on a compute node or a burst buffer node in a parallel computing system stores a data chunk generated by the parallel computing system to a shared data object on a storage node by compressing the data chunk; and providing the data compressed data chunk to the storage node that stores the shared object. The client and storage node may employ Log-Structured File techniques. The compressed data chunk can be de-compressed by the client when the data chunk is read. A storage node stores a data chunk as part of a shared object by receiving a compressed version of the data chunk from a compute node; and storing the compressed version of the data chunk to the shared data object on the storage node.
1. A method performed by a log-structured file system client executing on one or more of a compute node and a burst buffer node in a parallel computing system for storing a data chunk generated by said parallel computing system to a shared data object on a storage node in said parallel computing system, comprising:
processing said data chunk generated by said parallel computing system using said log structured file system client executing on at least one processing device, wherein said log structured file system client executes on one or more of said compute node and said burst buffer node in said parallel computing system;
compressing, by said log-structured file system client executing on at least one processing device, said data chunk as said data chunk is written by said log-structured file system client to said storage node, wherein a plurality of other log structured file system clients compress a corresponding plurality of other data chunks of said shared data object in parallel to generate a plurality of compressed other data chunks; and
providing, by said log-structured file system client executing on at least one processing device, said compressed data chunk to said storage node for storage as part of said shared object, wherein said storage node stores said compressed data chunk as part of said shared object with said plurality of compressed other data chunks, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.
2. The method of claim 1 , wherein said storage node comprises a Log-Structured File server.
3. The method of claim 2 , wherein said Log-Structured File server stores said compressed version of said data chunk to said shared data object on said compute node.
4. The method of claim 1 , further comprising the step of de-compressing said compressed data chunk when said data chunk is read from said storage node.
5. The method of claim 4 , further comprising the step of providing said de-compressed data chunk to an application.
6. A non-transitory machine-readable recordable storage medium for storing a data chunk generated by a parallel computing system to a shared data object on a storage node in said parallel computing system, wherein one or more software programs when executed by one or more processing devices implement the steps of the method of claim 1 .
7. A compute node apparatus in a parallel computing system for executing a log-structured file system client that stores a data chunk generated by said parallel computing system to a shared data object on a storage node in said parallel computing system, said compute node apparatus comprising:
a memory; and
at least one hardware device operatively coupled to the memory and configured to:
process said data chunk generated by said parallel computing system using said log structured file system client executing on at least one processing device, wherein said log structured file system client executes on one or more of said compute node and said burst buffer node in said parallel computing system;
compress, by said log-structured file system client executing on at least one processing device, said data chunk as said data chunk is written by said log-structured file system client to said storage node, wherein a plurality of other log structured file system clients compress a corresponding plurality of other data chunks of said shared data object in parallel to generate a plurality of compressed other data chunks; and
provide, by said log-structured file system client executing on at least one processing device, said compressed data chunk to said storage node for storage as part of said shared object, wherein said storage node stores said compressed data chunk as part of said shared object with said plurality of compressed other data chunks, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.
8. The compute node apparatus of claim 7 , wherein said storage node comprises a Log-Structured File server.
9. The compute node apparatus of claim 8 , wherein said Log-Structured File server stores said compressed version of said data chunk to said shared data object on said compute node.
10. The compute node apparatus of claim 7 , wherein said at least one hardware device is further configured to de-compress said compressed data chunk when said data chunk is read from said storage node.
11. The compute node apparatus of claim 10 , wherein said at least one hardware device is further configured to provide said de-compressed data chunk to an application.
12. The compute node apparatus of claim 7 , wherein said apparatus comprises one or more of a compute node and a burst buffer node.
13. A method performed by a storage node comprising a Log-Structured File server in a parallel computing system for storing a data chunk as part of a shared object, comprising:
receiving, by said Log-Structured File server executing on at least one processing device, a compressed version of said data chunk compressed by a log-structured file system client executing on at least one processing device of a compute node in said parallel computing system;
receiving, by said Log-Structured File server executing on at least one processing device, a compressed version of a plurality of other data chunks of said shared data object compressed by a plurality of other log-structured file system clients executing on at least one processing device of a plurality of other compute nodes in said parallel computing system; and
storing, by said Log-Structured File server executing on at least one processing device, said compressed version of said data chunk to said shared data object on said storage node with said plurality of compressed versions of said other data chunks of said shared data object from said plurality of other log-structured file system clients, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.
14. The method of claim 13 , further comprising the step of providing said compressed data chunk to a compute node when said data chunk is read from said storage node.
15. A non-transitory machine-readable recordable storage medium for storing a data chunk as a shared object in a parallel computing system, wherein one or more software programs when executed by one or more processing devices implement the steps of the method of claim 13 .
16. The method of claim 13 , wherein said compressed version is generated by a Log-Structured File System client.
17. The method of claim 13 , further comprising the steps of decompressing said compressed version and providing the decompressed data chunk to the compute node.
18. A storage node apparatus comprising a Log-Structured File server in a parallel computing system for storing a data chunk as part of a shared object, said storage node apparatus comprising:
a memory; and
at least one hardware device operatively coupled to the memory and configured to:
receive, by said Log-Structured File server executing on at least one processing device, a compressed version of said data chunk compressed by a log-structured file system client executing on at least one processing device of a compute node in said parallel computing system;
receive, by said Log-Structured File server executing on at least one processing device, a compressed version of a plurality of other data chunks of said shared data object compressed by a plurality of other log-structured file system clients executing on at least one processing device of a plurality of other compute nodes in said parallel computing systems; and
store, by said Log-Structured File server executing on at least one processing device, said compressed version of said data chunk to said shared data object on said storage node with said plurality of compressed versions of said other data chunks of said shared data object from said plurality of other log-structured file system clients, and wherein said shared object is shared by a plurality of distributed processes in said parallel computing system.
19. The storage node apparatus of claim 18 , wherein said at least one hardware device is further configured to provide said compressed data chunk to a compute node when said data chunk is read from said storage node.
20. The storage node apparatus of claim 18 , wherein said compressed version is generated by a Log-Structured File System client.