Parallel file system with metadata distributed across partitioned key-value store

1. An apparatus comprising:

a compute node of a parallel file system;

the compute node being configured to communicate with a plurality of object storage servers and with a plurality of other compute nodes over a network, wherein a plurality of applications executing on said plurality of compute nodes generate a shared file;

the compute node further configured to implement a Parallel Log Structured File System (PLFS) library to store at least one portion of said shared file generated by an application executing on said compute node and PLFS metadata for said at least one portion of said shared file on one or more of said plurality of object storage servers, wherein said PLFS metadata for said at least one portion of said shared file comprises a key-value pair having a key based on a logical offset of said at least one portion of said shared file;

the compute node further configured to implement a partitioned key-value data store for storing a partition of said PLFS metadata for said shared file, wherein said partitioned key-value data store communicates with partitioned key-value data stores on said other compute nodes using a message passing interface to cooperatively store the PLFS metadata, wherein said partition comprises a given sub-range of a key space for said PLFS metadata, wherein said key-value pair for said PLFS metadata for said at least one portion of said shared file is inserted into a corresponding one of said partitions stored by a respective partitioned key-value data store as a new key-value record based on said key when said at least one portion is written to said shared file; and

the compute node being implemented utilizing at least one processing device coupled to a memory.

2. The apparatus of claim 1 wherein said PLFS library communicates with said partitioned key-value data store using function calls.

3. The apparatus of claim 1 wherein said PLFS metadata for said shared file is sharded across said partitioned key-value data store.

4. The apparatus of claim 1 wherein each of the object storage servers is configured to interface with at least one object storage target.

5. The apparatus of claim 1 wherein the partitioned key-value data store is configured to process one or more metadata requests when one or more of said plurality of applications executing on said compute nodes performs a read operation on said shared file.

6. The apparatus of claim 1 wherein the partitioned key-value data store is implemented using Multidimensional Data Hashing Indexing Middleware (MDHIM).

7. The apparatus of claim 6 wherein one or more MDHIM clients run on each of said compute nodes and communicate with a plurality of MDHIM servers in said parallel file system.

8. The apparatus of claim 7 wherein each of said plurality of MDHIM servers store a given partition of said PLFS metadata for said shared file comprising said sub-range of said key space for said PLFS metadata for said shared file.

9. The apparatus of claim 8 wherein a given MDHIM server storing a particular sub-range of said PLFS metadata is contacted to read or write PLFS metadata within said sub-range.

10. The apparatus of claim 1 wherein the parallel file system comprises a Lustre file system.

11. A processing platform comprising a plurality of processing devices and the apparatus of claim 1 .

12. A method comprising:

configuring a compute node of a parallel file system to communicate with a plurality of object storage servers and with a plurality of other compute nodes over a network, wherein a plurality of applications executing on said plurality of compute nodes generate a shared file; and

configuring the compute node to implement a Parallel Log Structured File System (PLFS) library to store at least one portion of said shared file generated by an application executing on said compute node and PLFS metadata for said at least one portion of said shared file on one or more of said plurality of object storage servers, wherein said PLFS metadata for said at least one portion of said shared file comprises a key-value pair having a key based on a logical offset of said at least one portion of said shared file; and

configuring the compute node to implement a partitioned key-value data store for storing a partition of said PLFS metadata for said shared file, wherein said partitioned key-value data store communicates with partitioned key-value data stores on said other compute nodes using a message passing interface to cooperatively store the PLFS metadata, wherein said partition comprises a given sub-range of a key space for said PLFS metadata, wherein said key-value pair for said PLFS metadata for said at least one portion of said shared file is inserted into a corresponding one of said partitions stored by a respective partitioned key-value data store as a new key-value record based on said key when said at least one portion is written to said shared file;

the compute node being implemented utilizing at least one processing device coupled to a memory.

13. The method of claim 12 wherein said PLFS library communicates with said partitioned key-value data store using function calls.

14. The method of claim 12 wherein said PLFS metadata for said shared file is sharded across said partitioned key-value data store.

15. The method of claim 12 further comprising the step of configuring each of the object storage servers to interface with at least one object storage target.

16. The method of claim 12 further comprising the step of configuring the partitioned key-value data store to process one or more metadata requests when one or more of said plurality of applications executing on said compute nodes performs a read operation on said shared file.

17. The method of claim 12 wherein the partitioned key-value data store is implemented using Multidimensional Data Hashing Indexing Middleware (MDHIM).

18. The method of claim 17 wherein one or more MDHIM clients run on each of said compute nodes and communicate with a plurality of MDHIM servers in said parallel file system.

19. The method of claim 18 wherein each of said plurality of MDHIM servers store a given partition of said PLFS metadata for said shared file comprising said sub-range of said key space for said PLFS metadata for said shared file.

20. The method of claim 19 wherein a given MDHIM server storing a particular sub-range of said PLFS metadata is contacted to read or write PLFS metadata within said sub-range.

21. A computer program product comprising a processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed perform the steps of the method of claim 12 .

22. A parallel file system comprising:

a plurality of compute nodes; and

a plurality of object storage servers;

one of the plurality of compute nodes being configured to communicate with said plurality of object storage servers and with others of said plurality of compute nodes over a network, wherein a plurality of applications executing on said plurality of compute nodes generate a shared file;

the one compute node further configured to implement a Parallel Log Structured File System (PLFS) library to store at least one portion of said shared file generated by an application executing on said one compute node and PLFS metadata for said at least one portion of said shared file on one or more of said plurality of object storage servers, wherein said PLFS metadata for said at least one portion of said shared file comprises a key-value pair having a key based on a logical offset of said at least one portion of said shared file;

the one compute node further configured to implement a partitioned key-value data store for storing a partition of said PLFS metadata for said shared file, wherein said partitioned key-value data store communicates with partitioned key-value data stores on said other compute nodes using a message passing interface to cooperatively store the PLFS metadata, wherein said partition comprises a given sub-range of a key space for said PLFS metadata, wherein said key-value pair for said PLFS metadata for said at least one portion of said shared file is inserted into a corresponding one of said partitions stored by a respective partitioned key-value data store as a new key-value record based on said key when said at least one portion is written to said shared file; and

the one compute node being implemented utilizing at least one processing device coupled to a memory.

23. The parallel file system of claim 22 wherein the partitioned key-value data store is implemented using Multidimensional Data Hashing Indexing Middleware (MDHIM).

24. The parallel file system of claim 23 wherein one or more MDHIM clients run on each of said compute nodes and communicate with a plurality of MDHIM servers in said parallel file system, wherein each of said plurality of MDHIM servers store a given partition of said PLFS metadata for said shared file comprising said sub-range of said key space for said PLFS metadata for said shared file.

25. The parallel file system of claim 24 wherein a given MDHIM server storing a particular sub-range of said PLFS metadata is contacted to read or write PLFS metadata within said sub-range.