Advanced metadata management

1. A computer-executable method for managing metadata, using one or more processors and memory, in a distributed data storage system, wherein the distributed data storage system includes multiple burst buffers collectively providing a key-value store enabled to store metadata, the computer-executable method comprising:

receiving, from a compute node, metadata related to data stored within the distributed data storage system;

indexing the metadata from the compute node at a first one of the multiple burst buffers;

splitting the metadata from the compute node into two or more key spaces;

distributing the split metadata of the two or more key spaces to respective ones of two or more servers associated with respective other ones of the multiple burst buffers;

wherein the two or more servers comprise two or more Multidimensional Data Hashing Indexing Middleware (MDHIM) servers associated with the respective other ones of the multiple burst buffers;

receiving an aggregated request for a range of the split metadata;

sending the aggregated request to a particular server of the two or more servers having the range of the split metadata;

wherein the metadata from the compute node includes one or more transaction identifiers (TIDs);

wherein a user request for the split metadata includes the one or more TIDs combined with one or more keys; and

wherein the one or more keys are indexed according to one or more key-value parings of the one or more keys and the one or more TIDs; and

converting the user request including the one or more TIDs and the one or more keys to an MDHIM command, wherein the aggregated request is based at least in part on the MDHIM command;

wherein a plurality of input/output dispatchers are associated with the respective other ones of the multiple burst buffers; and

wherein a given one of the plurality of input/output dispatchers is associated with a first burst buffer node and is configured:

to convert the user request to the MDHIM command;

to communicate with at least one input/output dispatcher of the plurality of input/output dispatchers associated with a second burst buffer node different from the first burst buffer node; and

to dynamically reallocate storage of the split metadata between MDHIM servers associated with the first and second burst buffer nodes based on the communication.

2. The computer-executable method of claim 1 , further comprising:

receiving a metadata request from the compute node to purge metadata; and

processing the metadata request.

3. The computer-executable method of claim 2 , wherein the processing of the metadata request comprises:

analyzing a metadata index to determine which portions of the metadata to purge; and

purging the determined portions of the metadata.

4. The computer-executable method of claim 1 , wherein the sending of the aggregated request is performed by one or more MDHIM clients with knowledge of which of the two or more servers has the range of the split metadata.

5. The computer-executable method of claim 1 , wherein the aggregated request comprises a batch request including a range of keys.

6. The computer-executable method of claim 5 , wherein the keys in the range are indexed with corresponding split metadata in a partitioned key-value store.

7. The computer-executable method of claim 1 , wherein a first one of the MDHIM servers associated with the respective other ones of the multiple burst buffers stores metadata related to objects, a second one of the MDHIM servers associated with the respective other ones of the multiple burst buffers stores metadata related to containers and a third one of the MDHIM servers associated with the respective other ones of the multiple burst buffers stores metadata related to checkpoints and transactions.

8. A system, comprising:

a compute node;

a distributed data storage system in communication with the compute node, wherein the distributed data storage system includes multiple burst buffers collectively providing a key-value store enabled to store metadata; and

computer-executable program logic encoded in memory of one or more computers in communication with the distributed data storage system to enable management of metadata, using one or more processors and memory, wherein the computer-executable program logic is configured for the execution of:

receiving, from the compute node, metadata related to data stored within the distributed data storage system;

indexing the metadata from the compute node at a first one of the multiple burst buffers;

splitting the metadata from the compute node into two or more key spaces;

distributing the split metadata of the two or more key spaces to respective ones of two or more servers associated with respective other ones of the multiple burst buffers;

wherein the two or more servers comprise two or more Multidimensional Data Hashing Indexing Middleware (MDHIM) servers associated with the respective other ones of the multiple burst buffers;

receiving an aggregated request for a range of the split metadata;

sending the aggregated request to a particular server of the two or more servers having the range of the split metadata;

wherein the metadata from the compute node includes one or more transaction identifiers (TIDs);

wherein a user request for the split metadata includes the one or more TIDs combined with one or more keys; and

wherein the one or more keys are indexed according to one or more key-value parings of the one or more keys and the one or more TIDs; and

converting the user request including the one or more TIDs and the one or more keys to an MDHIM command, wherein the aggregated request is based at least in part on the MDHIM command;

wherein a plurality of input/output dispatchers are associated with the respective other ones of the multiple burst buffers; and

wherein a given one of the plurality of input/output dispatchers is associated with a first burst buffer node and is configured:

to convert the user request to the MDHIM command;

to communicate with at least one input/output dispatcher of the plurality of input/output dispatchers associated with a second burst buffer node different from the first burst buffer node; and

to dynamically reallocate storage of the split metadata between MDHIM servers associated with the first and second burst buffer nodes based on the communication.

9. The system of claim 8 , wherein the computer-executable program logic is further configured for the execution of:

receiving a metadata request from the compute node to purge metadata; and

processing the metadata request.

10. The system of claim 9 , wherein the processing of the metadata request comprises:

analyzing a metadata index to determine which portions of the metadata to purge; and

purging the determined portions of the metadata.

11. The system of claim 8 , wherein the sending of the aggregated request is performed by one or more MDHIM clients with knowledge of which of the two or more servers has the range of the split metadata.

12. The system of claim 8 , wherein the aggregated request comprises a batch request including a range of keys.

13. The system of claim 12 , wherein the keys in the range are indexed with corresponding split metadata in a partitioned key-value store.

14. The system of claim 8 , wherein a first one of the MDHIM servers associated with the respective other ones of the multiple burst buffers stores metadata related to objects, a second one of the MDHIM servers associated with the respective other ones of the multiple burst buffers stores metadata related to containers and a third one of the MDHIM servers associated with the respective other ones of the multiple burst buffers stores metadata related to checkpoints and transactions.

15. A computer program product for managing metadata, using one or more processors and memory, in a distributed data storage system, wherein the distributed data storage system includes multiple burst buffers collectively providing a key-value store enabled to store metadata, the computer program product comprising:

a non-transitory computer readable medium encoded with computer-executable program code configured to enable the execution of:

receiving, from a compute node, metadata related to data stored within the distributed data storage system;

indexing the metadata from the compute node at a first one of the multiple burst buffers;

splitting the metadata from the compute node into two or more key spaces;

distributing the split metadata of the two or more key spaces to respective ones of two or more servers associated with respective other ones of the multiple burst buffers;

wherein the two or more servers comprise two or more Multidimensional Data Hashing Indexing Middleware (MDHIM) servers associated with the respective other ones of the multiple burst buffers;

receiving an aggregated request for a range of the split metadata;

sending the aggregated request to a particular server of the two or more servers having the range of the split metadata;

wherein the metadata from the compute node includes one or more transaction identifiers (TIDs);

wherein a user request for the split metadata includes the one or more TIDs combined with one or more keys; and

wherein the one or more keys are indexed according to one or more key-value parings of the one or more keys and the one or more TIDs; and

converting the user request including the one or more TIDs and the one or more keys to an MDHIM command, wherein the aggregated request is based at least in part on the MDHIM command;

wherein a plurality of input/output dispatchers are associated with the respective other ones of the multiple burst buffers; and

wherein a given one of the plurality of input/output dispatchers is associated with a first burst buffer node and is configured:

to convert the user request to the MDHIM command;

to communicate with at least one input/output dispatcher of the plurality of input/output dispatchers associated with a second burst buffer node different from the first burst buffer node; and

to dynamically reallocate storage of the split metadata between MDHIM servers associated with the first and second burst buffer nodes based on the communication.

16. The computer program product of claim 15 , wherein the code is further configured to enable the execution of:

receiving a metadata request from the compute node to purge metadata; and

processing the metadata request.

17. The computer program product of claim 16 , wherein the processing of the metadata request comprises:

analyzing a metadata index to determine which portions of the metadata to purge; and

purging the determined portions of the metadata.

18. The computer program product of claim 15 , wherein the aggregated request comprises a batch request including a range of keys.

19. The computer program product of claim 18 , wherein the range of keys are indexed with corresponding split metadata in a partitioned key-value store.

20. The computer program product of claim 15 , wherein a first one of the MDHIM servers associated with the respective other ones of the multiple burst buffers stores metadata related to objects, a second one of the MDHIM servers associated with the respective other ones of the multiple burst buffers stores metadata related to containers and a third one of the MDHIM servers associated with the respective other ones of the multiple burst buffers stores metadata related to checkpoints and transactions.