Distributed metadata in a high performance computing environment

1. A computer-executable method of managing meta-data for a high performance computing (HPC) environment in a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store and one or more data storage arrays, the computer-executable method comprising:

receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value;

determining which of the one or more burst buffers stores the requested metadata;

upon determination that a first burst buffer of the one or more burst buffers stores the requested metadata, locating the key-value in a portion of the distributed key-value store accessible from the first burst buffer;

communicating between each of the one or more burst buffers to manage the distributed key-value store;

moving data between the one or more burst buffers and the one or more data storage arrays, in the distributed storage system, based on a frequency of use of data;

wherein the determination enables querying for the metadata in a portion of a total amount of metadata on the distributed key-value store;

determining a state of the distributed key-value store; and

based on the state, re-balancing the distributed key-value store.

2. The computer-executable method of claim 1 , wherein the distributed key-value store is a Multidimensional Data Hashing Indexing Middleware (MDHIM).

3. The computer-executable method of claim 1 , wherein the determining comprises:

accessing which of the one or more burst buffers stores a range of key-values, wherein the range includes the key-value.

4. A system, comprising:

a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store and one or more data storage arrays; and

computer-executable program logic encoded in memory of one or more computers in communication with the distributed storage system to enable management of metadata for a high performance computing (HPC) environment in the distributed storage system, wherein the computer-executable program logic is configured for the execution of:

receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value;

determining which of the one or more burst buffers stores the requested metadata;

upon determination that a first burst buffer of the one or more burst buffers stores the requested metadata, locating the key-value in a portion of the distributed key-value store accessible from the first burst buffer;

communicating between each of the one or more burst buffers to manage the distributed key-value store;

moving data between the one or more burst buffers and the one or more data storage arrays, in the distributed storage system, based on a frequency of use of data;

wherein the determination enables querying for the metadata in a portion of a total amount of metadata on the distributed key-value store;

determining a state of the distributed key-value store; and

based on the state, re-balancing the distributed key-value store.

5. The system of claim 4 , wherein the distributed key-value store is a Multidimensional Data Hashing Indexing Middleware (MDHIM).

6. The system of claim 4 , wherein the determining comprises:

accessing which of the one or more burst buffers stores a range of key-values, wherein the range includes the key-value.

7. A computer program product for managing meta-data for a high performance computing (HPC) environment in a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store and one or more data storage arrays, the computer program product comprising:

a non-transitory computer readable medium encoded with computer-executable program code for using read signatures in replication, the code configured to enable the execution of:

receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value;

determining which of the one or more burst buffers stores the requested metadata;

upon determination that a first burst buffer of the one or more burst buffers stores the requested metadata, locating the key-value in a portion of the distributed key-value store accessible from the first burst buffer;

communicating between each of the one or more burst buffers to manage the distributed key-value store;

moving data between the one or more burst buffers and the one or more data storage arrays, in the distributed storage system, based on a frequency of use of data;

wherein the determination enables querying for the metadata in a portion of a total amount of metadata on the distributed key-value store;

determining a state of the distributed key-value store; and

based on the state, re-balancing the distributed key-value store.

8. The computer program product of claim 7 , wherein the distributed key-value store is a Multidimensional Data Hashing Indexing Middleware (MDHIM).

9. The computer program product of claim 7 , wherein the determining comprises:

accessing which of the one or more burst buffers stores a range of key-values, wherein the range includes the key-value.