Distributed metadata servers in a file system with separate metadata servers for file metadata and directory metadata

1. A distributed metadata storage system, comprising:

a plurality of compute nodes, wherein at least one of said compute nodes comprises a file system client that generates one or more of a directory metadata request and a file metadata request;

a plurality of directory metadata servers to serve only directory metadata in response to a given directory metadata request;

a plurality of file metadata servers to serve only file metadata in response to a given file metadata request, wherein said plurality of directory metadata servers are distinct from said plurality of file metadata servers, and wherein said plurality of directory metadata servers and said plurality of file metadata servers comprise at least one processing device and are connected to said plurality of compute nodes using at least one network; and

at least one shared persistent key-value metadata store for storing said directory metadata and said file metadata, wherein said at least one shared persistent key-value metadata store is accessed by said plurality of directory metadata servers and said plurality of file metadata servers.

2. The distributed metadata storage system of claim 1 , wherein at least one of said compute nodes comprises an object client that requests object data from an object store system.

3. The distributed metadata storage system of claim 1 , wherein said plurality, N, of directory metadata servers and said plurality, M, of file metadata servers are determined independently to balance for different workloads.

4. The distributed metadata storage system of claim 1 , wherein said plurality, N, of directory metadata servers and said plurality, M, of file metadata servers are dynamically readjusted for evolving workloads.

5. The distributed metadata storage system of claim 1 , wherein each of said plurality of directory metadata servers and said plurality of file metadata servers comprises an abstract storage interface comprising a software interface module that communicates with said at least one shared persistent key-value metadata store.

6. The distributed metadata storage system of claim 5 , wherein said abstract storage interface provides a storage-specific interface for said at least one shared persistent metadata store.

7. The distributed metadata storage system of claim 1 , wherein a directory traversal evaluates only directory metadata in one or more of said plurality of directory metadata servers, without evaluating file metadata.

8. The distributed metadata storage system of claim 1 , wherein a load is dynamically redistributed among said plurality of directory metadata servers and said plurality of file metadata servers by modifying an operation that is used to select one of said plurality of directory metadata servers and said plurality of file metadata servers.

9. A directory metadata server in a distributed metadata storage environment, comprising:

at least one processing device configured to obtain a directory metadata request from at least one file system client on at least one compute node, wherein said directory metadata server is dedicated to processing only directory metadata request types and wherein a plurality of file metadata servers in said distributed metadata storage environment are dedicated to processing only file metadata request types; and

at least one processing device configured to process said directory metadata request using at least one shared persistent key-value metadata store accessed by said directory metadata server and said plurality of file metadata servers.

10. The directory metadata server of claim 9 , wherein said at least one compute node comprises an object client that requests object data from an object store system.

11. The directory metadata server of claim 9 , wherein said directory metadata server is one of a plurality, N, of directory metadata servers and wherein said distributed metadata storage environment further comprises a plurality, M, of said file metadata servers, and wherein said N and M values are determined independently to balance for different workloads.

12. The directory metadata server of claim 9 , wherein said directory metadata server is one of a plurality, N, of directory metadata servers and wherein said distributed metadata storage environment further comprises a plurality, M, of said file metadata servers, and wherein said N and M values are dynamically readjusted for evolving workloads.

13. The directory metadata server of claim 9 , wherein said directory metadata server is one of a plurality, N, of directory metadata servers and wherein said distributed metadata storage environment further comprises a plurality, M, of said file metadata servers, and wherein each of said plurality of directory metadata servers and said plurality of file metadata servers comprises an abstract storage interface comprising a software interface module that communicates with said at least one shared persistent key-value metadata store.

14. The directory metadata server of claim 9 , wherein a directory traversal evaluates only directory metadata in one or more of a plurality of said directory metadata servers, without evaluating file metadata.

15. A method performed by a directory metadata server in a distributed metadata storage environment, comprising the steps of:

obtaining, using at least one processing device, a directory metadata request from at least one file system client on at least one compute node, wherein said directory metadata server is dedicated to processing only directory metadata request types and wherein a plurality of file metadata servers in said distributed metadata storage environment are dedicated to processing only file metadata request types; and

processing said directory metadata request using at least one shared persistent key-value metadata store accessed by said directory metadata server and said plurality of file metadata servers.

16. The method of claim 15 , wherein said at least one compute node comprises an object client that requests object data from an object store system.

17. The method of claim 15 , wherein said directory metadata server is one of a plurality, N, of directory metadata servers and wherein said distributed metadata storage environment further comprises a plurality, M, of said file metadata servers, and wherein said N and M values are determined independently to balance for different workloads.

18. The method of claim 15 , wherein said directory metadata server is one of a plurality, N, of directory metadata servers and wherein said distributed metadata storage environment further comprises a plurality, M, of said file metadata servers, and wherein said N and M values are dynamically readjusted for evolving workloads.

19. The method of claim 15 , wherein said directory metadata server is one of a plurality, N, of directory metadata servers and wherein said distributed metadata storage environment further comprises a plurality, M, of said file metadata servers, and wherein each of said plurality of directory metadata servers and said plurality of file metadata servers comprises an abstract storage interface comprising a software interface module that communicates with said at least one shared persistent key-value metadata store.

20. The method of claim 15 , wherein a directory traversal evaluates only directory metadata in one or more of a plurality of said directory metadata servers, without evaluating file metadata.

21. A non-transitory machine-readable recordable storage medium, wherein one or more software programs when executed by one or more processing devices implement the steps of the method of claim 15 .