System and method for discovery and retrieval using an enumerator
A data storage system includes storage and a global enumerator. The storage stores data chunks, object level metadata associated with portions of the data chunks, and chunk level metadata associated with respective data chunks. The global enumerator obtains a data access request that includes a metadata characteristic; in response to obtaining the data access request: matches the metadata characteristic to at least one of the data chunks based on at least one selected from a group consisting of the object level metadata and the chunk level metadata; and provides a response based on the at least one data chunk of the data chunks in response to the data access request.
1. A data storage system, comprising:
storage for storing:
data chunks,
object level metadata associated with portions of the data chunks, and
chunk level metadata associated with respective data chunks; and
a global enumerator programmed to:
obtain a data access request comprising a metadata characteristic;
in response to obtaining the data access request:
match the metadata characteristic to at least one of the data chunks based on at least one selected from a group consisting of the object level metadata and the chunk level metadata; and
provide a response, based on the at least one data chunk of the data chunks, in response to the data access request; and
a plurality of regional enumerators associated with respective regions that store respective portions of the data chunks,
wherein matching the metadata characteristic to the at least one of the data chunks comprises:
identifying a portion of the regions in which respective portions of the data chunks are stored that include the at least one of the data chunks;
generating a plurality of sub-requests, based on the data access request, for respective regions of the portion of the regions;
distributing the sub-requests to a portion of the plurality of the regional enumerators based on the portion of the regions; and
aggregating responses to the sub-requests from the portion of the plurality of the regional enumerators.
2. The data storage system of claim 1 , wherein the data chunks comprise data integrity coded data.
3. The data storage system of claim 2 , wherein the data comprises system data from a client for which the data storage system provides data storage services.
4. The data storage system of claim 1 , wherein the storage comprises:
a first plurality of storage devices disposed in a first region;
a second plurality of storage devices disposed in a second region;
wherein the first region is geographically separated from the second region.
5. The data storage system of claim 4 , wherein a first portion of the first plurality of storage devices are housed in a first cluster and a second portion of the first plurality of storage devices are housed in a second cluster.
6. The data storage system of claim 1 , wherein the object level metadata comprises:
object identifiers associated with data of respective portions of the data chunks.
7. The data storage system of claim 6 , wherein the object level metadata is based on system metadata associated with data associated with the data chunks.
8. The data storage system of claim 1 , wherein the object level metadata comprises:
data integrity scheme identifiers associated with data integrity schemes used to encode data to obtain the data chunks.
9. The data storage system of claim 1 , wherein the chunk level metadata comprises:
data integrity scheme identifiers associated with respective data chunks, wherein the data integrity scheme identifiers are associated data integrity schemes used to encode data to obtain the respective data chunks.
10. The data storage system of claim 1 , wherein the chunk level metadata comprises:
storage device identifiers, associated with the respective data chunks, of devices in which the respective data chunks are stored.
11. The data storage system of claim 1 , wherein the chunk level metadata comprises:
classification identifiers, associated with respective data chunks, that specify a characteristic of the respective data chunks.
12. The data storage system of claim 1 , wherein matching the metadata characteristic to the at least one of the data chunks is performed without crawling the data chunks.
13. A method for managing a data storage system, comprising:
obtaining a data access request comprising a metadata characteristic;
in response to obtaining the data access request:
matching the metadata characteristic to at least one data chunk of data chunks managed by the data storage system based on at least one selected from a group consisting of object level metadata and chunk level metadata; and
providing a response, based on the at least one data chunk of the data chunks, in response to the data access request,
wherein matching the metadata characteristic to the at least one of the data chunks comprises:
identifying a portion of regions in which respective portions of the data chunks are stored that include the at least one of the data chunks;
generating a plurality of sub-requests, based on the data access request, for respective regions of the portion of the regions;
distributing the sub-requests to a portion of a plurality of regional enumerators based on the portion of the regions; and
aggregating responses to the sub-requests from the portion of the plurality of the regional enumerators.
14. The method of claim 13 , wherein the data chunks comprise data integrity coded data.
15. The method of claim 14 , wherein the data comprises system data from a client for which the data storage system provides data storage services.
16. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing a data storage system, the method comprising:
obtaining a data access request comprising a metadata characteristic;
in response to obtaining the data access request:
matching the metadata characteristic to at least one data chunk of data chunks managed by the data storage system based on at least one selected from a group consisting of object level metadata and chunk level metadata; and
providing a response, based on the at least one data chunk of the data chunks, in response to the data access request,
wherein matching the metadata characteristic to the at least one of the data chunks comprises:
identifying a portion of regions in which respective portions of the data chunks are stored that include the at least one of the data chunks;
generating a plurality of sub-requests, based on the data access request, for respective regions of the portion of the regions;
distributing the sub-requests to a portion of a plurality of regional enumerators based on the portion of the regions; and
aggregating responses to the sub-requests from the portion of the plurality of the regional enumerators.
17. The non-transitory computer readable medium of claim 16 , wherein the data chunks comprise data integrity coded data.