Asynchronous distributed de-duplication for replicated content addressable storage clusters
A method is performed by a device of a group of devices in a distributed data replication system. The method includes storing an index of objects in the distributed data replication system, the index being replicated while the objects are stored locally by the plurality of devices in the distributed data replication system. The method also includes conducting a scan of at least a portion of the index and identifying a redundant replica(s) of the at least one of the objects based on the scan of the index. The method further includes de-duplicating the redundant replica(s), and updating the index to reflect the status of the redundant replica.
1 . A computer-implemented method in a distributed data storage system comprising a plurality of storage clusters, the method comprising:
receiving, by a first storage cluster, a request for an object;
identifying, by the first storage cluster, locations of the object stored by a storage system by querying a global index stored in the first storage cluster, the locations including at least one first location and at least one replica location;
determining, by the first storage cluster, which of the identified locations to use to read the object based on one or more criteria;
retrieving the object from the determined location; and
transmitting the retrieved object to a requesting client,
wherein the global index is replicated in the plurality of storage clusters.
2 . The method of claim 1 , wherein the one of the one or more criteria is network resources.
3 . The method of claim 2 , wherein the network resources comprise bandwidth consumption.
4 . The method of claim 2 , wherein determining which of the locations to use to read the object minimizes the network resources.
5 . The method of claim 1 , wherein the one of the one or more criteria is geographic location of the first location and the least one replica location.
6 . The method of claim 5 , wherein determining which of the locations to use to read the object comprises selecting a closest geographic location of the stored object to a geographic location of a client requesting the object.
7 . The method of claim 1 , wherein the first location comprises a portion of the data store that is stored within a first storage cluster and the replica location comprises a second storage cluster different from the first storage cluster.
8 . The method of claim 1 , further comprising sending the retrieved object to a client that requested the object.
9 . A system, comprising:
a plurality of storage clusters, each storage cluster comprising:
a plurality of object replicas;
a global index identifying storage locations of the object replicas across the plurality of storage clusters;
one or more memory; and
one or more processors in communication with the memory and configured to:
receive a request for an object;
identify locations of the object stored by a storage system by querying the global index stored in the storage cluster, the locations including at least one first location and at least one replica location;
determine which of the locations to use to read the object based on one or more criteria;
retrieve the object from the determined location; and
transmitting the retrieved object to a requesting client,
wherein the global index is replicated in the plurality of storage clusters.
10 . The system of claim 9 , wherein the one of the one or more criteria is based on network resources.
11 . The system of claim 10 , wherein the network resources comprise bandwidth consumption.
12 . The system of claim 10 , wherein determining which of the locations to use to read the object minimizes the network resources.
13 . The system of claim 9 , wherein the one of the one or more criteria is based on geographic location of the first location and the least one replica location.
14 . The system of claim 13 , wherein determining which of the locations to use to read the object comprises selecting a closest geographic location of the stored object to a geographic location of a client requesting the object.
15 . The system of claim 9 , wherein the first location comprises a portion of the data store that is stored within a first storage cluster and the replica location comprises a second storage cluster different from the first storage cluster.
16 . The system of claim 9 , wherein the one or more processors are further configured to send the retrieved object to a client that requested the object.
17 . A non-transitory computer-readable medium storing instructed executable by one or more processors in a distributed data storage system comprising a plurality of storage clusters, cause the one or more processors to perform a method, comprising:
receiving, by a first storage cluster, a request for an object;
identifying, by the first storage cluster, locations of the object stored by a storage system by querying a global index stored in the first storage cluster, the locations including at least one first location and at least one replica location;
determining, by the first storage cluster, which of the identified locations to use to read the object based on one or more criteria; and
retrieving the object from the determined location; and
transmitting the retrieved object to a requesting client,
wherein the global index is replicated in the plurality of storage clusters.
18 . The non-transitory computer-readable medium of claim 17 , wherein the one or more criteria is based on network resources.
19 . The non-transitory computer-readable medium of claim 18 , wherein the network resources comprise bandwidth consumption.
20 . The non-transitory computer-readable medium of claim 18 , wherein determining which of the locations to use to read the object minimizes the network resources.