Asynchronous distributed de-duplication for replicated content addressable storage clusters
A method is performed by a device of a group of devices in a distributed data replication system. The method includes storing an index of objects in the distributed data replication system, the index being replicated while the objects are stored locally by the plurality of devices in the distributed data replication system. The method also includes conducting a scan of at least a portion of the index and identifying a redundant replica(s) of the at least one of the objects based on the scan of the index. The method further includes de-duplicating the redundant replica(s), and updating the index to reflect the status of the redundant replica.
1. A computer-implemented method comprising:
receiving a request to delete a first object associated with a first location of a data store;
identifying, based on a set of one or more records identifying locations of objects stored by a storage system, one or more replicas of the first object that are associated with locations of the storage system that are different from the first location;
determining whether all of the one or more replicas are designated for de-duplication;
storing a copy of the first object at a second location different from the first location when all of the one or more replicas are designated for de-duplication; and
deleting the first object from the first location.
2. The method of claim 1 wherein determining whether a replica is designated for de-duplication comprises determining whether a record of the set of one or more records designates the replica for de-duplication.
3. The method of claim 2 wherein a record of the set of one or more records designates the replica for de-duplication to remove the replica and reduce required storage space.
4. The method of claim 1 wherein the first object is stored in a storage cluster and the first location is associated with a unique identification of the storage cluster and an address within the storage cluster.
5. The method of claim 1 wherein the first object is stored within a portion of the data store that is stored within a first storage cluster and at least one replica of the first object is stored within a second storage cluster different from the first storage cluster.
6. The method of claim 1 wherein storing a copy of the first object at the second location further comprises storing the copy of the first object at the second location when all of the one or more replicas are designated for de-duplication and less than all of the one or more replicas are designated for deletion.
7. The method of claim 1 wherein storing a copy of the first object at the second location further comprises storing the copy of the first object at the second location when all of the one or more replicas are designated for de-duplication and the first object is not designated for de-duplication.
8. The method of claim 1 wherein the first object comprises a file uploaded by a client.
9. A storage system comprising:
one or more processors;
memory; and
one or more programs stored in the memory configured for execution by the one or more processors, the one or more programs comprising instructions for:
receiving a request to delete a first object associated with a first location of a data store;
identifying, based on a set of one or more records identifying locations of objects stored by the storage system, one or more replicas of the first object that are associated with locations of the storage system that are different from the first location;
determining whether all of the one or more replicas are designated for de-duplication;
storing a copy of the first object at a second location different from the first location when all of the one or more replicas are designated for de-duplication; and
deleting the first object from the first location.
10. The storage system of claim 9 wherein determining whether a replica is designated for de-duplication comprises determining whether a record of the set of one or more records designates the replica for de-duplication.
11. The storage system of claim 10 wherein a record of the set of one or more records designates the replica for de-duplication to remove the replica and reduce required storage space.
12. The storage system of claim 9 wherein the first object is stored in a storage cluster and the first location is associated with a unique identification of the storage cluster and an address within the storage cluster.
13. The storage system of claim 9 wherein the first object is stored within a portion of the data store that is stored within a first storage cluster and at least one replica of the first object is stored within a second storage cluster different from the first storage cluster.
14. The storage system of claim 9 wherein storing a copy of the first object at the second location further comprises storing the copy of the first object at the second location when all of the one or more replicas are designated for de-duplication and less than all of the one or more replicas are designated for deletion.
15. The storage system of claim 9 wherein storing a copy of the first object at the second location further comprises storing the copy of the first object at the second location when all of the one or more replicas are designated for de-duplication and the first object is not designated for de-duplication.
16. The storage system of claim 9 wherein the first object comprises a file uploaded by a client.
17. A non-transitory computer readable storage medium storing one or more programs configured for execution by a plurality of devices in a distributed storage system, each having one or more processors and memory, the one or more programs comprising instructions for:
receiving a request to delete a first object associated with a first location of a data store;
identifying, based on a set of one or more records identifying locations of objects stored by a storage system, one or more replicas of the first object that are associated with locations of the storage system that are different from the first location;
determining whether all of the one or more replicas are designated for de-duplication;
storing a copy of the first object at a second location different from the first location when all of the one or more replicas are designated for de-duplication; and
deleting the first object from the first location.
18. The storage medium of claim 17 wherein determining whether a replica is designated for de-duplication comprises determining whether a replica of the first object has been designated for removal as a duplicate of the first object.
19. The storage medium of claim 17 wherein storing a copy of the first object at the second location further comprises storing the copy of the first object at the second location when all of the one or more replicas are designated for de-duplication and less than all of the one or more replicas are designated for deletion.
20. The storage medium of claim 17 wherein storing a copy of the first object at the second location further comprises storing the copy of the first object at the second location when all of the one or more replicas are designated for de-duplication and the first object is not designated for de-duplication.