Pruning of blob replicas

1. A computer-implemented method for generating and distributing replica removal requests for objects in a distributed storage system, comprising:

at a computer system including one or more processors and memory storing one or more programs for execution by the one or more processors:

for a respective object in a distributed storage system,

identifying one or more replicas of the object to be removed from the distributed storage system based at least in part on replication policies for the object;

generating replica removal requests for the one or more replicas of the object, wherein a respective replication policy includes criteria that are used to determine when to remove respective replicas, wherein a respective replica removal request instructs a respective instance of the distributed storage system to remove a respective replica of the respective object so as to at least partially satisfy replication policies for the respective object, and wherein generating the replica removal requests for the one or more replicas of the object includes:

identifying violated replication policies for the object;

selecting the one or more replicas of the object to be removed from instances of the distributed storage system based on last access times of replicas of the respective object and the current storage space available at the instances of the distributed storage system including the replicas of the respective object; and

generating the replica removal requests for the one or more selected replicas of the respective object; and

distributing the replica removal requests for the one or more replicas of the object in the distributed storage system to respective instances of the distributed storage system corresponding to the replica removal requests for execution.

2. The method of claim 1 , wherein generating the replica removal requests for the one or more replicas of the object includes:

determining that an instance of the distributed storage system including the replica of the object is being deactivated;

determining whether the deactivation of the instance of the distributed storage system causes a number of replicas of the object to be below a minimum number of replicas of the object as specified by the replication policies for the object;

if the deactivation of the instance of the distributed storage system causes the number of replicas of the object to be below the minimum number of replicas of the object,

generating a replication request to replicate the object based at least in part on replication policies for the object and a current state of the distributed storage system;

distributing the replication request to a respective instance of the distributed storage system for execution; and

generating the replica removal request for the object only after the replication request to replicate the object has been completed.

3. The method of claim 2 , wherein the current state of the distributed storage system includes:

a current network state;

current user quotas for storage space in the distributed storage system;

storage space in the distributed storage system that are currently used by users;

current storage space available at instances of the distributed storage system;

current statuses of replication queues at instances of the distributed storage system;

current planned maintenance operations zones; and

a list of current replicas of objects in the distributed storage system.

4. The method of claim 1 , wherein a replication policy for an object includes criteria selected from the group consisting of:

a minimum number of replicas of the object that must be present in the distributed storage system;

a maximum number of the replicas of the object that are allowed to be present in the distributed storage system;

storage device types on which the replicas of the object are to be stored;

locations at which the replicas of the object may be stored;

locations at which the replicas of the object may not be stored; and

a range of ages for the object during which the replication policy for the object applies.

5. The method of claim 1 , wherein replica removal requests are generated for an object whose replicas violate replication policies for the object.

6. The method of claim 1 , wherein replica removal requests are generated for an object for which dynamic replication requests caused the number of replicas of the object to exceed the number of replicas of the object specified in the replication policies for the object, wherein a dynamic replication request generates a replica of the object based at least in part on a current level of demand for the object.

7. A system for generating and distributing replica removal requests for objects in a distributed storage system, comprising:

one or more processors;

memory; and

one or more programs stored in the memory, the one or more programs comprising instructions to:

identify one or more replicas of the object to be removed from the distributed storage system based at least in part on replication policies for the object;

generate replica removal requests for the one or more replicas of the object, wherein a respective replication policy includes criteria that are used to determine when to remove respective replicas, wherein a respective replica removal request instructs a respective instance of the distributed storage system to remove a respective replica of the respective object so as to at least partially satisfy replication policies for the respective object, and wherein instructions to generate the replica removal requests for the one or more replicas of the object includes instructions to:

identify violated replication policies for the object;

select the one or more replicas of the object to be removed from instances of the distributed storage system based on last access times of replicas of the respective object and the current storage space available at the instances of the distributed storage system including the replicas of the respective object; and

generate the replica removal requests for the one or more selected replicas of the respective object; and

distribute the replica removal requests for the one or more replicas of the object in the distributed storage system to respective instances of the distributed storage system corresponding to the replica removal requests for execution.

8. The system of claim 7 , wherein generating the replica removal requests for the one or more replicas of the object includes:

determining that an instance of the distributed storage system including the replica of the object is being deactivated;

determining whether the deactivation of the instance of the distributed storage system causes a number of replicas of the object to be below a minimum number of replicas of the object as specified by the replication policies for the object;

if the deactivation of the instance of the distributed storage system causes the number of replicas of the object to be below the minimum number of replicas of the object,

generating a replication request to replicate the object based at least in part on replication policies for the object and a current state of the distributed storage system;

distributing the replication request to a respective instance of the distributed storage system for execution; and

generating the replica removal request for the object only after the replication request to replicate the object has been completed.

9. The system of claim 8 , wherein the current state of the distributed storage system includes:

a current network state;

current user quotas for storage space in the distributed storage system;

storage space in the distributed storage system that are currently used by users;

current storage space available at instances of the distributed storage system;

current statuses of replication queues at instances of the distributed storage system;

current planned maintenance operations zones; and

a list of current replicas of objects in the distributed storage system.

10. The system of claim 7 , wherein a replication policy for an object includes criteria selected from the group consisting of:

a minimum number of replicas of the object that must be present in the distributed storage system;

a maximum number of the replicas of the object that are allowed to be present in the distributed storage system;

storage device types on which the replicas of the object are to be stored;

locations at which the replicas of the object may be stored;

locations at which the replicas of the object may not be stored; and

a range of ages for the object during which the replication policy for the object applies.

11. The system of claim 7 , wherein replica removal requests are generated for an object whose replicas violate replication policies for the object.

12. The system of claim 7 , wherein replica removal requests are generated for an object for which dynamic replication requests caused the number of replicas of the object to exceed the number of replicas of the object specified in the replication policies for the object, wherein a dynamic replication request generates a replica of the object based at least in part on a current level of demand for the object.

13. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions to:

identify one or more replicas of the object to be removed from the distributed storage system based at least in part on replication policies for the object;

generate replica removal requests for the one or more replicas of the object, wherein a respective replication policy includes criteria that are used to determine to remove respective replicas, wherein a respective replica removal request instructs a respective instance of the distributed storage system to remove a respective replica of the respective object so as to at least partially satisfy replication policies for the respective object, and wherein instructions to generate the replica removal requests for the one or more replicas of the object includes instructions to:

identify violated replication policies for the object;

select the one or more replicas of the object to be removed from instances of the distributed storage system based on last access times of replicas of the respective object and the current storage space available at the instances of the distributed storage system including the replicas of the respective object; and

generate the replica removal requests for the one or more selected replicas of the respective object; and

distribute the replica removal requests for the one or more replicas of the object in the distributed storage system to respective instances of the distributed storage system corresponding to the replica removal requests for execution.

14. The computer readable storage medium of claim 13 , wherein generating the replica removal requests for the one or more replicas of the object includes:

determining that an instance of the distributed storage system including the replica of the object is being deactivated;

determining whether the deactivation of the instance of the distributed storage system causes a number of replicas of the object to be below a minimum number of replicas of the object as specified by the replication policies for the object;

if the deactivation of the instance of the distributed storage system causes the number of replicas of the object to be below the minimum number of replicas of the object,

generating a replication request to replicate the object based at least in part on replication policies for the object and a current state of the distributed storage system;

distributing the replication request to a respective instance of the distributed storage system for execution; and

generating the replica removal request for the object only after the replication request to replicate the object has been completed.

15. The computer readable storage medium of claim 14 , wherein the current state of the distributed storage system includes:

a current network state;

current user quotas for storage space in the distributed storage system;

storage space in the distributed storage system that are currently used by users;

current storage space available at instances of the distributed storage system;

current statuses of replication queues at instances of the distributed storage system;

current planned maintenance operations zones; and

a list of current replicas of objects in the distributed storage system.

16. The computer readable storage medium of claim 13 , wherein a replication policy for an object includes criteria selected from the group consisting of:

a minimum number of replicas of the object that must be present in the distributed storage system;

a maximum number of the replicas of the object that are allowed to be present in the distributed storage system;

storage device types on which the replicas of the object are to be stored;

locations at which the replicas of the object may be stored;

locations at which the replicas of the object may not be stored; and

a range of ages for the object during which the replication policy for the object applies.

17. The computer readable storage medium of claim 13 , wherein replica removal requests are generated for an object whose replicas violate replication policies for the object.

18. The computer readable storage medium of claim 13 , wherein replica removal requests are generated for an object for which dynamic replication requests caused the number of replicas of the object to exceed the number of replicas of the object specified in the replication policies for the object, wherein a dynamic replication request generates a replica of the object based at least in part on a current level of demand for the object.