Intelligently scheduling resynchronization jobs in a distributed object-based storage system
Techniques for intelligently scheduling resynchronization jobs in a distributed object-based storage system are provided. In one set of embodiments, a storage node of the system can create a resynchronization job for a component of an object maintained by the system, where the resynchronization job defines one or more input/output (I/O) operations to be carried out with respect to the component. If a number of currently running resynchronization jobs on the storage node has reached a threshold, the storage node can further determine a priority level associated with the object; add the resynchronization job to an object queue for the object; and if the added resynchronization job is a first job in the object queue, add the object queue as a new queue entry to a global priority queue corresponding to the priority level associated with the object.
1. A method for performing intelligent resynchronization job scheduling in a distributed object-based storage system, the method comprising:
creating, by a storage node of the distributed object-based storage system, a resynchronization job for a component of an object maintained by the distributed object-based storage system, wherein the object is a container of data stored across one or more storage nodes of the distributed object-based storage system, wherein the component is a portion of the data of the object, and wherein the resynchronization job defines one or more input/output (PO) operations to be carried out with respect to the component; and
if a number of currently running resynchronization jobs on the storage node has reached a threshold:
determining, by the storage node, a priority level associated with the object;
adding, by the storage node, the resynchronization job to an object queue for the object; and
if the added resynchronization job is a first job in the object queue, adding the object queue as a new queue entry to a global priority queue corresponding to the priority level associated with the object.
2. The method of claim 1 further comprising, if the number of currently running resynchronization jobs has not yet reached the threshold:
immediately executing the resynchronization job; and
incrementing the number of currently running resynchronization jobs.
3. The method of claim 1 wherein the global priority queue is one of a plurality of global priority queues, each global priority queue corresponding to a possible object priority level.
4. The method of claim 3 further comprising, at a time the number of currently running resynchronization jobs falls below the threshold:
searching for a highest global priority queue having at least one queue entry; and
upon finding the highest global priority queue having at least one queue entry:
retrieving a first queue entry in the highest global priority queue;
retrieving a first object queue reference by the first queue entry;
retrieving a first resynchronization job in the first object queue;
removing the first resynchronization job from the first object queue; and
executing the first resynchronization job.
5. The method of claim 4 further comprising:
if the first object queue is empty after removing the first resynchronization job, removing the first queue entry from the highest global priority queue.
6. The method of claim 1 wherein the object is associated with a fault tolerance requirement indicating that the object should remain accessible in the face of one or more storage node failures.
7. The method of claim 1 wherein the resynchronization job is associated with a job priority level that is different from the priority level for the object.
8. A non-transitory computer readable storage medium having stored thereon program code executable by a storage node of a distributed object-based storage system, the program code embodying a method for implementing intelligent resynchronization job scheduling in the distributed object-based storage system, the method comprising:
creating a resynchronization job for a component of an object maintained by the distributed object-based storage system, wherein the object is a container of data stored across one or more storage nodes of the distributed object-based storage system, wherein the component is a portion of the data of the object, and wherein the resynchronization job defines one or more input/output (I/O) operations to be carried out with respect to the component; and
if a number of currently running resynchronization jobs on the storage node has reached a threshold:
determining a priority level associated with the object;
adding the resynchronization job to an object queue for the object; and
if the added resynchronization job is a first job in the object queue, adding the object queue as a new queue entry to a global priority queue corresponding to the priority level associated with the object.
9. The non-transitory computer readable storage medium of claim 8 wherein if the number of currently running resynchronization jobs has not yet reached the threshold, the method further comprises:
immediately executing the resynchronization job; and
incrementing the number of currently running resynchronization jobs.
10. The non-transitory computer readable storage medium of claim 8 wherein the global priority queue is one of a plurality of global priority queues, each global priority queue corresponding to a possible object priority level.
11. The non-transitory computer readable storage medium of claim 10 wherein the method further comprises, at a time the number of currently running resynchronization jobs falls below the threshold:
searching for a highest global priority queue having at least one queue entry; and
upon finding the highest global priority queue having at least one queue entry:
retrieving a first queue entry in the highest global priority queue;
retrieving a first object queue reference by the first queue entry;
retrieving a first resynchronization job in the first object queue;
removing the first resynchronization job from the first object queue; and
executing the first resynchronization job.
12. The non-transitory computer readable storage medium of claim 11 wherein the method further comprises:
if the first object queue is empty after removing the first resynchronization job, removing the first queue entry from the highest global priority queue.
13. The non-transitory computer readable storage medium of claim 8 wherein the object is associated with a fault tolerance requirement indicating that the object should remain accessible in the face of one or more storage node failures.
14. The non-transitory computer readable storage medium of claim 8 wherein the resynchronization job is associated with a job priority level that is different from the priority level for the object.
15. A storage node of a distributed object-based storage system, the storage node comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
create a resynchronization job for a component of an object maintained by the distributed object-based storage system, wherein the object is a container of data stored across one or more storage nodes of the distributed object-based storage system, wherein the component is a portion of the data of the object, and wherein the resynchronization job defines one or more input/output (I/O) operations to be carried out with respect to the component; and
if a number of currently running resynchronization jobs on the storage node has reached a threshold:
determine a priority level associated with the object;
add the resynchronization job to an object queue for the object; and
if the added resynchronization job is a first job in the object queue, add the object queue as a new queue entry to a global priority queue corresponding to the priority level associated with the object.
16. The storage node of claim 15 wherein if the number of currently running resynchronization jobs has not yet reached the threshold, the program code further causes the processor to:
immediately execute the resynchronization job; and
increment the number of currently running resynchronization jobs.
17. The storage node of claim 15 wherein the global priority queue is one of a plurality of global priority queues, each global priority queue corresponding to a possible object priority level.
18. The storage node of claim 17 wherein the program code further causes the processor to, at a time the number of currently running resynchronization jobs falls below the threshold:
search for a highest global priority queue having at least one queue entry; and
upon finding the highest global priority queue having at least one queue entry:
retrieve a first queue entry in the highest global priority queue;
retrieve a first object queue reference by the first queue entry;
retrieve a first resynchronization job in the first object queue;
remove the first resynchronization job from the first object queue; and
execute the first resynchronization job.
19. The storage node of claim 18 wherein the program code further causes the processor to:
if the first object queue is empty after removing the first resynchronization job, remove the first queue entry from the highest global priority queue.
20. The storage node of claim 15 wherein the object is associated with a fault tolerance requirement indicating that the object should remain accessible in the face of one or more storage node failures.
21. The storage node of claim 15 wherein the resynchronization job is associated with a job priority level that is different from the priority level for the object.