Job management and resource allocation in a data protection system
A system according to certain aspects improves the process of managing jobs and allocating resources to those jobs. The system can include a job manager that processes and generates resource allocation requests for a plurality of jobs and a resource allocator that allocates computing resources for the plurality of jobs based on the resource allocation requests. The process of allocating resources may involve locking one or more resource objects before allocating the computing resources for the plurality of jobs and unlocking the locked resource objects after the computing resources have been allocated for the plurality of jobs.
1. A method for managing resource allocation in a data storage system, the method comprising:
with one or more computing devices in a networked data storage system,
initiating processing of a first data protection job of a plurality of data protection jobs;
asynchronously sending a resource allocation request for allocation of one or more computing resources within the data storage system for use in performing the first data protection job, wherein the computing resources include one or more secondary storage devices;
initiating processing of a second data protection job of the plurality of data protection jobs without waiting for completion of the resource allocation request;
in response to the resource allocation request, locking one or more resource objects associated with the computing resources such that the one or more resource objects become unmodifiable;
allocating, while the one or more resource objects are locked, a portion of the computing resources for use in performing the first data protection job; and
subsequent to said allocating, unlocking the one or more resource objects associated with the computing resources such that the one or more resource objects are no longer unmodifiable.
2. The method of claim 1 , wherein the one or more computing devices are not blocked by the resource allocation request from further processing additional data protection jobs.
3. The method of claim 1 , further comprising, after the resource allocation request associated with the first data protection job has been sent, but has not yet been fulfilled, performing one or more job management tasks associated with the first data protection job.
4. The method of claim 1 wherein the one or more resource objects comprise tables including metadata associated with the computing resources.
5. The method of claim 1 wherein the one or more computing devices implement asynchronous software threads,
wherein a first thread of the asynchronous software threads performs said initiating processing of the first data protection job, said asynchronously sending, and said initiating processing of the second data protection job,
wherein a second thread of the asynchronous software threads performs said locking, said allocating, and said unlocking.
6. The method of claim 1 , wherein the resource allocation request specifies a first amount of computing resources needed to perform the first data protection job, and wherein the method further comprises performing said initiating the processing of the second data protection job before completing said allocating.
7. The method of claim 6 , further comprising before completing said allocating, sending another resource allocation request based on the second data protection job, the another resource allocation request specifying a second amount of computing resources needed to perform the second data protection job.
8. The method of claim 1 , further comprising creating, in one or more of the resource objects locked in response to the resource allocation request, an association between the first data protection job and the respective portion of the computing resources allocated to the first data protection job.
9. The method of claim 8 , wherein said one or more resource objects comprise a resource allocation table.
10. The method of claim 1 , wherein the one or more computing resources further include one or more computing devices.
11. The method of claim 1 ,
wherein said asynchronously sending comprises asynchronously sending a single combined resource allocation request for allocation of one or more computing resources within the data storage system for use in performing the first data protection job and for use in performing at least one other data protection job,
further wherein the computing resources for use in performing the first data protection job and for use in performing at least one other data protection job are allocated without locking and unlocking the one or more resource objects more than once.
12. The method of claim 1 , further comprising processing a group of resource allocation requests for a corresponding group of the data protection jobs in bulk, to allocate computing resources requested in the group of resource allocation requests without locking and unlocking the one or more resource objects more than once.
13. A system comprising:
one or more hardware processors within a networked data storage system and programmed to:
initiate processing of a first data protection job of a plurality of data protection jobs;
asynchronously send a resource allocation request for allocation of one or more computing resources within the data storage system for the first data protection job, wherein the computing resources include one or more secondary storage devices;
initiate processing of a second data protection job of the plurality of data protection jobs without waiting for completion of the resource allocation request;
in response to the resource allocation request, lock one or more resource objects associated with the computing resources;
allocate, while the one or more resource objects are locked, a respective portion of the computing resources for the first data protection job; and
subsequent to the allocation of the respective portion of the computing resources, unlock the one or more resource objects associated with the computing resources.
14. The system of claim 13 , wherein the one or more hardware processors are further programmed to, after the resource allocation request associated with the first data protection job has been issued but has not yet been fulfilled, perform one or more job management tasks associated with the first data protection job.
15. The system of claim 13 , wherein the resource allocation request specifies a first amount of computing resources needed to perform the first data protection job, wherein the one or more hardware processors are further programmed to initiate the processing of the second data protection job before the first amount of computing resources is allocated for the first data protection job based on the first resource allocation request.
16. The system of claim 15 , wherein the one or more hardware processors are further programmed to send, before the first amount of computing resources is allocated for the first data protection job based on the first resource allocation request, another resource allocation request based on the second data protection job, said another resource allocation request specifying a second amount of computing resources needed to perform the second data protection job.
17. The system of claim 13 wherein the one or more processors are programmed to:
send a single combined resource allocation request for allocation of computing resources within the data storage system for use in performing the first data protection job and for use in performing at least one other data protection job, and
wherein the one or more processors process the single resource allocation request such that the computing resources for use in performing the first data protection job and the computing resources for use in performing at least one other data protection job are allocated without locking and unlocking the one or more resource objects more than once.
18. The system of claim 13 , wherein the one or more processors are further programmed to process a group of resource allocation requests for a corresponding group of the data protection jobs in bulk, to allocate computing resources requested in the group of resource allocation requests without locking and unlocking the one or more resource objects more than once.
19. Non-transitory computer-readable memory comprising instructions, which when executed by a computer, cause the computer to perform a method for managing resource allocation in a data storage system, the method comprising:
initiating processing of a first data protection job of a plurality of data protection jobs;
asynchronously sending a resource allocation request for allocation of one or more computing resources within the data storage system for use in performing the first data protection job, wherein the computing resources include one or more secondary storage devices;
initiating processing of a second data protection job of the plurality of data protection jobs without waiting for completion of the resource allocation request;
in response to the resource allocation request, locking one or more resource objects associated with the computing resources;
allocating, while the one or more resource objects are locked, a portion of the computing resources for the first data protection job; and
subsequent to said allocating, unlocking the one or more resource objects associated with the computing resources.