System and method for scheduling backup workloads using a trained job resource mapping model
Techniques described herein relate to a method for managing jobs. Such techniques may include: generating a job completion entry that specifies that execution of a first job was completed within an estimated period of time for the first job, generating a second job completion entry that specifies that execution of a second job was not completed within a second estimated period of time for the second job, updating the job resource mapping model based on the first job completion entry and the second job completion entry to obtain a new updated job resource mapping model.
1. A method for managing jobs, the method comprising:
generating a job completion entry that specifies that execution of a first job was completed within an estimated period of time for the first job, wherein the first job comprises at least one of: a clone generation of an application executing in a first client device and a migration of the application between client devices;
generating a second job completion entry that specifies that execution of a second job was not completed within a second estimated period of time for the second job, wherein the second job comprises at least one of: a second clone generation of a second application and a second migration of the second application between the client devices;
updating the job resource mapping model based on the job completion entry and the second job completion entry to obtain a new updated job resource mapping model;
obtaining, by a backup manager, a new set of jobs to be executed;
generating, by a job time estimator, an estimated period of time for each job in the new set of jobs to obtain a set of estimated periods of time, wherein the job estimator is not modified using the job completion entry or the second job completion entry;
performing, using the set of estimated periods of time, a job resource mapping for each job in the new set of jobs using the updated job resource mapping model, wherein the first job and the second job are not in the new set of jobs; and
scheduling of the new set of jobs on a set of resources based on the job resource mapping, wherein execution of at least one of the new set of jobs is initiated after the scheduling.
2. The method of claim 1 , wherein generating the job completion entry comprises:
obtaining a notification that the job is complete;
making a determination that the job was completed within the estimated period of time; and
based on the determination, generating the job completion entry that specifies the job was completed within the estimated period of time.
3. The method of claim 2 , further comprising:
prior to performing the job resource mapping, obtaining the job time estimation of the job using the job time estimator.
4. The method of claim 1 , wherein the execution of the job results in storage of a backup in a backup storage system.
5. The method of claim 1 , wherein the job resource mapping model is based on job time estimations of previous executions of a previous set of jobs.
6. The method of claim 1 , wherein the scheduling of the new set of jobs on the set of resources is further based on a prioritization associated with each job in the new set of jobs.
7. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing jobs, the method comprising:
generating a job completion entry that specifies that execution of a first job was completed within an estimated period of time for the first job, wherein the first job comprises at least one of: a clone generation of an application executing in a first client device and a migration of the application between client devices;
generating a second job completion entry that specifies that execution of a second job was not completed within a second estimated period of time for the second job, wherein the second job comprises at least one of: a second clone generation of a second application and a second migration of the second application between the client devices;
updating the job resource mapping model based on the job completion entry and the second job completion entry to obtain a new updated job resource mapping model;
obtaining, by a backup manager, a new set of jobs to be executed;
generating, by a job time estimator, an estimated period of time for each job in the new set of jobs to obtain a set of estimated periods of time, wherein the job estimator is not modified using the job completion entry or the second job completion entry;
performing, using the set of estimated periods of time, a job resource mapping for each job in the new set of jobs using the updated job resource mapping model, wherein the first job and the second job are not in the new set of jobs; and
scheduling of the new set of jobs on a set of resources based on the job resource mapping, wherein execution of at least one of the new set of jobs is initiated after the scheduling.
8. The non-transitory computer readable medium of claim 7 , wherein generating the job completion entry comprises:
obtaining a notification that the job is complete;
making a determination that the job was completed within the estimated period of time;
based on the determination, generating the job completion entry that specifies the job was completed within the estimated period of time.
9. The non-transitory computer readable medium of claim 8 , further comprising:
prior to performing the job resource mapping, obtaining the job time estimation of the job using the job time estimator.
10. The non-transitory computer readable medium of claim 7 , wherein the execution of the job results in storage of a backup in a backup storage system.
11. The non-transitory computer readable medium of claim 7 , wherein the job resource mapping model is based on job time estimations of previous executions of a previous set of jobs.
12. The non-transitory computer readable medium of claim 7 , wherein the scheduling of the new set of jobs on the set of resources is further based on a prioritization associated with each job in the new set of jobs.
13. A system for forecasting backup failures, the system comprising:
a plurality of client devices each comprising a processor, a memory, and a persistent storage device;
a plurality of storage devices; and
a backup storage controller operatively connected to the plurality of client devices and the plurality of storage devices, and configured to:
generate a job completion entry that specifies that execution of a first job was completed within an estimated period of time for the first job, wherein the first job comprises at least one of: a clone generation of an application executing in a first client device and a migration of the application between client devices;
generate a second job completion entry that specifies that execution of a second job was not completed within a second estimated period of time for the second job, wherein the second job comprises at least one of: a second clone generation of a second application and a second migration of the second application between the client devices;
update the job resource mapping model based on the job completion entry and the second job completion entry to obtain a new updated job resource mapping model;
obtain, by a backup manager, a new set of jobs to be executed;
generate, by a job time estimator, an estimated period of time for each job in the new set of jobs to obtain a set of estimated periods of time, wherein the job estimator is not modified using the job completion entry or the second job completion entry;
perform, using the set of estimated periods of time, a job resource mapping for each job in the new set of jobs using the updated job resource mapping model, wherein the first job and the second job are not in the new set of jobs; and
schedule of the new set of jobs on a set of resources based on the job resource mapping, wherein execution of at least one of the new set of jobs is initiated after the scheduling.
14. The system of claim 13 , wherein generating the job completion entry comprises:
obtaining a notification that the job is complete;
making a determination that the job was completed within the estimated period of time; and
based on the determination, generating the job completion entry that specifies the job was completed within the estimated period of time.
15. The system of claim 14 , wherein the backup storage controller is further programmed to:
prior to performing the job resource mapping, obtain the job time estimation of the job using the job time estimator.
16. The system of claim 13 , wherein the job resource mapping model is based on job time estimations of previous executions of a previous set of jobs.
17. The system of claim 13 , wherein the scheduling of the new set of jobs on the set of resources is further based on a prioritization associated with each job in the new set of jobs.