SCALABLE JOB MANAGER SERVICE FOR DATA MANAGEMENT-AS-A-SERVICE (DMaaS)
An improved system architecture for scaling deployments of data management-as-a-service (DMaaS) distributes “job manager” functionality so that a large-scale data storage management system may be deployed in a cloud computing environment as DMaaS without experiencing performance bottlenecks that slow down core data protection operations at scale. The disclosed solution creates an infrastructure where job manager features run on any number of distinct machines or compute resources that act as local job managers. In the illustrative DMaaS, any number of components that scale horizontally (“the distributed components”) carry job management responsibility for storage management operations such as backup jobs, auxiliary copy jobs, archive jobs, etc. These distributed components are configured to perform locally as job managers and to see each storage management job through from beginning to end without centralized control. These distributed components or local job managers also maintain, collect, and locally store job metadata.
1 . A system comprising:
a first computing device comprising one or more first hardware processors and computer memory carrying computer programming instructions, which, when executed by the one or more first hardware processors, configure the first computing device to:
receive a job identifier for a first backup job to be performed for a dataset,
determine that, in a management database that is maintained by the first computing device, a media agent is associated with the dataset, and
based on determining that a second computing device that hosts the media agent also hosts a local job manager, delegate management of the first backup job to the local job manager at the second computing device; and
the second computing device, comprising one or more second hardware processors and computer memory carrying computer programming instructions, which, when executed by the one or more second hardware processors, configure the second computing device to:
receive, at the local job manager, a delegation from the first computing device to manage the first backup job, wherein the delegation comprises first metadata obtained from the management database, and wherein the first metadata comprises preferences for performing the first backup job,
store the first metadata at a local database configured at the second computing device,
autonomously manage the first backup job, for a lifecycle of the first backup job, without receiving further instructions about the first backup job from the first computing device, wherein autonomously managing the first backup job comprises instructing a data agent and the media agent to back up the dataset into a secondary copy, instructing the media agent to store the secondary copy at a secondary storage device indicated in the preferences, and storing metadata generated by the first backup job in the local database, wherein in the first backup job: the data agent and the media agent generate the secondary copy and the media agent stores the secondary copy at the secondary storage device, and
after the lifecycle of the first backup job has ended, transmit, to the first computing device, at least some metadata generated by the first backup job; and
wherein the first computing device is further configured to: update the management database with the at least some metadata received from the second computing device.
2 . The system of claim 1 , wherein the first backup job generates the secondary copy and stores the secondary copy at the secondary storage device.
3 . The system of claim 1 , wherein the data agent executes at the second computing device.
4 . The system of claim 1 , wherein the data agent executes at a third computing device that generates the dataset.
5 . The system of claim 1 , wherein the first computing device is further configured to: based on having delegated management of the first backup job of the dataset to the local job manager at the second computing device, delegate management of a subsequent backup job to be performed for the dataset to the local job manager at the second computing device.
6 . The system of claim 1 , wherein the second computing device is further configured to: before the lifecycle of the first backup job has ended, transmit status information about the first backup job to the first computing device.
7 . The system of claim 6 , wherein the first computing device is further configured to: display a status of the first backup job based on the status information received from the second computing device.
8 . The system of claim 1 , wherein the second computing device is further configured to: query the first computing device about configuration information related to the dataset.
9 . The system of claim 1 , wherein autonomously managing the first backup job comprises causing a process for the first backup job to be initiated at one or more of: the data agent, and the media agent.
10 . The system of claim 1 , wherein autonomously managing the first backup job comprises causing a process for the first backup job to be restarted at one or more of: the data agent, and the media agent.
11 . The system of claim 1 , wherein the first computing device is further configured to: delegate, to a local job manager at a third computing device, management of a second backup job to be performed for the dataset, wherein the second backup job starts after the lifecycle of the first backup job has ended; and
wherein the system further comprises the third computing device, comprising one or more third hardware processors and computer memory carrying computer programming instructions, which, when executed by the one or more third hardware processors, configure the third computing device to autonomously manage the second backup job, for a lifecycle of the second backup job, without receiving further instructions about the first backup job from the first computing device.
12 . The system of claim 1 , wherein the first computing device and the second computing device operate in a cloud computing environment, and wherein the first computing device is further configured to identify a plurality of third computing devices that operate in the cloud computing environment, wherein each third computing device hosts a local job manager; and delegate management to each third computing device at least one backup job, wherein the respective local job manager at each third computing device is configured to autonomously manage the at least one backup job delegated thereto by the first computing device, without receiving further instructions thereabout from the first computing device.
13 . A system operating in a cloud computing environment, the system comprising:
a first computing device comprising one or more first hardware processors and executing a storage manager that is configured to:
receive a job identifier for a first backup job to be performed for a dataset,
determine that, in a management database that is maintained by the storage manager, a media agent is associated with the dataset, and
based on determining that a second computing device that hosts the media agent also hosts a local job manager, delegate management of the first backup job to the local job manager at the second computing device; and
the second computing device, comprising one or more second hardware processors and executing the media agent and the local job manager, wherein the local job manager is configured to:
receive a delegation from the storage manager to manage the first backup job, wherein the delegation comprises first metadata obtained from the management database, and wherein the first metadata comprises preferences for performing the first backup job,
store the first metadata at a local database configured at the second computing device,
autonomously manage the first backup job, through an end of the first backup job, without receiving further instructions about the first backup job from the storage manager, wherein to autonomously manage the first backup job, the local job manager is configured to: instruct a data agent that is associated with the dataset and the media agent to generate a secondary copy of the dataset, instruct the media agent to store the secondary copy at a secondary storage device indicated in the preferences, and store metadata generated by the first backup job in the local database, and
after the first backup job has ended, transmit, to the storage manager, at least some metadata generated by the first backup job; and
wherein the first computing device is further configured to: update the management database with the at least some metadata received from the second computing device; and
wherein the first computing device and the second computing device operate in the cloud computing environment.
14 . The system of claim 13 , wherein, responsive to instructions received from the local job manager, the data agent and the media agent generate the secondary copy and the media agent stores the secondary copy at the secondary storage device.
15 . The system of claim 13 , wherein the data agent executes at the second computing device.
16 . The system of claim 13 , wherein the data agent executes at a third computing device that generates the dataset.
17 . The system of claim 13 , wherein the first computing device is further configured to: based on having delegated management of the first backup job of the dataset to the local job manager at the second computing device, delegate management of a subsequent backup job to be performed for the dataset to the local job manager at the second computing device.
18 . The system of claim 13 , wherein the second computing device is further configured to: before the first backup job ends, transmit status information about the first backup job to the storage manager.
19 . The system of claim 6 , wherein the storage manager is further configured to: cause a user interface to display a status of the first backup job based on the status information received from the second computing device.
20 . The system of claim 13 , wherein the local job manager is further configured to: query the storage manager about configuration information related to the dataset.
21 . The system of claim 13 , wherein to autonomously manage the first backup job, the local job manager is configured to: cause a process for the first backup job to be initiated at one or more of: the data agent, and the media agent.
22 . The system of claim 13 , wherein to autonomously manage the first backup job, the local job manager is configured to: cause a process for the first backup job to be restarted at one or more of: the data agent, and the media agent.
23 . The system of claim 13 , wherein the storage manager is further configured to: delegate, to a local job manager at a third computing device, management of a second backup job to be performed for the dataset, wherein the second backup job starts after the first backup job has ended; and
wherein the system further comprises the third computing device, comprising one or more third hardware processors, wherein the local job manager at the third computing device is configured to: autonomously manage the second backup job, through an end of the end of the second backup job, without receiving further instructions about the first backup job from the storage manager.