SCALED DEPLOYMENT OF DATA PROTECTION RESOURCES FOR DATA ORIGINATING IN A PRODUCTION CLUSTER
Data protection resources are automatically scaled to the needs of data source(s) in an application orchestrator computing environment, such as a cluster in a Kubernetes deployment. The approach is adaptable to data sources in production clusters or application suites that are not application orchestrator deployments, such as a cloud-based database-as-a-service (DBaaS). A data storage management system protects cluster-based data with an elastic number of data protection resources (e.g., data agents, media agents), which are deployed on demand. The number of data protection resources deployed for a particular job are appropriate to the workload(s) at present and depend on a variety of scaling factors. In some embodiments, data protection resources are deployed within the same cluster as the data sources. In other embodiments, a separate infrastructure cluster provides the data protection resources on demand, and connects to any number and types of data sources, whether cloud-based or otherwise, without limitation.
1 . A system comprising a storage manager, which comprises one or more hardware processors coupled to non-transitory computer-readable media that carry computer programming instructions, which configure the storage manager to:
based on having configured, at the storage manager, an administrative entity that corresponds to an application orchestrator computing environment comprising first data sources to be protected by the system, and before initiating any storage operations for the first data sources, generate a plurality of client identifiers for future use;
based on having initiated a backup operation to back up a first data source among the first data sources,
receive a request for a client identifier from a first data agent that is associated with the first data source,
wherein the first data agent executes on a first compute resource that is distinct from the storage manager;
assign a first client identifier to the first data agent from among the plurality of client identifiers;
based on the first client identifier, register the first data agent as a first client entity, which enables the first data agent to participate in the backup operation;
after the first data agent is deactivated, based on a completion of the backup operation, de-register the first data agent; and
in a subsequent backup operation, re-use the first client identifier with a second data agent, whether or not the second data agent is identical to the first data agent.
2 . The system of claim 1 , wherein the application orchestrator computing environment is configured as a first Kubernetes cluster.
3 . The system of claim 2 , wherein the first compute resource operates in a second Kubernetes cluster that is distinct from the first Kubernetes cluster.
4 . The system of claim 1 , wherein the storage manager operates outside the application orchestrator computing environment.
5 . The system of claim 1 , wherein the first compute resource operates outside the application orchestrator computing environment.
6 . The system of claim 1 , wherein the storage manager is further configured to:
initiate the backup operation, comprising transmitting notice of the backup operation to a coordinator data agent, wherein the coordinator data agent executes on a second compute resource that is distinct from the storage manager and from the first compute resource;
wherein the coordinator data agent is deployed as a containerized application that is hosted by a second compute resource within a first computing environment, and wherein the coordinator data agent is configured to:
receive the notice of the backup operation,
responsive to the notice, cause the first data agent to be deployed as a containerized application that is hosted by the first compute resource within the first computing environment;
distribute tasks of the backup operation to at least the first data agent; and
after completion of the backup operation, cause the first data agent to be deactivated, wherein the coordinator data agent continues to operate within the first computing environment;
wherein the backup operation generates one or more secondary copies of the first data source, which are stored outside the application orchestrator computing environment.
7 . The system of claim 6 , wherein the first computing environment is configured as a first Kubernetes cluster, and wherein the application orchestrator computing environment is configured as a second Kubernetes cluster that is distinct from the first Kubernetes cluster;
wherein the first compute resource is configured as a pod within the first Kubernetes cluster, and wherein the second compute resource is configured as a second pod within the first Kubernetes cluster, and wherein deactivating the first data agent comprises deactivating the second pod.
8 . The system of claim 6 , wherein the storage manager comprises a data structure, which comprises one or more configuration files that act as templates for instantiating and containerizing the first data agent.
9 . The system of claim 6 , wherein the coordinator data agent is further configured to:
determine a number of data agents needed for the backup operation, wherein the first data agent is among the number of data agents;
responsive to the notice, cause the number of data agents to be deployed as a respective containerized application that is hosted by a corresponding compute resource within the first computing environment;
distribute tasks of the backup operation to the number of data agents; and
after completion of the backup operation, cause the number of data agents to be deactivated.
10 . The system of claim 1 , wherein at least one of: the application orchestrator computing environment, the storage manager, and the first data agent, operates in a cloud computing environment.
11 . The system of claim 1 , wherein at least one of: the application orchestrator computing environment, the storage manager, and the first data agent, operates in a non-cloud data center.
12 . A system comprising a storage manager, which comprises one or more hardware processors coupled to non-transitory computer-readable media that carry computer programming instructions, which configure the storage manager to:
based on having configured, at the storage manager, an administrative entity that corresponds to a production cluster comprising first data sources to be protected by the system, and before initiating any storage operations for the first data sources, generate a plurality of client identifiers for future use;
based on having initiated a backup operation to back up a first data source among the first data sources,
receive a request for a client identifier from a first data agent that is associated with the first data source,
wherein the first data agent executes on a first compute resource that is distinct from the storage manager;
assign a first client identifier to the first data agent from among the plurality of client identifiers;
based on the first client identifier, register the first data agent as a first client entity, which enables the first data agent to participate in the backup operation;
after the first data agent is deactivated based on a completion of the backup operation, de-register the first data agent; and
in a subsequent backup operation, re-use the first client identifier with a second data agent, whether or not the second data agent is identical to the first data agent.
13 . The system of claim 12 , wherein the production cluster is configured as a first Kubernetes cluster.
14 . The system of claim 13 , wherein the first compute resource operates in a second Kubernetes cluster that is distinct from the first Kubernetes cluster.
15 . The system of claim 12 , wherein the storage manager operates outside the production cluster.
16 . The system of claim 12 , wherein the first compute resource operates outside the production cluster.
17 . The system of claim 12 , wherein the storage manager is further configured to:
initiate the backup operation, comprising transmitting notice of the backup operation to a coordinator data agent, wherein the coordinator data agent executes on a second compute resource that is distinct from the storage manager and from the first compute resource;
wherein the coordinator data agent executes on a second compute resource within a first computing environment that is distinct from the production cluster, and wherein the coordinator data agent is configured to:
receive the notice of the backup operation,
responsive to the notice, cause the first data agent to be deployed on the first compute resource within the first computing environment;
distribute tasks of the backup operation to at least the first data agent; and
after completion of the backup operation, cause the first data agent to be deactivated, wherein the coordinator data agent continues to operate within the first computing environment;
wherein the backup operation generates one or more secondary copies of the first data source, which are stored outside the production cluster.
18 . The system of claim 17 , wherein the first computing environment is configured as a first Kubernetes cluster, and wherein the production cluster is configured as a second Kubernetes cluster that is distinct from the first Kubernetes cluster;
wherein the first compute resource is configured as a pod within the first Kubernetes cluster, and wherein the second compute resource is configured as a second pod within the first Kubernetes cluster, and wherein deactivating the first data agent comprises deactivating the second pod.
19 . The system of claim 17 , wherein the storage manager comprises a data structure, which comprises one or more configuration files that act as templates for instantiating the first data agent.
20 . The system of claim 17 , wherein the coordinator data agent is further configured to:
determine a number of data agents needed for the backup operation, wherein the first data agent is among the number of data agents;
responsive to the notice, cause the number of data agents to be deployed on a corresponding compute resource within the first computing environment;
distribute tasks of the backup operation to the number of data agents; and
after completion of the backup operation, cause the number of data agents to be deactivated.