Scaled deployment of data protection resources for data originating in an application orchestrator
Data protection resources are automatically scaled to the needs of data source(s) in an application orchestrator computing environment, such as a cluster in a Kubernetes deployment. The approach is adaptable to data sources in production clusters or application suites that are not application orchestrator deployments, such as a cloud-based database-as-a-service (DBaaS). A data storage management system protects cluster-based data with an elastic number of data protection resources (e.g., data agents, media agents), which are deployed on demand. The number of data protection resources deployed for a particular job are appropriate to the workload(s) at present and depend on a variety of scaling factors. In some embodiments, data protection resources are deployed within the same cluster as the data sources. In other embodiments, a separate infrastructure cluster provides the data protection resources on demand, and connects to any number and types of data sources, whether cloud-based or otherwise, without limitation.
1 . A method implemented by computing resources that comprise one or more hardware processors coupled to non-transitory computer-readable media that carry computer programming instructions, which configure the computing resources to perform the method, comprising:
by a first data agent that is deployed as a containerized application within a first computing environment, receiving notice that a storage operation has been initiated for one or more first data sources that operate in an application orchestrator computing environment,
wherein the containerized application is hosted by a first computing resource operating outside the application orchestrator computing environment;
by the first data agent, responsive to the notice, causing a first number of second data agents to be deployed,
wherein each second data agent is deployed as a containerized application that is hosted by a corresponding second computing resource within the first computing environment;
by the first data agent, distributing tasks of the storage operation among the first number of second data agents;
by the first data agent, after completion of the storage operation, causing each second data agent to be deactivated, wherein the first data agent continues to operate within the first computing environment; and
wherein the storage operation generates one or more secondary copies of the one or more first data sources, and
wherein the storage operation stores the one or more secondary copies at data storage resources, which are configured outside the application orchestrator computing environment.
2 . The method of claim 1 further comprising:
by the first data agent, receiving notice that a second storage operation has been initiated for second data sources that operate in the application orchestrator computing environment, wherein the second data sources are distinct from the one or more first data sources; and
by the first data agent, determining a second number of second data agents needed for the second storage operation, wherein the second number differs from the first number, based on a difference between the second data sources and the one or more first data sources.
3 . The method of claim 1 , wherein the application orchestrator computing environment is configured as a first Kubernetes cluster.
4 . The method of claim 1 , wherein the first computing environment is configured as a first Kubernetes cluster, and wherein the application orchestrator computing environment is configured as a second Kubernetes cluster that is distinct from the first Kubernetes cluster.
5 . The method of claim 4 , wherein the first computing resource is configured as a pod within the first Kubernetes cluster, and wherein each corresponding second computing resource is configured as a second pod within the first Kubernetes cluster, and wherein deactivating each second data agent comprises deactivating the second pod.
6 . The method of claim 1 , wherein at least one of the application orchestrator computing environment and the first computing environment operates in a cloud computing environment.
7 . The method of claim 1 , wherein at least one of the application orchestrator computing environment and the first computing environment operates in a non-cloud data center.
8 . The method of claim 1 further comprising:
by each second data agent during the storage operation, transmitting to a media agent, first data obtained from one of the one or more first data sources, wherein the media agent executes on a computing resource; and
by the media agent, generating at least one of the one or more secondary copies, and storing the at least one of the one or more secondary copies at one of the data storage resources.
9 . The method of claim 1 further comprising:
by the first data agent, causing a third number of third data agents to be deployed, wherein each third data agent is deployed as a containerized application that is hosted by a corresponding third computing resource within the application orchestrator computing environment, wherein each third data agent is configured to transmit to a corresponding second data agent, first data obtained by the respective third data agent from one of the one or more first data sources; and
by the first data agent, after completion of the storage operation, causing each third data agent to be deactivated within the application orchestrator computing environment.
10 . The method of claim 1 , further comprising:
before initiating the storage operation, pre-configuring, at a storage manager, a plurality of client entities, wherein each client entity among the plurality of client entities is assigned a unique client identifier, without having a corresponding connection to a deployed data agent;
by each second data agent deployed in the first computing environment, requesting from the storage manager, a client identifier to enable the respective second data agent to register as a client;
by the storage manager, assigning to each requesting second data agent a unique client identifier from the plurality of client entities that were pre-configured; and
after each second data agent is deactivated, de-registering, by the storage manager, the respective second data agent; and
by the storage manager, after the de-registering, re-using in subsequent storage operations at least one client entity among the pre-configured plurality of client entities;
wherein the storage manager is hosted by a third computing resource that is distinct from the first computing resource.
11 . The method of claim 10 further comprising:
by the storage manager, initiating a second storage operation of second data sources of the application orchestrator computing environment;
by the first data agent, determining a second number of second data agents needed for the second storage operation; and
wherein the second number differs from the first number, based on a difference between the second data sources and the one or more first data sources.
12 . A method implemented by computing resources that comprise one or more hardware processors coupled to non-transitory computer-readable media that carry computer programming instructions, which configure the computing resources to perform the method, comprising:
by a first data agent that is deployed as a containerized application on a first pod within a first Kubernetes cluster, receiving notice that a storage operation has been initiated for one or more first data sources that operate in a second Kubernetes cluster;
by the first data agent, responsive to the notice, causing a first number of second data agents to be deployed,
wherein each second data agent is deployed as a containerized application on a corresponding second pod within the first Kubernetes cluster;
by the first data agent, distributing tasks of the storage operation among the first number of second data agents,
wherein the storage operation generates one or more secondary copies of the one or more first data sources, and
wherein the storage operation stores the one or more secondary copies at data storage resources, which are configured outside the second Kubernetes cluster; and
by the first data agent, after completion of the storage operation, causing each second data agent to be deactivated, wherein the first data agent continues to operate within the first Kubernetes cluster.
13 . The method of claim 12 , wherein deactivating each second data agent comprises deactivating its corresponding second pod.
14 . The method of claim 12 , wherein at least one of the second Kubernetes cluster and the first Kubernetes cluster operates in a cloud computing environment.
15 . The method of claim 12 , wherein at least one of the second Kubernetes cluster and the first Kubernetes cluster operates in a non-cloud data center.
16 . The method of claim 12 further comprising:
by each second data agent during the storage operation, transmitting to a media agent, first data obtained from one of the one or more first data sources, wherein the media agent executes on a computing resource outside the first Kubernetes cluster; and
by the media agent, generating at least one of the one or more secondary copies, and storing the at least one of the one or more secondary copies at one of the data storage resources, which are configured outside the first Kubernetes cluster.
17 . The method of claim 12 further comprising:
by the first data agent, causing a third number of third data agents to be deployed, wherein each third data agent is deployed as a containerized application on a pod within the second Kubernetes cluster, wherein each third data agent is configured to transmit to a corresponding second data agent, first data obtained by the respective third data agent from one of the one or more first data sources; and
by the first data agent, after completion of the storage operation, causing each third data agent to be deactivated within the second Kubernetes cluster.
18 . The method of claim 12 further comprising:
by the first data agent, receiving notice that a second storage operation has been initiated for second data sources that operate in the second Kubernetes cluster, wherein the second data sources are distinct from the one or more first data sources;
by the first data agent, determining a second number of second data agents needed for the second storage operation; and
wherein the second number differs from the first number, based on a difference between the second data sources and the one or more first data sources.
19 . The method of claim 12 , wherein the first number of second data agents needed for the storage operation is based on one or more of: (i) a number of applications among the one or more first data sources, (ii) a total amount of data in the one or more first data sources, (iii) a data throughput rate between a deployed second data agent and a media agent that stores secondary copies, generated by the storage operation, into data storage resources, (iv) a number of data streams administered to operate between a deployed second data agent and a media agent that stores secondary copies, generated by the storage operation, into data storage resources.
20 . The method of claim 12 , wherein the first number of second data agents needed for the storage operation is based on one or more of: (i) a recovery point objective, expressed as a duration of time, for the one or more first data sources, (ii) an amount of random access memory (RAM) configured at the first pod hosting the first data agent, (iii) an amount of computing resources (CPU) configured at the first pod hosting the first data agent, and (iv) a pre-configured maximum value for the first number of second data agents.