Backup from a Kubernetes cluster using lightweight on-demand in-cluster resources
Because Kubernetes clusters can be ephemeral, backing up in-cluster data to storage outside the cluster is important. Prior art solutions used the cluster's API server, which facilitates communications with the cluster control plane, to transfer backup data through the API server. However, the API server as a data transfer node has resiliency weaknesses and can slow down backup job performance. The present solution provides a more streamlined and scalable approach, which circumvents the API server and additionally includes more robust error checking, log capture, and realtime job monitoring to provide improved data protection resilience. The disclosed approach employs a “sponsor” data agent outside the cluster and temporarily deploys a specialized backup resource within the cluster during a backup job, such as a lightweight Kubernetes File Client and/or an enhanced File System Data Agent, both of which present substantial performance and resiliency advantages over the API server.
1 . A computer-implemented method comprising:
by a first computing device that executes a sponsor data agent, wherein the first computing device comprises one or more hardware processors, and wherein the first computing device operates outside a Kubernetes cluster:
responsive to a notice of a backup job of primary data that is stored in a first persistent volume of the Kubernetes cluster,
using a first communicative connection with an Application Programming Interface server (API server) of the Kubernetes cluster to: (a) cause the Kubernetes cluster to create a secondary persistent volume that comprises primary data captured in a snapshot of the first persistent volume, (b) cause the Kubernetes cluster to instantiate a worker pod, which is attached to the secondary persistent volume and is configured with computer-executable instructions for a file client and with a cloned security certificate of the sponsor data agent, and (c) cause the worker pod to execute the file client, and
terminating the first communicative connection to the API server, based on determining that the worker pod is executing the file client;
by the file client executing at the worker pod in the Kubernetes cluster, wherein the worker pod executes on a second computing device that comprises one or more hardware processors:
using the cloned security certificate for a communication tunnel between the worker pod and the first computing device,
identifying, among the primary data in the secondary persistent volume, payload data and payload-related metadata that changed since a reference time, and
transmitting, to the sponsor data agent, the payload data and the payload-related metadata that changed, by way of a file backup pipeline within the communication tunnel;
by the sponsor data agent:
transmitting, to a media agent that executes on the first computing device, backup data resulting from converting the payload data and the payload-related metadata received from the file client into a format that is proprietary to a system comprising the sponsor data agent and the media agent, and
based on determining that all of the payload data and the payload-related metadata have been received from the file client, using a second communicative connection with the API server to: cause the Kubernetes cluster to tear down one or more of: the worker pod, the secondary persistent volume, and the snapshot; and
by the media agent:
generating one or more backup copies based on the backup data received from the sponsor data agent, and
storing the one or more backup copies at a secondary storage that is configured outside the Kubernetes cluster.
2 . The computer-implemented method of claim 1 , wherein the backup job is an incremental backup job, and wherein the notice of the backup job includes the reference time.
3 . The computer-implemented method of claim 1 , wherein establishing the communication tunnel is initiated at the worker pod.
4 . The computer-implemented method of claim 1 , wherein establishing the communication tunnel is initiated at the first computing device.
5 . The computer-implemented method of claim 1 , further comprising: by the file client executing at the worker pod in the Kubernetes cluster: transmitting statistics about the backup job, by way of a control pipeline within the communication tunnel, to the sponsor data agent.
6 . The computer-implemented method of claim 1 further comprising: by the file client executing at the worker pod in the Kubernetes cluster: transmitting logs corresponding to activities performed by the file client, by way of a control pipeline within the communication tunnel, to the sponsor data agent; and
by the sponsor data agent, storing the logs at one or more of: the first computing device and the secondary storage.
7 . The computer-implemented method of claim 1 further comprising: by the file client executing at the worker pod in the Kubernetes cluster: transmitting error information corresponding to activities performed by the file client, by way of a control pipeline within the communication tunnel, to the sponsor data agent; and
by the sponsor data agent, storing the error information at one or more of: the first computing device and the secondary storage.
8 . The computer-implemented method of claim 1 further comprising: by the sponsor data agent, before causing the Kubernetes cluster to create the secondary persistent volume, using the first communicative connection with the API server to: (i) discover a configuration file associated with a workload in the Kubernetes cluster, wherein the workload generated the primary data that is stored in the first persistent volume of the Kubernetes cluster, and (ii) cause the Kubernetes cluster to take the snapshot of the first persistent volume.
9 . The computer-implemented method of claim 1 further comprising: by the sponsor data agent, before causing the Kubernetes cluster to create the secondary persistent volume, using the first communicative connection with the API server to transfer, to the worker pod, one or more configuration files that are specific to the backup job, wherein the one or more configuration files include an identification of the sponsor data agent.
10 . The computer-implemented method of claim 1 further comprising: by the sponsor data agent, before causing the Kubernetes cluster to create the secondary persistent volume, using the first communicative connection with the API server to transfer, to the worker pod, one or more configuration files that are specific to the backup job, wherein the one or more configuration files include the cloned security certificate of the sponsor data agent.
11 . The computer-implemented method of claim 1 , wherein the system that comprises the sponsor data agent and the media agent further comprises a storage manager that executes on a third computing device, wherein the third computing device comprises one or more hardware processors and operates outside the Kubernetes cluster;
wherein the backup job is initiated by the storage manager, and wherein a first security certificate for the sponsor data agent is issued by the storage manager; and
wherein the cloned security certificate, which is a clone of the first security certificate, is generated by the sponsor data agent for use by the file client in the backup job.
12 . The computer-implemented method of claim 1 , wherein the backup job uses the communication tunnel, and not the first communicative connection with the API server, to transmit to the sponsor data agent the payload data and the payload-related metadata that changed.
13 . The computer-implemented method of claim 1 further comprising:
by the first computing device that executes the sponsor data agent:
receiving a notice of a second backup job of primary data that is stored in the first persistent volume of the Kubernetes cluster,
using a third communicative connection with the API server to: (i) cause the Kubernetes cluster to create an other secondary persistent volume that comprises primary data captured in a snapshot of the first persistent volume, (ii) cause the Kubernetes cluster to instantiate a worker pod, which is attached to the other secondary persistent volume and is configured with computer-executable instructions for a file system data agent and with a cloned security certificate corresponding to the Kubernetes cluster, and (iii) cause the worker pod to execute the file system data agent, and
terminating the third communicative connection to the API server, based on determining that the worker pod is executing the file system data agent for the second backup job.
14 . A system comprising:
a first computing device that operates outside a Kubernetes cluster, wherein the first computing device comprises one or more first hardware processors and non-transitory computer-readable media comprising computer-executable instructions, which, when executed by the one or more first hardware processors, cause the first computing device to:
receive a notice of a backup job of primary data that is stored in a first persistent volume of the Kubernetes cluster,
use a first communicative connection with an Application Programming Interface server (API server) of the Kubernetes cluster to: (a) cause the Kubernetes cluster to create a secondary persistent volume that comprises primary data captured in a snapshot of the first persistent volume, (b) cause the Kubernetes cluster to instantiate a worker pod, which is attached to the secondary persistent volume and is configured with computer-executable instructions for a file client and with a cloned security certificate of the first computing device, and (c) cause the worker pod to execute the file client, and
terminate the first communicative connection to the API server, based on determining that the worker pod is executing the file client;
wherein the worker pod executes on a second computing device that comprises one or more hardware processors, wherein the cloned security certificate is used for a communication tunnel between the worker pod and the first computing device, and wherein the file client executing at the worker pod is configured to:
identify, among the primary data in the secondary persistent volume, payload data and payload-related metadata that changed since a reference time, and
transmit the payload data and the payload-related metadata that changed, by way of a file backup pipeline within the communication tunnel, to the first computing device;
wherein the computer-executable instructions further cause the first computing device to:
based on determining that all of the payload data and the payload-related metadata have been received from the file client, use a second communicative connection with the API server to cause the Kubernetes cluster to tear down one or more of: the worker pod, the secondary persistent volume, and the snapshot, and
generate one or more backup copies that are based on the payload data and the payload-related metadata received from the file client, and
store the one or more backup copies at a secondary storage that is configured outside the Kubernetes cluster.
15 . The system of claim 14 , wherein the backup job is an incremental backup job, and wherein the notice of the backup job includes the reference time.
16 . The system of claim 14 further comprising: by the first computing device, before causing the Kubernetes cluster to create the secondary persistent volume, using the first communicative connection with the API server to: (i) discover a configuration file associated with a workload in the Kubernetes cluster, wherein the workload generates the primary data that is stored in the first persistent volume of the Kubernetes cluster, and (ii) cause the Kubernetes cluster to take the snapshot of the first persistent volume.
17 . The system of claim 14 further comprising: by the first computing device, before causing the Kubernetes cluster to create the secondary persistent volume, using the first communicative connection with the API server to transfer, to the worker pod, one or more configuration files that are specific to the backup job, wherein the one or more configuration files include an identification of the first computing device.
18 . The system of claim 14 further comprising: by the first computing device, before causing the Kubernetes cluster to create the secondary persistent volume, using the first communicative connection with the API server to transfer, to the worker pod, one or more configuration files that are specific to the backup job, wherein the one or more configuration files include the cloned security certificate of the first computing device.
19 . The system of claim 14 , further comprising: a third computing device that comprises one or more hardware processors and operates outside the Kubernetes cluster;
wherein the backup job is initiated by the third computing device;
wherein a first security certificate for the first computing device is issued by the third computing device; and
wherein the cloned security certificate, which is a clone of the first security certificate, is generated by the first computing device for use by the file client in the backup job.
20 . The system of claim 14 , wherein the backup job uses the communication tunnel, and not the first communicative connection with the API server, to transmit to the first computing device the payload data and the payload-related metadata that changed.