Migrating stateful workloads between container clusters with different storage backends
Techniques for migrating a stateful workload between a source container cluster and a destination container cluster that have different storage backends are provided. In one set of embodiments, these techniques involve creating and running, at the time of the migration, transfer processes in the source and destination container clusters respectively, where (1) the source-side transfer process mounts the workload's persistent volumes (referred to as source persistent volumes) in the source container cluster, and (2) the destination-side transfer process mounts destination persistent volumes that have been dynamically provisioned in the destination container cluster in accordance with the specifications of the source persistent volumes. Upon being created and run, the source-side and destination-side transfer processes can work in concert to securely copy the data in the source persistent volumes to their corresponding destination persistent volumes, thereby transferring that data between the storage backends of the source and destination container clusters.
1 . A method for migrating a stateful workload from a source container cluster to a destination container cluster, wherein the source container cluster is connected to a first storage backend, wherein the destination container cluster is connected to a second storage backend distinct from the first storage backend, and wherein the stateful workload makes use of a source persistent volume in the source container cluster that maps to a storage region or volume in the first storage backend, the method comprising:
provisioning, by a migration process in communication with the source and destination container clusters, a destination persistent volume in the destination container cluster that maps to a storage region or volume in the second storage backend, the destination persistent volume being based on the source persistent volume;
launching, by the migration process, a destination-side transfer process in the destination container cluster; and
launching, by the migration process, a source-side transfer process in the source container cluster,
wherein upon being launched, the source-side and destination-side transfer processes synchronize, over a secure network protocol, data contents of the source persistent volume to the destination persistent volume;
wherein prior to launching the source-side and destination-side transfer processes, the method further comprising:
generating, by the migration process, security metadata for enabling secure network communication between the source and destination container clusters; and
transmitting, by the migration process, at least a portion of the security metadata to the source and destination container clusters respectively:
wherein upon being launched, the destination-side transfer process:
mounts the destination persistent volume;
mounts the portion of the security metadata transmitted by the migration process to the destination container cluster;
exposes a network port from communicating with the source-side transfer process via the secure network protocol; and
waits for a connection from the source-side transfer process via the secure network protocol:
wherein upon being launched, the source-side transfer process:
mounts the source persistent volume;
mounts the portion of the security metadata transmitted by the migration process to the source container cluster:
connects, using the mounted security metadata, to the destination-side transfer process via the secure network protocol; and
repeatedly synchronizes the data contents of the source persistent volume to the destination persistent volume until an amount of time needed to synchronize a remaining difference in data between the source and destination persistent volumes falls below a threshold:
wherein, upon completion of execution of the source-side transfer process:
the method further comprising:
deleting the source-side transfer process;
removing, from one or more external Domain Name Service (DNS) servers, DNS records for forwarding workload traffic to the source container cluster;
shutting down the workload; and
launching another source-side transfer process in the source container cluster.
2 . The method of claim 1 wherein the provisioning comprises:
determining, from a specification of a persistent volume claim (PVC) bound to the source persistent volume in the source container cluster, a capacity and an access mode of the source persistent volume; and
provisioning the destination persistent volume to have the determined capacity and the determined access mode.
3 . The method of claim 1 wherein said another source-side transfer process synchronizes the remaining difference in data from the source persistent volume to the destination persistent volume.
4 . A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code embodying a method for migrating a stateful workload from a source container cluster to a destination container cluster, wherein the source container cluster is connected to a first storage backend, wherein the destination container cluster is connected to a second storage backend distinct from the first storage backend, and wherein the stateful workload makes use of a source persistent volume in the source container cluster that maps to a storage region or volume in the first storage backend, the method comprising:
provisioning a destination persistent volume in the destination container cluster that maps to a storage region or volume in the second storage backend, the destination persistent volume being based on the source persistent volume;
launching a destination-side transfer process in the destination container cluster; and
launching a source-side transfer process in the source container cluster,
wherein upon being launched, the source-side and destination-side transfer processes synchronize, over a secure network protocol, data contents of the source persistent volume to the destination persistent volume;
wherein the method further comprises, prior to launching the source-side and destination-side transfer processes:
generating security metadata for enabling secure network communication between the source and destination container clusters; and
transmitting at least a portion of the security metadata to the source and destination container clusters respectively:
wherein upon being launched, the destination-side transfer process:
mounts the destination persistent volume;
mounts the portion of the security metadata transmitted to the destination container cluster:
exposes a network port from communicating with the source-side transfer process via the secure network protocol; and
waits for a connection from the source-side transfer process via the secure network protocol;
wherein upon being launched. the source-side transfer process:
mounts the source persistent volume;
mounts the portion of the security metadata transmitted to the source container cluster:
connects, using the mounted security metadata, to the destination-side transfer process via the secure network protocol; and
repeatedly synchronizes the data contents of the source persistent volume to the destination persistent volume until an amount of time needed to synchronize a remaining difference in data between the source and destination persistent volumes falls below a threshold:
wherein the method further comprises, upon completion of execution of the source-side transfer process:
deleting the source-side transfer process;
removing, from one or more external Domain Name Service (DNS) servers, DNS records for forwarding workload traffic to the source container cluster;
shutting down the workload; and
launching another source-side transfer process in the source container cluster.
5 . The non-transitory computer readable storage medium of claim 4 wherein the provisioning comprises:
determining, from a specification of a persistent volume claim (PVC) bound to the source persistent volume in the source container cluster, a capacity and an access mode of the source persistent volume; and
provisioning the destination persistent volume to have the determined capacity and the determined access mode.
6 . The non-transitory computer readable storage medium of claim 4 wherein said another source-side transfer process synchronizes the remaining difference in data from the source persistent volume to the destination persistent volume.
7 . A computer system comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that causes the processor to migrate a stateful workload from a source container cluster to a destination container cluster, wherein the source container cluster is connected to a first storage backend, wherein the destination container cluster is connected to a second storage backend distinct from the first storage backend, and wherein the stateful workload makes use of a source persistent volume in the source container cluster that maps to a storage region or volume in the first storage backend, the migrating comprising:
provisioning a destination persistent volume in the destination container cluster that maps to a storage region or volume in the second storage backend, the destination persistent volume being based on the source persistent volume;
launching a destination-side transfer process in the destination container cluster; and
launching a source-side transfer process in the source container cluster,
wherein upon being launched, the source-side and destination-side transfer processes synchronize, over a secure network protocol, data contents of the source persistent volume to the destination persistent volume;
wherein the migrating further comprises, prior to launching the source-side and destination-side transfer processes:
generating security metadata for enabling secure network communication between the source and destination container clusters; and
transmitting at least a portion of the security metadata to the source and destination container clusters respectively:
wherein upon being launched, the destination-side transfer process:
mounts the destination persistent volume;
mounts the portion of the security metadata transmitted to the destination container cluster:
exposes a network port from communicating with the source-side transfer process via the secure network protocol; and
waits for a connection from the source-side transfer process via the secure network protocol;
wherein upon being launched, the source-side transfer process:
mounts the source persistent volume;
mounts the portion of the security metadata transmitted to the source container cluster:
connects, using the mounted security metadata, to the destination-side transfer process via the secure network protocol; and
repeatedly synchronizes the data contents of the source persistent volume to the destination persistent volume until an amount of time needed to synchronize a remaining difference in data between the source and destination persistent volumes falls below a threshold;
wherein the migrating further comprises, upon completion of execution of the source-side transfer process:
deleting the source-side transfer process:
removing. from one or more external Domain Name Service (DNS) servers, DNS records for forwarding workload traffic to the source container cluster;
shutting down the workload; and
launching another source-side transfer process in the source container cluster.
8 . The computer system of claim 7 wherein the provisioning comprises:
determining, from a specification of a persistent volume claim (PVC) bound to the source persistent volume in the source container cluster, a capacity and an access mode of the source persistent volume; and
provisioning the destination persistent volume to have the determined capacity and the determined access mode.
9 . The computer system of claim 7 wherein said another source-side transfer process synchronizes the remaining difference in data from the source persistent volume to the destination persistent volume.