Dynamic data migration for containerized environments
An illustrative method includes a storage management system accessing workload metrics data associated with a containerized runtime environment, accessing, storage metrics data associated with the first backend storage system and the second backend storage system, evaluating the workload metrics data and the storage metrics data, and directing, based on the evaluation, the first backend storage system to migrate one or more datasets stored at a first backend storage system to a second backend storage system. In some embodiments, the evaluation includes providing the workload metrics data and the storage metrics data as input to one or more machine learning models.
1 . A method comprising:
accessing, by a storage management system configured to provide persistent storage to containerized applications deployed at a containerized runtime environment using a first backend storage system and a second backend storage system, workload metrics data associated with the containerized runtime environment;
accessing, by the storage management system, storage metrics data associated with the first backend storage system and the second backend storage system;
evaluating, by the storage management system, the workload metrics data and the storage metrics data; and
directing, by the storage management system based on the evaluation, the first backend storage system to migrate one or more datasets stored at the first backend storage system to the second backend storage system.
2 . The method of claim 1 , wherein the evaluating the metrics data comprises providing the workload metrics data and the storage metrics data as input to one or more machine learning models; and
wherein the directing the first backend storage system to migrate the one or more datasets is further based on one or more outputs of the one or more machine learning models.
3 . The method of claim 2 , wherein the one or more machine learning models comprise a reinforcement learning model.
4 . The method of claim 2 , further comprising training, by the storage management system, the one or more machine learning models using past workload metrics data associated with previous deployments of the containerized applications and past storage metrics data associated with the first backend storage system and the second backend storage system.
5 . The method of claim 2 , wherein the one or more machine learning models comprise a workload machine learning model configured to output one or more workload predictions associated with the containerized runtime environment and a storage machine learning model configured to output one or more scores for the first backend storage system and the second backend storage system.
6 . The method of claim 5 , wherein the migrating of the datasets is further based on the workload predictions indicating increased workload activity associated with the containerized runtime environment or the scores indicating that the second backend storage system is more suitable than the first backend storage system for providing the persistent storage.
7 . The method of claim 1 , wherein the workload metrics data comprises one or more of a network throughput rate, a network traffic rate, a processor usage rate, a memory usage rate, or a number of active network connections.
8 . The method of claim 1 , wherein the storage metrics data comprises one or more of an I/O latency rate, an IOPS rate, an I/O bandwidth rate, a storage capacity level, or an error rate.
9 . The method of claim 1 , wherein each of the one or more datasets comprises a plurality of replicas stored at the first backend storage system; and
wherein the directing the first backend storage system to migrate the one or more datasets comprises directing the first backend storage system to incrementally migrate each replica for each of the one or more datasets while continuing data operations on the one or more datasets.
10 . The method of claim 1 , wherein the storage metrics data is generated by the first backend storage system, and wherein the workload metrics data is generated by a kubelet component deployed at the containerized runtime environment.
11 . The method of claim 1 , further comprising receiving, by the storage management system using an application programming interface (API), the workload metrics data from a distributed database deployed at the containerized runtime environment.
12 . The method of claim 1 , further comprising:
accessing, by the storage management system, additional storage metrics data and additional workload metrics data, the additional storage metrics data and the additional workload metrics data generated after the migrating of the datasets;
evaluating, by the storage management system, the additional workload metrics data and the additional storage metrics data; and
directing, by the storage management system based on the evaluation of the additional workload metrics data and the additional storage metrics data, the second backend storage system to migrate the datasets stored at the second backend storage system to a third backend storage system.
13 . A system comprising:
a memory storing instructions; and
one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising:
accessing workload metrics data associated with a containerized runtime environment, wherein a storage management system is configured to provide persistent storage to containerized applications deployed at the containerized runtime environment using a first backend storage system and a second backend storage system;
accessing storage metrics data associated with the first backend storage system and the second backend storage system;
evaluating the workload metrics data and the storage metrics data; and
directing, based on the evaluation, the first backend storage system to migrate one or more datasets stored at the first backend storage system to the second backend storage system.
14 . The system of claim 13 , wherein the evaluating the metrics data comprises providing the workload metrics data and the storage metrics data as input to one or more machine learning models; and
wherein the directing the first backend storage system to migrate the one or more datasets is further based on one or more outputs of the one or more machine learning models.
15 . The system of claim 14 , wherein the one or more machine learning models comprise a workload machine learning model configured to output one or more workload predictions associated with the containerized runtime environment and a storage machine learning model configured to output one or more scores for the first backend storage system and the second backend storage system.
16 . The system of claim 15 , wherein the migrating of the datasets is further based on the workload predictions indicating increased workload activity associated with the containerized runtime environment or the scores indicating that the second backend storage system is more suitable than the first backend storage system for providing the persistent storage.
17 . The system of claim 13 , wherein the workload metrics data comprises one or more of a network throughput level, a network traffic level, a processor usage level, a memory usage level, a number of active network connections, or an I/O activity level.
18 . The system of claim 13 , wherein the storage metrics data comprises one or more of an I/O latency level, an IOPS level, an I/O throughout level, a storage capacity level, or an error rate level.
19 . The system of claim 13 , wherein each of the one or more datasets comprises a plurality of replicas stored at the first backend storage system; and
wherein the directing the first backend storage system to migrate the one or more datasets comprises directing the first backend storage system to incrementally migrate each replica for each of the one or more datasets while continuing data operations on the one or more datasets.
20 . A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
accessing workload metrics data associated with a containerized runtime environment, wherein a storage management system is configured to provide persistent storage to containerized applications deployed at the containerized runtime environment using a first backend storage system and a second backend storage system;
accessing storage metrics data associated with the first backend storage system and the second backend storage system;
evaluating the workload metrics data and the storage metrics data; and
directing, based on the evaluation, the first backend storage system to migrate one or more datasets stored at the first backend storage system to the second backend storage system.