Coupled compute and storage resource autoscaling
Techniques are provided for coupled compute and storage resource autoscaling. Applications may be hosted within an application hosting environment (e.g., containerized applications hosted within Kubernetes) that allocations certain amounts of compute resources (e.g., processor and memory resources) to the applications for execution. The applications may store data within persistent of a backend storage platform separate from the application hosting environment. An autoscaler monitors the health of storage resources allocated to an application during deployment and runtime of the application so that issues with the storage resources can be preemptively identified and resolved for non-disruptive operation of the application. The autoscaler scales both the compute resources and the storage resources assigned to the application so that the application can continue operation in a non-disruptive manner.
1 . A system, comprising:
a compute region interface providing access to a compute allocator of a container orchestration platform hosted within a compute region separate from both a first storage region and a second storage region;
a storage region interface providing access to storage application programming interfaces (APIs) of the first storage region and the second storage region; and
an autoscaler providing coupled compute and storage scaling for a containerized application within the container orchestration platform by:
determining, based upon a task assigned to the containerized application, a determined amount of compute resources to allocate from the compute region to the containerized application within the container orchestration platform;
calculating, based upon the task and using the determined amount of compute resources, a calculated amount of storage resources to allocate to the containerized application;
triggering, using the storage region interface, a verification that the first storage region has first operational storage to allocate as the calculated amount of storage resources;
in response to the verification succeeding, executing a coupled compute and storage autoscale operation to control the compute allocator to allocate the determined amount of compute resources and the storage region interface to allocate, using the storage APIs, the calculated amount of storage resources from the first storage region for the containerized application;
in response to the verification failing, performing a self-healing operation to repair storage resources of the first storage region; and
in response to the self-healing operation failing to repair the storage resources:
evaluating the second storage region to identify second operational storage to allocate as the calculated amount of storage resources; and
executing the coupled compute and storage autoscale operation to control the compute allocator to allocate the determined amount of compute resources and the storage region interface to allocate, using the storage APIs, the calculated amount of storage resources from the second storage region for the containerized application.
2 . The system of claim 1 , wherein the autoscaler:
utilizes the storage region interface to perform a health verification to determine whether first storage of the first storage region is operational for allocation as the first operational storage to the containerized application;
in response to the health verification indicating that the first storage is not operational, utilizes the storage region interface to identify a backend storage platform of the second storage region with second storage that is operational for allocation; and
in response to verifying that the second storage is operational and accessible to the compute region, triggers execution of the coupled compute and storage autoscale operation to:
utilize the compute region interface to instruct the compute allocator to allocate the determined amount of compute resources from the compute region to the containerized application; and
trigger the storage region interface to instruct the backend storage platform to allocate the calculated amount of storage resources to the containerized application using the second storage.
3 . The system of claim 1 , wherein the calculated amount of storage resources are allocated from the first storage region hosted by a first backend storage platform, and wherein the autoscaler:
utilizes the storage region interface to transmit a health checkup request, during execution of the containerized application, to the first backend storage platform to identify a health status of the first operational storage; and
in response to the health status indicating a warning for the first operational storage, triggers execution of the coupled compute and storage autoscale operation for the containerized application by allocating the calculated amount of storage resources from the second operational storage to the containerized application, wherein the calculated amount of storage resources from the first storage region are deallocated from the containerized application.
4 . The system of claim 1 , wherein the calculated amount of storage resources are allocated from the first storage region hosted by a first backend storage platform within a first cloud storage region, and wherein the autoscaler:
in response to determining that the first backend storage platform of the first cloud storage region is experience a failure where the containerized application cannot access the first storage region, triggers execution of the coupled compute and storage autoscale operation using the second storage region hosted by a second backend storage platform within a second cloud storage region.
5 . The system of claim 1 , wherein the calculated amount of storage resources are allocated from the first storage region hosted by a first backend storage platform, and wherein the autoscaler:
monitors progress of the task being executed by the containerized application and communication between the containerized application and the first storage region; and
in response to identifying at least one of a failure of the task or a communication issue where the containerized application is unable to access the first storage region, triggers execution of the coupled compute and storage autoscale operation to allocate the calculated amount of storage resources from the second operational storage to the containerized application.
6 . The system of claim 1 , wherein the autoscaler:
accesses security credentials associated with the containerized application and the container orchestration platform;
utilizes the security credentials to authenticate with the container orchestration platform for executing the coupled compute and storage autoscale operation to instruct the compute allocator to allocate the determined amount of compute resources from the compute region to the containerized application; and
utilizes the security credentials to authenticate with a backend storage platform to instruct the backend storage platform to allocate the calculated amount of storage resources from the first region for access by the containerized application.
7 . The system of claim 1 , wherein the autoscaler:
executes the coupled compute and storage autoscale operation to instruct a backend storage platform to add one or more storage nodes to the first storage region according to a random assignment for allocating the calculated amount of storage resources to the containerized application, wherein the one or more storage nodes are hosted within a cloud computing environment implementing the backend storage platform; and
executes the coupled compute and storage autoscale operation to instruct the compute allocator to assign one or more compute nodes as the determined amount of compute resources allocated to the containerized application.
8 . The system of claim 1 , wherein the autoscaler:
evaluates a request, for the containerized application to perform the task, to determine the determined amount of compute resources to be allocated to the containerized application for performing the task.
9 . The system of claim 1 , wherein the autoscaler:
evaluates a request, for the containerized application to perform the task, to determine the determined amount of compute resources to be allocated to the containerized application for performing the task; and
calculates the calculated amount of storage resources to allocate to the containerized application based upon an amount of storage that the task will utilize.
10 . The system of claim 1 , wherein the autoscaler:
evaluates a backup policy, to be implemented by the containerized application, to identify a frequency of creating backups; and
determines the determined amount of compute resources and calculates the calculated amount of storage resources to allocate for creating and storing the backups according to the frequency specified by the backup policy.
11 . The system of claim 1 , wherein the autoscaler:
evaluates a restore operation, to be implemented by the containerized application for restoring data using a snapshot, to determine the amount of determined compute resources and calculate the calculated amount of storage resources to be allocated to the containerized application for implementing the restore operation.
12 . The system of claim 1 , wherein the autoscaler:
calculates the amount of calculated storage resources based upon an amount of storage that can be utilized by the determined amount of compute resources.
13 . The system of claim 2 , wherein the autoscaler:
in response to identifying a version mismatch between a version of the container orchestration platform and a version of a first backend storage platform hosting the first storage region, triggers execution of the coupled compute and storage autoscale operation to autoscale the calculated amount of storage resources from the second storage region hosted by a second backend storage platform matching the version of the container orchestration platform.
14 . The system of claim 1 , wherein the autoscaler:
scales the compute resources and the storage resources together during deployment and before operation of the containerized application, wherein the calculated amount of storage resources is calculated based upon an amount of inoperable storage detected by the verification.
15 . The system of claim 14 , wherein the autoscaler:
calculates the calculated amount of storage resources to allocate to the containerized application based upon an amount of storage that the compute resources will utilize to store data associated with the task assigned to the containerized application to perform.
16 . The system of claim 1 , wherein the autoscaler:
evaluates a backup policy to be implemented by the containerized application to identify an amount of data to backup; and
determines the determined amount of compute resources and calculates the calculated amount of storage resources to allocate for creating and storing backups that include the amount of data identified from the backup policy.
17 . A method, comprising:
determining, based upon a task assigned to a containerized application, a determined amount of compute resources to allocate from a compute region to the containerized application within a container orchestration platform, wherein the compute region is hosted separate from both a first storage region and a second storage region;
calculating, based upon the task and using the determined amount of compute resources, a calculated amount of storage resources to allocate to the containerized application;
triggering, using a storage region interface, a verification that the first storage region has first operational storage to allocate as the calculated amount of storage resources;
in response to the verification succeeding, executing a coupled compute and storage autoscale operation to control a compute allocator to allocate the determined amount of compute resources and the storage region interface to allocate, using a first storage application programming interface (API) of the first storage region, the calculated amount of storage resources from the first storage region for the containerized application;
in response to the verification failing, performing a self-healing operation to repair storage resources of the first storage region; and
in response to the self-healing operation failing to repair the storage resources: evaluating the second storage region to identify second operational storage to allocate as the calculated amount of storage resources; and
executing the coupled compute and storage autoscale operation to control the compute allocator to allocate the determined amount of compute resources and the storage region interface to allocate, using a second storage API of the second storage region, the calculated amount of storage resources from the second storage region for the containerized application.
18 . The method of claim 17 , comprising:
evaluating a configuration file for the containerized application to identify determine the determined amount of compute resources to allocate to the containerized application; and
evaluating a workload to be performed by the containerized application using the determined amount of compute resources in order to calculate the calculated amount of storage resources to allocate to the containerized application.
19 . A non-transitory machine readable medium comprising instructions, which when executed by a machine, causes the machine to:
determine, based upon a task assigned to a containerized application, a determined amount of compute resources to allocate from a compute region to the containerized application within a container orchestration platform, wherein the compute region is hosted separate from both a first storage region and a second storage region;
calculate, based upon the task and using the determined amount of compute resources, a calculated amount of storage resources to allocate to the containerized application;
trigger, using a storage region interface, a verification that the first storage region has first operational storage to allocate as the calculated amount of storage resources;
in response to the verification succeeding, execute a coupled compute and storage autoscale operation to control a compute allocator to allocate the determined amount of compute resources and the storage region interface to allocate, using a first storage application programming interface (API) of the first storage region, the calculated amount of storage resources from the first storage region for the containerized application;
in response to the verification failing, perform a self-healing operation to repair storage resources of the first storage region; and
in response to the self-healing operation failing to repair the storage resources:
evaluate the second storage region to identify second operational storage to allocate as the calculated amount of storage resources; and
execute the coupled compute and storage autoscale operation to control the compute allocator to allocate the determined amount of compute resources and the storage region interface to allocate, using a second storage API of the second storage region, the calculated amount of storage resources from the second storage region for the containerized application.
20 . The non-transitory machine readable medium of claim 19 , wherein the instructions cause the machine to:
notify a load balancer of one or more storage nodes added to a backend storage platform for hosting the calculated amount of storage resources allocated to the containerized application, wherein the load balancer routes storage access requests from the containerized application to the one or more storage nodes.