Systems and methods for tuning containers in a high availability environment
The disclosed technology teaches tuning containers in pods, in a high availability environment that runs two or more pods to implement a service, while the containers are running, including instrumenting a tunable container to communicate application metrics to a tuning engine that runs in a pod and maintains current configuration state information for the container. The tuning engine accesses the application metrics and a rule that specifies tuning of resource configuration for the container. The rule combines variables in the metrics to determine whether an update should be applied to the container. The tuning engine determines a new resource configuration for the tunable container and updates the configuration state information for the container according to the new configuration. An update manager detects updating of the configuration state information and performs a rolling update of instances of the tunable container in the pods that implement the service.
1. A computer-implemented method for tuning containers in pods, in a high availability environment that runs two or more pods to implement a service implemented on a cloud-based computing cluster, while the containers are running, including:
a tuning engine accessing application metrics from a tunable container and a rule that specifies tuning of a resource configuration for the tunable container, sometimes increasing and sometimes decreasing resources in the resource configuration, wherein the rule combines a plurality of variables from the application metrics to determine whether a tuning update should be applied to the tunable container;
the turning engine, responsive to applying the rule, updating the resource configuration for the tunable container to produce an updated resource configuration that includes updates to one or more of a Java virtual machine (abbreviated JVM) setting, a cache size, a number of active sessions, a number of concurrent connections, a number of work threads in a pool, and a number of open files allowed; and
an update manager applying the updated resource configuration by performing a rolling update of instances of the tunable container in the pods, while the service is running.
2. The computer-implemented method of claim 1 , further including the update manager deploying updated tunable container instances from the updated resource configuration, in pods during the rolling update of instances.
3. The computer-implemented method of claim 1 , further including the update manager, during the rolling update of instances of the tunable container:
continuing to run a configurable minimum number of instances of the tunable container in the pods, and
deleting legacy pods that were instantiated from outdated resource configuration.
4. The computer-implemented method of claim 1 , wherein the resource configuration for the tunable container is expressed in a yet another markup language (abbreviated YAML) file that describes a related set of resources in the high availability environment.
5. The computer-implemented method of claim 1 , wherein the tuning update includes adding an index associated with a field whose count of field lookups occurring during a configurable time period is greater than a configurable percent of a total count of field lookups during the configurable time period.
6. The computer-implemented method of claim 1 , wherein the tuning update includes deleting an index associated with a field that has an index and whose count of field lookups occurring during a configurable time period is less than a configurable percent of a total count of field lookups during the configurable time period.
7. The computer-implemented method of claim 1 , further including the tuning engine accessing the application metrics from which phases of operation can be determined and a rule that, based on the determined phase of operation, resets resource configuration of one or more of a Java virtual machine (abbreviated JVM) setting, a cache size, a number of active sessions, a number of concurrent connections, a number of work threads in a pool, and a number of open files allowed.
8. The computer-implemented method of claim 1 , wherein the tuning engine uses persistent storage for maintaining current configuration state information for the tunable container to avoid loss of state information.
9. A tangible non-transitory computer readable storage media, including program instructions loaded into memory that, when executed on processors cause the processors to implement a method for tuning containers in pods, in a high availability environment that runs two or more pods to implement a server implemented on a cloud-based computing cluster, while the containers are running, the method including:
a tuning engine accessing application metrics from a tunable container and a rule that specifies tuning of a resource configuration for the tunable container, sometimes increasing and sometimes decreasing resources in the resource configuration, wherein the rule combines a plurality of variables from the application metrics to determine whether a tuning update should be applied to the tunable container;
the turning engine, responsive to applying the rule, updating the resource configuration for the tunable container to produce an updated resource configuration that includes updates to one or more of a Java virtual machine (abbreviated JVM) setting, a cache size, a number of active sessions, a number of concurrent connections, a number of work threads in a pool, and a number of open files allowed; and
an update manager applying the updated resource configuration by performing a rolling update of instances of the tunable container in the pods, while the service is running.
10. The tangible non-transitory computer readable storage media of claim 9 , further including the update manager deploying updated tunable container instances from the updated resource configuration, in pods during the rolling update of instances.
11. The tangible non-transitory computer readable storage media of claim 9 , further including the update manager, during the rolling update of instances of the tunable container:
continuing to run a configurable minimum number of instances of the tunable container in the pods, and
deleting legacy pods that were instantiated from outdated resource configuration.
12. The tangible non-transitory computer readable storage media of claim 9 , wherein the resource configuration for the tunable container is expressed in a yet another markup language (abbreviated YAML) file that describes a related set of resources in the high availability environment.
13. The tangible non-transitory computer readable storage media of claim 9 , wherein the tuning update includes adding an index associated with a field whose count of field lookups occurring during a configurable time period is greater than a configurable percent of a total count of field lookups during the configurable time period.
14. The tangible non-transitory computer readable storage media of claim 9 , wherein the tuning update includes deleting an index associated with a field that has an index and whose count of field lookups occurring during a configurable time period is less than a configurable percent of a total count of field lookups during the configurable time period.
15. The tangible non-transitory computer readable storage media of claim 9 , further including the tuning engine accessing the application metrics from which phases of operation can be determined and a rule that, based on the determined phase of operation, resets resource configuration of one or more of a Java virtual machine (abbreviated JVM) setting, a cache size, a number of active sessions, a number of concurrent connections, a number of work threads in a pool, and a number of open files allowed.
16. The tangible non-transitory computer readable storage media of claim 9 , wherein the tuning engine uses persistent storage for maintaining current configuration state information for the tunable container to avoid loss of state information.
17. A system for tuning containers in pods, in a high availability environment that runs two or more pods to implement a service implemented on a cloud-based computing cluster, while the containers are running, the system including a processor coupled to the non-transitory computer readable storage media and computer instructions of claim 9 .
18. The system of claim 17 , wherein the tuning update includes adding an index associated with a field whose count of field lookups occurring during a configurable time period is greater than a configurable percent of a total count of field lookups during the configurable time period.
19. The system of claim 17 , wherein the tuning update includes deleting an index associated with a field that has an index and whose count of field lookups occurring during a configurable time period is less than a configurable percent of a total count of field lookups during the configurable time period.
20. The system of claim 17 , further including the tuning engine accessing the application metrics from which phases of operation can be determined and a rule that, based on the determined phase of operation, resets resource configuration of one or more of a Java virtual machine (abbreviated JVM) setting, a cache size, a number of active sessions, a number of concurrent connections, a number of work threads in a pool, and a number of open files allowed.