Systems and methods for tuning containers in a high availability environment
The disclosed technology teaches tuning containers in pods, in a high availability environment that runs two or more pods to implement a service, while the containers are running, including instrumenting a tunable container to communicate application metrics to a tuning engine that runs in a pod and maintains current configuration state information for the container. The tuning engine accesses the application metrics and a rule that specifies tuning of resource configuration for the container. The rule combines variables in the metrics to determine whether an update should be applied to the container. The tuning engine determines a new resource configuration for the tunable container and updates the configuration state information for the container according to the new configuration. An update manager detects updating of the configuration state information and performs a rolling update of instances of the tunable container in the pods that implement the service.
1. A computer-implemented method of tuning of containers in pods, in a high availability environment that runs two or more pods to implement a particular service, while the containers are running, including:
instrumenting a tunable container to communicate application metrics to a tuning engine that runs in a pod and maintains current configuration state information for the tunable container;
the tuning engine accessing the application metrics and a rule that specifies tuning of resource configuration for the tunable container, wherein the rule combines a plurality of variables in the application metrics to determine whether a tuning update should be applied to the tunable container;
the tuning engine, responsive to applying the rule, determining a new resource configuration for the tunable container and updating the configuration state information for the tunable container according to the new resource configuration;
wherein the new resource configuration updates one or more of a cache size, number of concurrent connections, number of work threads in a pool, and number of open files allowed, for the tunable container; and
an update manager detecting updating of the configuration state information and performing a rolling update of instances of the tunable container in the pods, while the particular service is running.
2. The computer-implemented method of claim 1 , wherein the instances of the tunable container include a deployment running in containers.
3. The computer-implemented method of claim 1 , wherein the current configuration state information for the tunable container is expressed in a yet another markup language (abbreviated YAML) file that describes a related set of resources in the high availability environment.
4. The computer-implemented method of claim 1 , wherein the current configuration state information for the tunable container is expressed in a deployment resource bundle with at least one file that describes a related set of resources for pods in the high availability environment.
5. The computer-implemented method of claim 1 , further including the update manager deploying updated tunable container instances from the updated configuration state information, in pods during the rolling update of instances.
6. The computer-implemented method of claim 1 , further including the update manager, during the rolling update of instances of the tunable container:
continuing to run a configurable minimum number of instances of the tunable container in the pods, and
deleting legacy pods that were instantiated from outdated configuration state information.
7. The computer-implemented method of claim 1 , wherein the tuning update includes adding an index associated with a field whose count of field lookups occurring during a configurable time period is greater than a configurable percent of a total count of field lookups during the configurable time period.
8. The computer-implemented method of claim 1 , wherein the tuning update includes deleting an index associated with a field that has an index and whose count of field lookups occurring during a configurable time period is less than a configurable percent of a total count of field lookups during the configurable time period.
9. The computer-implemented method of claim 1 , further including the tuning engine accessing the application metrics from which phases of operation can be determined and a rule that, based on the determined phase of operation, resets resource configuration of one or more of a cache size, number of concurrent connections, number of work threads in a pool, and number of open files allowed, for the tunable container.
10. The computer-implemented method of claim 1 , wherein the tuning engine uses persistent storage for maintaining current configuration state information for the tunable container to avoid loss of state information.
11. A tangible non-transitory computer readable storage media, including program instructions loaded into memory that, when executed on processors cause the processors to implement a method of tuning of containers in pods, in a high availability environment that runs two or more pods to implement a particular service, while the containers are running, the method including:
instrumenting a tunable container to communicate application metrics to a tuning engine that runs in a pod and maintains current configuration state information for the tunable container;
the tuning engine accessing the application metrics and a rule that specifies tuning of resource configuration for the tunable container, wherein the rule combines a plurality of variables in the application metrics to determine whether a tuning update should be applied to the tunable container;
the tuning engine, responsive to applying the rule, determining a new resource configuration for the tunable container and updating the configuration state information for the tunable container according to the new resource configuration;
wherein the new resource configuration updates one or more of a cache size, number of concurrent connections, number of work threads in a pool, and number of open files allowed, for the tunable container; and
an update manager detecting updating of the configuration state information and performing a rolling update of instances of the tunable container in the pods, while the particular service is running.
12. The tangible non-transitory computer readable storage media of claim 11 , wherein the instances of the tunable container include a deployment running in containers.
13. The tangible non-transitory computer readable storage media of claim 11 , wherein the current configuration state information for the tunable container is expressed in a yet another markup language (abbreviated YAML) file that describes a related set of resources in the high availability environment.
14. The tangible non-transitory computer readable storage media of claim 11 , wherein the current configuration state information for the tunable container is expressed in a deployment resource bundle with at least one file that describes a related set of resources for pods in the high availability environment.
15. The tangible non-transitory computer readable storage media of claim 11 , further including the update manager, during the rolling update of instances of the tunable container:
deploying updated tunable container instances from the updated configuration state information,
continuing to run a configurable minimum number of instances of the tunable container in the pods, and
deleting legacy pods that were instantiated from outdated configuration state information.
16. The tangible non-transitory computer readable storage media of claim 11 , wherein the tuning update includes adding an index associated with a field whose count of field lookups occurring during a configurable time period is greater than a configurable percent of a total count of field lookups during the configurable time period.
17. The tangible non-transitory computer readable storage media of claim 11 , wherein the tuning update includes deleting an index associated with a field that has an index and whose count of field lookups occurring during a configurable time period is less than a configurable percent of a total count of field lookups during the configurable time period.
18. A system for tuning containers in pods, in a high availability environment that runs two or more pods to implement a service, while the containers are running, the system including a processor coupled to the non-transitory computer readable storage media and computer instructions of claim 11 .
19. The system of claim 18 , further including computer instructions implementing the update manager, during the rolling update of instances of the tunable container:
deploying updated tunable container instances from the updated configuration state information,
continuing to run a configurable minimum number of instances of the tunable container in the pods, and
deleting legacy pods that were instantiated from outdated configuration state information.
20. The system of claim 18 , wherein the tuning update includes adding an index associated with a field that does not have an index, whose count of field lookups occurring during a configurable time period is greater than a configurable percent of a total count of field lookups during the configurable time period.