Readiness probe for management applications
A process for providing a framework for responding to readiness probes is described herein. In embodiments, a virtual machine communicates with one or more external dependencies and generates metrics data which is stored in a cache. In embodiments, an operating system agent transmits readiness probes to the virtual machine, which responds by comparing the metrics to objective values indicated in an external dependency definition. Based on the response, in embodiments, the operating system may remove the virtual machine from service.
1 . A method, comprising:
instantiating a virtual machine including a cache, wherein the cache includes an area of memory of the virtual machine and is generated during instantiation of the virtual machine, based on configuration information including an external dependency definition defining a set of external dependencies associated with the virtual machine, metrics data to be obtained, and an objective value, the cache stores a set of summary data, where first summary data of the set of summary data is associated with a first external dependency of the set of external dependencies indicated in the external dependency definition and is generated based on network traffic between the virtual machine and the first external dependency;
obtaining, by the virtual machine over a first interval of time, network traffic between the virtual machine and the first external dependency of the set of external dependencies;
storing the first summary data in the cache, the first summary data generated based on a portion of the network traffic obtained over the first interval of time;
obtaining a readiness probe from an operating system agent associated with the virtual machine;
generating a response to the readiness probe based on a comparison of the first summary data to the objective value indicated in the external dependency definition, where the objective value indicates a metric associated with the first external dependency used to determine whether the first external dependency is in a non-functional or a limited-functional state; and
providing the response to the operating system agent.
2 . The method of claim 1 , wherein providing the response to the operating system agent causes the operating system agent to remove the virtual machine from service based on the response and at least one previous response to a previous readiness probe.
3 . The method of claim 2 , wherein removing the virtual machine from service comprises removing, from a Domain Name System (DNS), at least one entry associated with the virtual machine.
4 . The method of claim 2 , further comprising:
obtaining a second readiness probe from the operating system agent at an expiration of a second interval of time;
generating a second response to the second readiness probe based on updated summary data generated based on additional network traffic obtained during the second interval of time; and
returning the virtual machine to service based on the second response.
5 . The method of claim 1 , further comprising removing the summary data from the cache at an expiration of a second interval in response to not obtaining additional network traffic from the first external dependency during the second interval of time.
6 . The method of claim 1 , wherein the summary data includes a service level indicator (SLI) and the objective value includes a service level objective (SLO).
7 . The method of claim 6 , wherein the response indicates whether the SLI satisfies the SLO.
8 . The method of claim 6 , wherein the summary data indicates a percentile associated with the SLI based on a normal probability distribution determined based on previous network traffic obtained from the first external dependency.
9 . The method of claim 1 , where the network traffic further includes requests from the virtual machine to the first external dependency and responses from the first dependency.
10 . The method of claim 1 , wherein the summary data indicates an average response time associated with the first external dependency over the first interval of time.
11 . The method of claim 1 , wherein the summary data includes an average of response times to requests from the virtual machine to the first external dependency over the first interval of time.
12 . The method of claim 1 , wherein the virtual machine obtains the readiness probe from the operating system agent at an expiration of the first interval of time.
13 . A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
obtaining network traffic from an external dependency;
at an expiration of an interval of time, generating a metric corresponding to the network traffic from the external dependency;
storing the metric in a cache, wherein the cache includes an area of memory of a virtual machine generated based on an external dependency definition associated with the external dependency, the area of memory generated during instantiation of the virtual machine, where the external dependency definition indicates an objective value that indicates a threshold for determining if the external dependency has limited functionality;
obtaining a readiness probe from an operating system;
determining a response to the readiness probe based on the metric and the threshold indicated in the external dependency definition associated with the external dependency; and
providing the response to the operating system responsible for managing the virtual machine.
14 . The computing device of claim 13 , wherein the metric indicates a response time of the external dependency.
15 . The computing device of claim 13 , wherein the objective value includes a service level objective.
16 . The computing device of claim 13 , wherein the response indicates the metric satisfies the threshold or fails to satisfy the threshold.
17 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
transmitting a readiness probe to a virtual machine at an expiration of an interval of time;
obtaining a response to the readiness probe from the virtual machine based on summary data maintained in a cache of the virtual machine, wherein the cache includes an area of memory of the virtual machine and is generated during instantiation of the virtual machine, based on configuration information including an external dependency definition defining a set of external dependencies associated with the virtual machine, where the summary data indicates a metric corresponding to network traffic between the virtual machine and at least one external dependency relative to the virtual machine is below an objective value defined in the external dependency definition provided to the virtual machine;
determining the response satisfies a criteria for removing the virtual machine from service; and
preventing requests from being transmitted to the virtual machine.
18 . The non-transitory computer-readable medium claim 17 , wherein the operations further include:
transmitting a second readiness probe to the virtual machine;
obtaining a second response to the second readiness probe; and
based on the second response, allowing the requests to be transmitted to the virtual machine.
19 . The non-transitory computer-readable medium claim 17 , wherein the criteria indicates a number of negative responses to readiness probes.
20 . The non-transitory computer-readable medium claim 19 , wherein the negative responses to the readiness probes indicate that the summary data included in the cache of the virtual machine fails to satisfy a service level objective associated with an external dependency of the virtual machine.