Method to identify the performance bottle neck in the complex enterprise virtualized environment
One example method includes identifying a source of a performance issue in a virtualized environment. Telemetry data is collected relative to the flow of a request/response in the virtualized environment. The collected telemetry data can be compared to normal data. A probability can be generated for each layer to identify which of the layers is the most likely source of the performance issue. The layers can be prioritized based on their probability. The most likely layer or virtual machine is recommended for analysis to determine the cause of the performance issue.
1. A method, comprising:
collecting telemetry data for a virtual machine, the telemetry data including request times and response times associated with a request flowing through layers of the virtual machine, wherein the request times include times associated with the request as the request flows through the layers of the virtual machine from a user layer to a hardware layer, wherein each of the times corresponds to a time for the request to flow from one layer to the next layer, wherein the response times include times associated with a response as the response flows back from the hardware layer to the user layer of the virtual machine, wherein each of the times corresponds to a time for the response to flow from one layer to the next layer;
evaluating the collected telemetry data based on expected telemetry data, wherein the expected telemetry data includes expected request times and expected response times;
generating a probability for each layer of the virtual machine, wherein the probability represents a likelihood that a corresponding layer is a potential source of a performance issue;
identifying the potential source of the performance issue in the virtual machine based on the evaluation and the probability by troubleshooting the performance issue starting in the layer associated with a highest probability; and
updating the expected telemetry data based on the collected telemetry data.
2. The method of claim 1 , further comprising collecting the telemetry data in response to an alert indicating that the virtual machine or another virtual machine has the performance issue.
3. The method of claim 1 , further comprising collecting the telemetry data for multiple virtual machines.
4. The method of claim 1 , further comprising prioritizing the layers for analysis at a console to determine a cause of the performance issue.
5. The method of claim 4 , further comprising setting the expected telemetry data to default values.
6. The method of claim 5 , further comprising adjusting the expected telemetry data over time based on telemetry data collected when no performance issues are present.
7. The method of claim 1 , wherein the request times include a time T 1 from a user layer to an application layer, a time T 2 from the application layer to a first OS layer, a time T 3 from the first OS layer to the virtual machine layer, a time T 4 from the virtual machine layer to a hypervisor layer, a time T 5 from the hypervisor layer to a second operating system layer, and a time T 6 from the second operating system layer to a hardware layer and wherein the response times include a time T 7 from the hardware layer to the second operating system layer, a time T 8 from the second operating system layer to the hypervisor layer, a time T 9 from the hypervisor layer to the virtual machine layer, a time T 10 from the virtual machine layer to the first OS layer, a time T 1 from the first OS layer to the application layer, and a time T 12 from the application layer to the user layer.
8. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
collecting telemetry data for a virtual machine, the telemetry data including request times and response times associated with a request flowing through layers of the virtual machine, wherein the request times include times associated with the request as the request flows through the layers of the virtual machine from a user layer to a hardware layer, wherein each of the times corresponds to a time for the request to flow from one layer to the next layer, wherein the response times include times associated with a response as the response flows back from the hardware layer to the user layer of the virtual machine, wherein each of the times corresponds to a time for the response to flow from one layer to the next layer;
evaluating the collected telemetry data based on expected telemetry data, wherein the expected telemetry data includes expected request times and expected response times;
generating a probability for each layer of the virtual machine, wherein the probability represents a likelihood that a corresponding layer is a potential source of a performance issue;
identifying the potential source of the performance issue in the virtual machine based on the evaluation and the probability by troubleshooting the performance issue starting in the layer associated with a highest probability; and
updating the expected telemetry data based on the collected telemetry data.
9. The non-transitory storage medium of claim 8 , further comprising collecting the telemetry data in response to an alert indicating that the virtual machine or another virtual machine has the performance issue.
10. The non-transitory storage medium of claim 8 , further comprising collecting the telemetry data for multiple virtual machines.
11. The non-transitory storage medium of claim 8 , further comprising prioritizing the layers for analysis at a console to determine a cause of the performance issue.
12. The non-transitory storage medium of claim 11 , further comprising setting the expected telemetry data to default values.
13. The non-transitory storage medium of claim 12 , further comprising adjusting the expected telemetry data over time based on telemetry data collected when no performance issues are present.
14. The non-transitory storage medium of claim 8 , wherein the request times include a time T 1 from a user layer to an application layer, a time T 2 from the application layer to a first OS layer, a time T 3 from the first OS layer to the virtual machine layer, a time T 4 from the virtual machine layer to a hypervisor layer, a time T 5 from the hypervisor layer to a second operating system layer, and a time T 6 from the second operating system layer to a hardware layer and wherein a the response times include a time T 7 from the hardware layer to the second operating system layer, a time T 8 from the second operating system layer to the hypervisor layer, a time T 9 from the hypervisor layer to the virtual machine layer, a time T 10 from the virtual machine layer to the first OS layer, a time T 11 from the first OS layer to the application layer, and a time T 12 from the application layer to the user layer.