Methods and systems for pre-processing sensor measurements
Described herein are techniques to decompose a measurement of the latency experienced by a virtual machine into its additive components (e.g., host latency, kernel latency, network latency and storage latency). The host latency may be determined as the aggregate latency reported from the virtual disk of the virtual machine minus the aggregate latency reported from the datastore on which the virtual disk is stored. The network latency may be determined as the device latency reported from the datastore minus the storage latency reported by a storage system. Various sensor measurements (e.g., aggregate latency reported from the virtual disk and/or datastore) may be first pre-processed to compensate for non-idealities before being combined with other sensor measurements.
1. A method, comprising:
receiving, by a processor, a first time series of storage latency from a storage system, wherein each storage latency of the first time series represents a time taken by the storage system to process a read or a write request;
receiving, by the processor, a second time series of device latency from a hypervisor manager, the hypervisor manager managing a hypervisor that interfaces a virtual machine of a host device with the storage system via a network, wherein each device latency of the second time series represents a time taken by the host device to read from the storage system or write to the storage system via the network;
receiving, by the processor, a third time series of kernel latency from the hypervisor manager, wherein each kernel latency of the third time series represents a time taken by a kernel of the hypervisor to process the read or the write request including a time that the read or the write request remains in a queue of the hypervisor;
computing, by the processor, a fourth time series of host latency based on the second time series of device latency and the third time series of kernel latency received from the hypervisor manager, wherein each host latency in the fourth time series represents a time taken by the virtual machine of the host device to process the read or write request;
computing, by the processor, a fifth time series of network latency by subtracting a respective storage latency of the first time series from a respective device latency of the second time series to obtain each network latency of the fifth time series, wherein each network latency of the fifth time series represents a time taken by the network to transmit the read or write request and a reply of the read or write request; and
facilitating, by the processor, a display of the first time series of storage latency, the fourth time series of host latency, and the fifth time series of network latency.
2. The method of claim 1 , wherein the time taken by the storage system to process the read or write request is an average time taken by the storage system to process the read or write request over the time period.
3. The method of claim 1 , wherein the time taken by the host device to read from the storage system or write to the storage system is an average time taken by the host device to read from the storage system or write to the storage system via the network over a time period.
4. The method of claim 1 , comprising: processing the second time series of device latency by temporally arranging samples of the second time series of device latency with respect to samples of the first time series of storage latency.
5. The method of claim 1 , comprising: processing the first time series of storage latency by substituting estimated samples in place of missing samples of the first time series of storage latency.
6. The method of claim 1 , comprising: processing the second time series of device latency by substituting erroneous samples of the second time series with zero values.
7. The method of claim 1 , wherein computing the fourth time series of host latency comprises:
determining a time series of aggregate kernel and device latency;
determining a time series of aggregate host, kernel, and device latency; and
subtracting the time series of aggregate kernel and device latency from the time series of aggregate host, kernel, and device latency.
8. An analysis server, comprising:
a processor; and
a memory storing a set of instructions that, when executed by the processor, causes the processor to:
receive a first time series of storage latency from a storage system, wherein each storage latency in the first time series represents an average time taken by the storage system to process a read or a write request;
receive a second time series of device latency from a hypervisor manager, the hypervisor manager managing a hypervisor that interfaces a virtual machine of a host device with the storage system via a network, wherein each device latency in the second time series represents a time taken by the host device to read from the storage system or write to the storage system via the network;
receive a third time series of kernel latency from the hypervisor manager, wherein each kernel latency in the third time series represents a time taken by a kernel of the hypervisor to process the read or the write request including a time that the read or the write request remains in a queue of the hypervisor;
compute a fourth time series of host latency based on the second time series of device latency and the third time series of kernel latency received from the hypervisor manager, wherein each host latency in the fourth time series represents a time taken by the virtual machine of the host device to process the read or write request;
compute a fifth time series of network latency by subtracting a respective storage latency of the first time series from a respective device latency of the second time series to obtain each network latency of the fifth time series, wherein each network latency of the fifth time series represents a time taken by the network to transmit the read or write request and a reply of the read or write request; and
facilitate a display of the first time series of storage latency, the fourth time series of host latency, and the fifth time series of network latency.
9. The analysis server of claim 8 , wherein the storage latency represents an average time taken by the storage system to process a read or a write request, the storage latency being averaged over a time period.
10. The analysis server of claim 8 , wherein the device latency represents an average time taken by the host device to read from the storage system or write to the storage system, the device latency being averaged over a time period.
11. The analysis server of claim 8 , wherein the set of instructions is executable to cause the processor to temporally arrange samples of the second time series of device latency with respect to samples of the first time series of storage latency.
12. The analysis server of claim 8 , wherein the set of instructions is executable to cause the processor to process the first time series of storage latency by substituting estimated samples in place of missing samples of the first time series of storage latency.
13. The analysis server of claim 8 , wherein the set of instructions is to cause the processor to process the second time series of device latency by substituting erroneous samples of the second latency times series with zero values.
14. The analysis server of claim 8 , wherein to compute the fifth time series of host latency, the set of instructions is executable to cause the processor to:
determine a time series of aggregate kernel and device latency;
determine a time series of aggregate host, kernel, and device latency; and
subtract the time series of aggregate kernel and device latency from the time series of aggregate host, kernel, and device latency.
15. A non-transitory machine-readable storage medium comprising machine readable instructions that, when executed by a processor, cause the processor to:
receive a first time series of storage latency from a storage system, wherein each storage latency of the latency time series represents a time taken by the storage system to process a read or a write request;
receive a second time series of device latency from a hypervisor manager, the hypervisor manager managing a hypervisor that interfaces a virtual machine of a host device with the storage system via a network, wherein each device latency of the second time series represents a time taken by the host device to read from the storage system or write to the storage system via the network;
receive a third time series of kernel latency from the hypervisor manager, wherein each kernel latency in the third time series represents a time taken by a kernel of the hypervisor to process the read or the write request including a time that the read or the write request remains in a queue of the hypervisor;
compute a fourth time series of host latency based on the second time series of device latency and the third time series of kernel latency received from the hypervisor manager, wherein each host latency of the fourth time series represents a time taken by the virtual machine of the host device to process the read or write request;
compute a fifth time series of network latency by subtracting a respective storage latency of the first time series from a respective device latency of the second time series to obtain each network latency of the fifth time series, wherein each network latency in the fifth time series represents a time taken by the network to transmit the read or write request and a reply of the read or write request; and facilitate a display of the first time series of storage latency, the fourth time series of host latency, and the fifth time series of network latency.
16. The non-transitory machine-readable storage medium of claim 15 , wherein the storage latency represents an average time taken by the storage system to process a read or a write request, the storage latency being averaged over a time period.
17. The non-transitory machine-readable storage medium of claim 15 , wherein the device latency represents an average time taken by the host device to read from the storage system or write to the storage system, the device latency being averaged over a time period.
18. The non-transitory machine-readable storage medium of claim 15 , wherein the machine-readable instructions are executable to cause the processor to temporally arrange samples of the second time series of device latency with respect to samples of the first time series of storage latency.
19. The non-transitory machine-readable storage medium of claim 15 , wherein the machine readable instructions are executable to cause the processor to process the first time series of storage latency by substituting estimated samples in place of missing samples of the first time series of storage latency.
20. The non-transitory machine-readable storage medium of claim 15 , wherein the machine readable instructions are executable to cause the processor to process the second time series of device latency by substituting erroneous samples of the second times series with zero values.