Performance monitoring and enhancement
A computer implemented method including generating one or more threads to operate a storage system, the storage system including one or more system resources, and each of the one or more threads associated with one of a plurality of tasks of the storage system. The method includes updating one or more performance counters of the storage system in response to a change in operating state of one of the one or more threads. The method includes generating a performance snapshot by sampling a state of the tasks associated with the threads and the performance counters, identifying one or more intervals of performance degradation of the storage system from the performance snapshot, and determining a correlation between one or more of the plurality of tasks and the one or more identified intervals of performance degradation.
1. A computer implemented method comprising:
generating one or more threads to operate a storage system, the storage system comprising one or more system resources, and each of the one or more threads associated with one of a plurality of tasks of the storage system;
updating one or more performance counters of the storage system in response to a change in operating state of one of the one or more threads;
generating performance snapshots by sampling a state of the tasks associated with the threads and the performance counters, any of the performance snapshots including one or more task-specific directed graphs;
identifying one or more intervals of performance degradation of the storage system from the performance snapshots; and
clustering time-series of samples from the performance snapshots to identify one or more of the plurality of tasks as a cause of any of the intervals of performance degradation.
2. The method of claim 1 , wherein the performance degradation comprises an interval of relatively slower execution of the one or more of the plurality of tasks.
3. The method of claim 1 wherein the performance degradation comprises performance-jitter of the storage system for at least one data sequence.
4. The method of claim 1 , further comprising associating the one or more performance counters with a given task, the one or more performance counters comprising one or more of a counter for accumulating a number of accesses of an associated resource by the given task and a counter for accumulating a number of requested units from the associated resource by the given task.
5. The method of claim 1 , further comprising associating the one or more performance counters with a given task, the one or more performance counters comprising one or more of: a counter for accumulating a time of the given task in a suspended state, a counter for determining a number of suspended requests to an associated resource by the given task, and performance counters associated with sequences of processing instructions comprising one or more of: a counter for accumulating a number of passes of the given task, a counter for accumulating a time of one or more operating states of the given task, and a counter for accumulating a number of cooperative yields of the given task.
6. The method of claim 1 wherein identifying one or more intervals of performance degradation comprises applying a low pass filter to the performance snapshots to remove sampling noise from the performance snapshots.
7. The method of claim 1 wherein identifying one or more intervals of performance degradation comprises normalizing the performance snapshots such that the mean value of one or more performance counters is normalized to zero and the standard deviation of the one or more performance counters normalized to one.
8. The method of claim 1 further comprising modifying an execution of at least one of the tasks to reduce further occurrences of performance degradation of the storage system that were identified.
9. The method of claim 8 wherein modifying execution comprises smoothing a sequence of execution of the at least one of the tasks.
10. The method of claim 1 , wherein the clustering of time-series of samples from the performance snapshots is performed by using a clustering technique, the clustering technique including at least one of K-means clustering, Mean-shift clustering, Density-based spatial clustering, Expectation-Maximization clustering using Gaussian Mixture Models (GMM).
11. A computer processing system comprising:
one or more processors programmed and configured to cause the execution of:
generating one or more threads to operate a storage system, the storage system comprising one or more system resources, and each of the one or more threads associated with one of a plurality of tasks of the storage system;
updating one or more performance counters of the storage system in response to a change in operating state of one of the one or more threads;
generating performance snapshots by sampling a state of the tasks associated with the threads and the performance counters, any of the performance snapshots including one or more task-specific directed graphs;
identifying one or more intervals of performance degradation of the storage system from the performance snapshots; and
clustering time-series of samples from the performance snapshots to identify one or more of the plurality of tasks as a cause of any of the intervals of performance degradation.
12. The system of claim 11 , wherein the performance degradation comprises an interval of relatively slower execution of the one or more of the plurality of tasks.
13. The system of claim 11 wherein the performance degradation comprises performance-jitter of the storage system.
14. The system of claim 11 , wherein the one or more processors are further configured to associate the one or more performance counters with a given task, the one or more performance counters comprising one or more of a counter for accumulating a number of accesses of an associated resource by the given task and a counter for accumulating a number of requested units from the associated resource by the given task.
15. The system of claim 11 , wherein the one or more processors are further configured to associate the one or more performance counters with a given task, the one or more performance counters comprising one or more of: a counter for accumulating a time of the given task in a suspended state, a counter for determining a number of suspended requests to an associated resource by the given task, and performance counters associated with sequences of processing instructions comprising one or more of: a counter for accumulating a number of passes of the given task, a counter for accumulating a time of one or more operating states of the given task, and a counter for accumulating a number of cooperative yields of the given task.
16. The system of claim 11 wherein identifying one or more intervals of performance degradation comprises applying a low pass filter to the performance snapshots to remove sampling noise from the performance snapshots.
17. The system of claim 11 wherein identifying one or more intervals of performance degradation comprises normalizing the performance snapshots such that the mean value of one or more performance counters is normalized to zero and the standard deviation of the one or more performance counters normalized to one.
18. The system of claim 11 wherein the one or more processors are configured to modify an execution of at least one of the tasks to reduce further occurrences of the intervals of performance degradation of the storage system that were identified.
19. The system of claim 11 , wherein the clustering of time-series of samples from the performance snapshots is performed by using a clustering technique, the clustering technique including at least one of K-means clustering, Mean-shift clustering, Density-based spatial clustering, Expectation-Maximization clustering using Gaussian Mixture Models (GMM).
20. The system of claim 18 wherein modifying execution comprises smoothing a sequence of execution of the at least one of the tasks.