Remedial action based on maintaining process awareness in data storage management
An illustrative data storage management system comprises “awareness logic” that executes on computing devices hosting storage management components such as storage manager, data agent, media agent, and/or other storage management applications. The illustrative awareness logic operates within each of these illustrative components, e.g., as a thread within processes of the storage management component, such as storage management core process, file identifier process, log monitoring process, etc. The awareness logic monitors the targeted process over time and triggers remedial action when criteria are met. Certain vital statistics of each process are collected periodically and analyzed by the illustrative awareness logic, such as CPU usage, memory usage, and handle counts. Criteria for corrective action include rising trends based on local minima data points for one or more vital statistics of the process. Other criteria include exceeding a threshold based on a logarithm function of the collected data points.
1. A computer-implemented method for determining whether a software process executing on a computing device performs unacceptably, the method comprising:
on a computing device that comprises one or more hardware processors and executes a process for data storage management in at least one software-implemented component of a data storage management system:
fitting a regression curve to data points that are local minima among a set of previously-measured data points of a first vital statistic of the process,
wherein the first vital statistic is one of: CPU usage, computer-memory usage, and a count of process handles;
determining that the process operates unacceptably based on determining that:
(i) the regression curve of the local minima data points has a positive slope, and
(ii) a first data point of the first vital statistic exceeds a threshold value based on a logarithm function of the set of previously-measured data points of the first vital statistic of the process; and
based on the determining that the process operates unacceptably, applying a remedial measure to the process.
2. The method of claim 1 wherein the first data point is measured after measuring the set of previously-measured data points.
3. The method of claim 1 , wherein the determining that the process operates unacceptably is based on additionally determining that: (iii) the first data point of the first vital statistic exceeds a maximum allowed value that is a statistical function of values of the first vital statistic in the set of previously-measured data points.
4. The method of claim 1 , wherein the set of previously-measured data points are maintained in a sliding window and the method further comprising:
based on determining that the process operates acceptably, replacing an oldest data point in the sliding window with the first data point of the first vital statistic; and
based on the determining that the process operates unacceptably, excluding the first data point of the first vital statistic from the sliding window.
5. The method of claim 1 , further comprising:
determining that the process operates acceptably with regard to the first vital statistic based on determining at least one of:
(A) the regression curve of the local minima data points does not have a positive slope, and
(B) the first data point of the first vital statistic does not exceed the threshold value.
6. The method of claim 1 , wherein the process is a core process for storage management and the remedial measure comprises taking a core dump.
7. The method of claim 1 , wherein the process is not a core process for storage management and the remedial measure comprises one or more of: lowering a priority of the process and restarting the process.
8. The method of claim 1 , wherein the process is one of: a storage management core process, a file identifier process, and a log monitoring process.
9. The method of claim 1 , wherein the at least one software-implemented component is one of a storage manager, a data agent, a media agent, a system monitor, and a report server.
10. A system comprising:
a computing device comprising one or more hardware processors that provide a virtualized computing configuration for executing at least one software-implemented component of a data storage management system;
wherein the at least one software-implemented component of the data storage management system is configured to:
periodically measure a first data point of a first vital statistic of a process that executes on the computing device, wherein the first vital statistic is one of: CPU usage, computer-memory usage, and a count of process handles,
fit a regression curve to data points that are local minima among a set of previously-measured data points of the first vital statistic of the process for data storage management, and
apply a remedial measure to the process based on determining that:
(i) the regression curve of the local minima data points has a positive slope, and
(ii) a first data point of the first vital statistic exceeds a threshold value based on a logarithm function of the set of previously-measured data points of the first vital statistic of the process.
11. The system of claim 10 , wherein the first data point is measured after measuring the set of previously-measured data points.
12. The system of claim 10 , wherein the remedial measure is applied based on additionally determining that (iii) the first data point of the first vital statistic exceeds a maximum allowed value that is a statistical function of values in the set of previously-measured data points.
13. The system of claim 10 , wherein the set of previously-measured data points are maintained in a sliding window stored in a cache associated with the at least one software-implemented component of the data storage management system.
14. The system of claim 13 , wherein the first data point of the first vital statistic replaces an oldest data point in the sliding window based on determining that the process operates acceptably, and wherein the first data point of the first vital statistic is not added to the sliding window based on determining that the process operates unacceptably.
15. The system of claim 10 , wherein the process is part of the at least one software-implemented component of the data storage management system, wherein the at least one software-implemented component is one of a storage manager, a data agent, a media agent, a system monitor, and a report server.
16. A system comprising:
a computing device comprising one or more hardware processors that provide a virtualized computing configuration for executing at least one software-implemented component of a data storage management system;
wherein the at least one software-implemented component of the data storage management system is configured to:
periodically measure a first data point of a first vital statistic of a process for data storage management, wherein the process executes on the computing device, wherein the first vital statistic is one of: CPU usage, computer-memory usage, and a count of process handles,
fit a regression curve to data points that are local minima among a set of previously-measured data points of the first vital statistic of the process, and
apply a remedial measure to the process based on determining that:
(i) the regression curve of the local minima data points has a positive slope,
(ii) a first data point of the first vital statistic exceeds a threshold value based on a logarithm function of the set of previously-measured data points of the first vital statistic of the process, and
(iii) the first data point of the first vital statistic exceeds a maximum allowed value that is a statistical function of values in the set of previously-measured data points.
17. The system of claim 16 , wherein the first data point is measured after measuring the set of previously-measured data points.
18. The system of claim 16 , wherein the set of previously-measured data points are maintained in a sliding window stored in a cache associated with the at least one software-implemented component of the data storage management system.
19. The system of claim 18 , wherein the first data point of the first vital statistic replaces an oldest data point in the sliding window based on determining that the process operates acceptably, and wherein the first data point of the first vital statistic is not added to the sliding window based on determining that the process operates unacceptably.
20. The system of claim 16 , wherein the process is part of the at least one software-implemented component of the data storage management system, wherein the at least one software-implemented component is one of: a storage manager, a data agent, a media agent, a system monitor, and a report server.