Process activeness check
Described herein are techniques for dynamically monitoring process activeness of processes running on a computing node. Problems affecting processes to serve their designated functions on the computing node can be relatively quickly detected and dealt with, thereby making restoring process activeness on the computing node much more quickly than otherwise.
1. A method, comprising the computer-implemented steps of:
a plurality of processes executing on a node, wherein each process of said plurality of processes writes to one or more progress logs operation completion information that specifies operation types for said each process, and for each operation type, information about how often each operation type was performed in the past;
a counter-part process monitoring the one or more progress logs;
the counter-part process determining, based on the operation completion information in the one or more progress logs, whether a particular process of said plurality of processes is running at a normal level of activity, wherein the counter-part process is different than the particular process; and
in response to determining, based on the operation completion information in the one or more progress logs, that the particular process is not running at a normal level of activity, the counter-part process determining whether the particular process is restorable to be running at the normal level of activity.
2. A method as recited in claim 1 , wherein the plurality of processes forms a process group that supports a common set of operation types.
3. A method as recited in claim 1 , wherein the counter-part process monitoring the one or more progress logs includes taking a first snapshot of a progress profile related to a first time from the one or more progress logs and a second snapshot of the progress profile related to a second time from the one or more progress logs and comparing the first snapshot and the second snapshot.
4. A method as recited in claim 1 , wherein the counter-part process monitoring the one or more progress logs includes calculating an average rate for an operation type.
5. A method as recited in claim 1 , wherein determining whether the particular process is restorable to be running at the normal level of activity includes determining whether the particular process should be assigned with a high priority.
6. A method as recited in claim 1 , wherein determining whether the particular process is restorable to be running at the normal level of activity includes determining whether the particular process should be killed.
7. A method as recited in claim 1 , wherein determining whether at least one of the plurality of processes is restorable to be running at the normal level of activity includes determining whether a new process should be spawned to share workload with the plurality of processes.
8. A method as recited in claim 1 , wherein determining whether the particular process is restorable to be running at the normal level of activity includes determining whether a node on which the plurality of processes reside should be restarted to restart all processes on the node.
9. A method as recited in claim 1 , wherein the one or more progress logs include one or more timestamp indicating times at which the one or more progress logs were last updated in the past by the plurality of processes.
10. A method as recited in claim 1 , wherein the one or more progress logs include a count of operations of a particular type that have been completed by the plurality of processes.
11. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform:
a plurality of processes executing on a node, wherein each process of said plurality of processes writes to one or more progress logs operation completion information that specifies operation types for said each process, and for each operation type, information about how often each operation type was performed in the past;
a counter-part process monitoring the one or more progress logs;
the counter-part process determining, based on the operation completion information in the one or more progress logs, whether a particular process of said plurality of processes is running at a normal level of activity, wherein the counter-part process is different than the particular process; and
in response to determining, based on the operation completion information in the one or more progress logs, that the particular process is not running at a normal level of activity, the counter-part process determining whether the particular process is restorable to be running at the normal level of activity.
12. A medium as recited in claim 11 , wherein the plurality of processes forms a process group that supports a common set of operation types.
13. A medium as recited in claim 11 , wherein the counter-part process monitoring the one or more progress logs includes taking a first snapshot of a progress profile related to a first time from the one or more progress logs and a second snapshot of the progress profile related to a second time from the one or more progress logs and comparing the first snapshot and the second snapshot.
14. A medium as recited in claim 11 , wherein the counter-part process monitoring the one or more progress logs includes calculating an average rate for an operation type.
15. A medium as recited in claim 11 , wherein determining whether the particular process is restorable to be running at the normal level of activity includes determining whether the particular process should be assigned with a high priority.
16. A medium as recited in claim 11 , wherein determining whether the particular process is restorable to be running at the normal level of activity includes determining whether the particular process should be killed.
17. A medium as recited in claim 11 , wherein determining whether the particular process is restorable to be running at the normal level of activity includes determining whether a new process should be spawned to share workload with the plurality of processes.
18. A medium as recited in claim 11 , wherein determining whether the particular process is restorable to be running at the normal level of activity includes determining whether a node on which the plurality of processes reside should be restarted to restart all processes on the node.
19. A medium as recited in claim 11 , wherein the one or more progress logs include one or more timestamp indicating times at which the one or more progress logs were last updated in the past by the plurality of processes.
20. A medium as recited in claim 11 , wherein the one or more progress logs include a count of operations of a particular type that have been completed by the plurality of processes.