Milestone based dynamic multiple watchdog timeouts and early failure detection
A dynamic, milestone-based solution for managing computer operations, as may be embodied by various systems, methods, and non-transitory computer-readable storage media, may involve analyzing a computer operation invoked by an invoking module. The solution may involve identifying and classifying discrete steps or phases within the invoked operation and applying context-specific watchdog timers and operational milestones based on the detected phases. The solution may involve monitoring the progress of the operation and terminating the operation when an operational milestone is not achieved before an applied watchdog timer expires during a given phase.
1. A dynamic, milestone-based method for managing a plurality of computer operations, the method comprising:
executing an invoking module, the invoking module including a plurality of executable instructions stored in a memory of a computing device, wherein executing the invoking module invokes a computer operation of the plurality of computer operations; and
executing a monitoring engine, the monitoring engine including a plurality of executable instructions stored in the memory of the computing device, wherein executing the monitoring engine:
classifies a first phase of the invoked computer operation;
selects a first watchdog timer, a first watchdog timer timeout value, and a first milestone for the first phase based on the classification of the first phase and a priority of the invoked computer operation;
activates the first watchdog timer;
monitors progress of the first phase;
compares the progress to the first milestone;
terminates the operation when the progress fails to meet the first milestone before the first watchdog timer reaches the timeout value; and
permits the operation to progress to a second phase when the progress meets the first milestone before the first watchdog timer reaches the timeout value.
2. The method of claim 1 , wherein further execution of the monitoring engine:
classifies a second phase of the invoked computer operation;
selects a second watchdog timer, a second watchdog timer timeout value, and a second milestone for the second phase based on the classification of the second phase and the priority of the invoked computer operation;
activates the second watchdog timer;
monitors progress of the second phase;
compares the progress of the second phase to the second milestone; and
terminates the operation when the progress of the second phase fails to meet the second milestone before the second watchdog timer reaches the second watchdog timeout value.
3. The method of claim 1 , wherein further execution of the invoking module deactivates the monitoring engine when the operation completes.
4. The method of claim 1 , wherein monitoring progress of the first phase includes examining process messages.
5. The method of claim 4 , wherein examining process messages includes examining process messages at Standard Out or Standard Error.
6. The method of claim 1 , wherein monitoring progress of the first phase includes examining log messages.
7. The method of claim 1 , wherein monitoring progress of the first phase includes detecting when the operation creates a predetermined file.
8. The method of claim 7 , wherein the predetermined file is a temporary file.
9. The method of claim 1 , wherein monitoring progress of the first phase includes detecting when the computing device has received a predetermined threshold level of data of a file being received.
10. The method of claim 1 , wherein monitoring progress of the first phase includes examining kernel activity.
11. The method of claim 10 , wherein examining kernel activity includes examining syscalls using a ptrace function.
12. The method of claim 1 , wherein monitoring progress of the first phase includes examining packet traffic.
13. The method of claim 12 , wherein examining packet traffic includes analyzing packet traffic for the presence of a predetermined packet.
14. The method of claim 12 , wherein examining packet traffic includes analyzing packet traffic for the absence of a predetermined packet.
15. The method of claim 12 , wherein examining packet traffic includes comparing characteristics of real-time packet traffic to an expected packet traffic data of the invoked operation.
16. The method of claim 1 , wherein monitoring progress of the first phase includes intercepting a library function call made by the invoked operation.
17. The method of claim 16 , wherein intercepting the library function call includes:
creating a mirror library that includes the same functions present in a standard library used by the invoked operation;
specifying the mirror library during preload such that function calls made by the invoked operation are sent to the mirror library instead of the standard library used by the invoked operation;
receiving a function call from the invoked operation;
determining a level of progress made by the invoked operation based on the received function call; and
passing the function call to the standard library used by the invoked operation.
18. The method of claim 1 , wherein monitoring progress of the first phase includes using debug mode.
19. The method of claim 1 , wherein monitoring progress of the first phase includes:
performing a minor operation identical to the invoked operation;
monitoring progress of a first phase of the minor operation; and
deeming the progress of the mirror operation as identical to progress of the invoked operation.
20. A non-transitory computer-readable storage medium having a computer program embodied thereon, the computer program executable by a processor to perform a dynamic, milestone-based method for managing a plurality of computer operations, the method comprising:
invoking a computer operation of the plurality of computer operations;
classifying a first phase of the invoked computer operation;
selecting a first watchdog timer, a first watchdog timer timeout value, and a first milestone for the first phase based on the classification of the first phase and a priority of the invoked computer operation;
activating the first watchdog timer;
monitoring progress of the first phase;
comparing the progress to the first milestone;
terminating the operation when the progress fails to meet the first milestone before the first watchdog timer reaches the timeout value; and
when the progress meets the first milestone before the first watchdog timer reaches the timeout value:
classifying a second phase of the invoked computer operation;
selecting a second watchdog timer, a second watchdog timer timeout value, and a second milestone for the second phase based on the classification of the second phase and the priority of the invoked computer operation;
activating the second watchdog timer;
monitoring progress of the second phase;
comparing the progress of the second phase to the second milestone; and
terminating the operation when the progress of the second phase fails to meet the second milestone before the second watchdog timer reaches the second watchdog timeout value.