Query watchdog
A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.
1. A system for monitoring job execution, comprising:
an interface configured to:
receive an indication to start a cluster processing job; and
a processor configured to:
determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion, wherein the watchdog criterion comprises one or more of the following:
a first time limit exceeded for processing the data instance;
a second time limit exceeded for processing the last N data instances, wherein N corresponds to a predetermined number of data instances for the second time limit;
a first output limit exceeded for processing the data instance; and/or
a second output limit exceeded for processing the last M data instances, wherein M corresponds to a predetermined number of data instances for the second output limit and
in response to a determination that the processing of the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.
2. The system of claim 1 , wherein the data instance comprises a row.
3. The system of claim 1 , wherein the output limit comprises a number of output bytes.
4. The system of claim 1 , wherein the output limit relates to a number of output data instances.
5. The system of claim 1 , wherein the output limit corresponds to a number of output files.
6. The system of claim 1 , wherein the processor is further configured to cause processing of the cluster processing job to be killed.
7. The system of claim 6 , wherein the processor is further configured to indicate that the cluster processing job has been killed.
8. The system of claim 6 , wherein the processor is further configured to indicate the data instance that caused the cluster processing job to be killed.
9. The system of claim 1 , wherein the processor is further configured to indicate that the watchdog criterion has been satisfied.
10. The system of claim 1 , wherein the processor is further configured to:
after the causing of the data instance to be killed, cause processing of a next data instance.
11. The system of claim 10 , wherein the processor is further configured to indicate that the data instance was caused to be killed.
12. The system of claim 10 , wherein the processor is further configured to store an indication in an output file that the data instance was caused to be killed.
13. The system of claim 10 , wherein the processor is further configured to increment a killed data instance counter.
14. The system of claim 13 , wherein the processor is further configured to:
in response to a determination that a value of the killed data instance counter exceeds a threshold, cause execution of the cluster processing job to be killed.
15. A method for monitoring job execution, comprising:
receiving an indication to start a cluster processing job; and
determining, using a processor, whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion, wherein the watchdog criterion comprises one or more of the following:
a first time limit exceeded for processing the data instance;
a second time limit exceeded for processing the last N data instances, wherein N corresponds to a predetermined number of data instances for the second time limit;
a first output limit exceeded for processing the data instance; and/or
a second output limit exceeded for processing the last M data instances, wherein M corresponds to a predetermined number of data instances for the second output limit; and;
in response to a determination that the processing of the data instance satisfies the watchdog criterion, causing the processing of the data instance to be killed.
16. A computer program product for monitoring job execution, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving an indication to start a cluster processing job; and
determining whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion, wherein the watchdog criterion comprises one or more of the following:
a first time limit exceeded for processing the data instance;
a second time limit exceeded for processing the last N data instances, wherein N corresponds to a predetermined number of data instances for the second time limit;
a first output limit exceeded for processing the data instance; and/or
a second output limit exceeded for processing the last M data instances, wherein M corresponds to a predetermined number of data instances for the second output limit; and
in response to a determination that the processing of the data instance satisfies the watchdog criterion, causing the processing of the data instance to be killed.
17. The system of claim 1 , wherein the watchdog criterion comprises two or more of the following:
a first time limit exceeded for processing the data instance;
a second time limit exceeded for processing the last N data instances, wherein N corresponds to a predetermined number of data instances for the second time limit;
a first output limit exceeded for processing the data instance; and/or
a second output limit exceeded for processing the last M data instances, wherein M corresponds to a predetermined number of data instances for the second output limit.
18. The system of claim 1 , wherein the watchdog criterion comprises three or more of the following:
a first time limit exceeded for processing the data instance;
a second time limit exceeded for processing the last N data instances, wherein N corresponds to a predetermined number of data instances for the second time limit;
a first output limit exceeded for processing the data instance; and/or
a second output limit exceeded for processing the last M data instances, wherein M corresponds to a predetermined number of data instances for the second output limit.
19. The system of claim 1 , wherein the watchdog criterion comprises the following:
a first time limit exceeded for processing the data instance;
a second time limit exceeded for processing the last N data instances, wherein N corresponds to a predetermined number of data instances for the second time limit;
a first output limit exceeded for processing the data instance; and
a second output limit exceeded for processing the last M data instances, wherein M corresponds to a predetermined number of data instances for the second output limit.