Query watchdog
A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.
1. A system for monitoring job execution, comprising:
an interface configured to:
receive an indication to start a cluster processing job; and
a processor configured to:
divide the cluster processing job into a plurality of data instances;
determine whether processing a data instance of the plurality of data instances satisfies a watchdog criterion; and
in the event that processing the data instance satisfies the watchdog criterion:
cause the processing of the data instance to be killed;
increment a killed data instance counter;
determine whether the killed data instance counter satisfies a job killing threshold; and
in response to a determination that the killed data instance counter satisfies the job killing threshold, cause processing of the cluster processing job to be killed.
2. The system of claim 1 , wherein the data instance comprises a row.
3. The system of claim 1 , wherein the watchdog criterion comprises a time limit exceeded for processing the data.
4. The system of claim 1 , wherein the watchdog criterion comprises a time limit exceeded during processing of a most recent N data instances, wherein N comprises a predetermined number of data instances for the time limit.
5. The system of claim 1 , wherein the watchdog criterion comprises an output limit exceeded during processing of the data instance.
6. The system of claim 5 , wherein the output limit comprises a number of output bytes.
7. The system of claim 5 , wherein the output limit comprises a number of output data instances.
8. The system of claim 5 , wherein the output limit comprises a number of output files indicating an unacceptable amount of resource usage for the data instance.
9. The system of claim 1 , wherein the watchdog criterion comprises an output limit exceeded during processing of a most recent N data instances, wherein N comprises a predetermined number of data instances for the output limit.
10. The system of claim 1 , wherein the processor is further configured to indicate that the cluster processing job has been killed.
11. The system of claim 1 , wherein the processor is further configured to indicate the data instance that caused the cluster processing job to be killed.
12. The system of claim 1 , wherein the processor is further configured to indicate the watchdog criterion that has been satisfied.
13. The system of claim 1 , wherein the processor is further configured to:
after processing of the data instance is caused to be killed, cause processing of a next data instance of the plurality of data instances.
14. The system of claim 13 , wherein the processor is further configured to in the event that processing the data instance satisfies the watchdog criterion, indicate that processing of the data instance was caused to be killed.
15. The system of claim 13 , wherein the processor is further configured to store an indication in an output file that processing of the data instance was caused to be killed.
16. A method for monitoring job execution, comprising:
receiving an indication to start a cluster processing job;
dividing the cluster processing job into a plurality of data instances; and
determining, using a processor, whether processing a data instance of the plurality of data instances satisfies a watchdog criterion; and
in the event that processing the data instance satisfies the watchdog criterion:
causing the processing of the data instance to be killed;
incrementing a killed data instance counter;
determining whether the killed data instance counter satisfies a job killing threshold; and
in response to a determination that the killed data instance counter satisfies the job killing threshold, causing processing of the cluster processing job to be killed.
17. A computer program product for monitoring job execution, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving an indication to start a cluster processing job;
dividing the cluster processing job into a plurality of data instances; and
determining whether processing a data instance of the plurality of data instances satisfies a watchdog criterion; and
in the event that processing the data instance satisfies the watchdog criterion:
causing the processing of the data instance to be killed;
incrementing a killed data instance counter;
determining whether the killed data instance counter satisfies a job killing threshold; and
in response to a determination that the killed data instance counter satisfies the job killing threshold, causing processing of the cluster processing job to be killed.