Report reexecution framework
The present disclosure provides techniques and solutions for analyzing failed computing tasks (or “jobs”) for resubmission. Computing jobs can include requests to generate recurring reports according to a schedule. Information regarding failed jobs can be submitted to a job resubmission framework. Various parameters can be extracted from the failed job information and used to determine whether a job should be submitted for reexecution. Jobs that are to be submitted for reexecution can be placed in a queue or released according to a schedule. Determining whether a job should be submitted for reexecution can include determining a relative priority of reexecuting a particular job compared with other jobs to be reexecuted. Disclosed techniques can help ensure that more important jobs are executed before lower priority jobs, as well as helping to avoid overloading a computing system with job reexecution processes.
1 . A computing system comprising:
at least one hardware processor;
at least one memory coupled to the at least one hardware processor; and
one or more computer-readable storage media storing computer-executable instructions that, when executed, cause the computing system to perform operations comprising:
receiving first job failure information for a first execution instance of a first scheduled job, the first scheduled job being periodically reexecuted according to a first schedule, the job failure information comprising structured error data;
parsing the structured error data of the first job failure information to obtain a first set of values for a first set of one or more job parameters, the first set of job parameters including at least an error code comprising a machine-generated code or message emitted by a job execution component that identifies a failure mode of the first execution instance;
determining, prior to a next scheduled execution of the first scheduled job, that the first execution instance of the first scheduled job is to be resubmitted for execution, and responsive thereto, issuing a resubmission instruction to a job execution component, the determining being based at least in part on the error code and on one or more of:
a downstream job dependency, including parsing the job failure information to identify one or more jobs dependent on the failed job and prioritizing resubmission of the failed job to enable execution of the dependent jobs;
a tenant associated with the job, including extracting a tenant identifier from the job failure information and evaluating a workload of the tenant to adjust timing or prioritization of resubmission of the first execution instance; or
an availability of computing resources, including detecting from the error code a resource exhaustion condition and monitoring current resource usage of the job server to determine a threshold level below which the first execution instance is to be resubmitted; and
causing, by issuing the resubmission instruction to a job execution component, the first execution instance of the first scheduled job to be reexecuted, wherein the job execution component reexecutes the first execution instance responsive to the resubmission instruction.
2 . The computing system of claim 1 , wherein the first set of one or more job parameters comprises a job failure time.
3 . The computing system of claim 1 , wherein the first set of one or more job parameters comprises a recurrence frequency of the first scheduled job.
4 . The computing system of claim 1 , wherein the first set of one or more job parameters comprises a time the first scheduled job will be next executed.
5 . The computing system of claim 1 , wherein the first set of one or more job parameters comprises a destination specified for the first scheduled job.
6 . The computing system of claim 5 , wherein the destination specifies a job execution component used in executing the first instance of the first scheduled job.
7 . The computing system of claim 5 , wherein the destination specifies a location to which job execution results will be sent.
8 . The computing system of claim 1 , wherein the first set of one or more job parameters comprises a tenant from which the first scheduled job originated.
9 . The computing system of claim 1 , wherein the first set of one or more job parameters comprises an indication of any jobs depending on the first scheduled job.
10 . The computing system of claim 1 , the operations further comprising:
determining an execution priority for reexecuting the first execution instance of the first scheduled job.
11 . The computing system of claim 10 , the operations further comprising:
enqueuing the first execution instance of the first scheduled job in a priority queue based at least in part on the execution priority.
12 . The computing system of claim 10 , wherein the determining an execution priority comprises analyzing a plurality of job parameters of the first set of one or more job parameters.
13 . The computing system of claim 1 , the operations further comprising:
determining a reexecution time for reexecuting the first execution instance of the first scheduled job; and
scheduling the first executing instance of the first scheduled job at the reexecution time.
14 . The computing system of claim 13 , wherein the determining a reexecution time comprises analyzing a plurality of job parameters of the first set of one or more job parameters.
15 . The computing system of claim 13 , wherein the determining that the first execution instance of the first scheduled job is to be resubmitted for execution is based at least in part on the availability of computing resources, including the detecting from the error code a resource exhaustion condition and monitoring current resource usage of the job server to determine a threshold level below which the first execution instance is to be resubmitted.
16 . The computing system of claim 1 , the operations further comprising:
receiving second job failure information for a first execution instance of a second scheduled job, wherein the first execution instance of the second scheduled job is a second instance of the first scheduled job or is a first execution instance of a scheduled job other than the first scheduled job; and
determining that the second execution instance of the second scheduled job is not to be reexecuted.
17 . The computing system of claim 16 , the operations further comprising:
sending a job failure alert in response to the determining that the second execution instance of the second scheduled job is not to be reexecuted.
18 . The method of claim 1 , further comprising:
determining an execution priority for reexecuting the first execution instance of the first scheduled job, wherein the determining an execution priority comprises analyzing a plurality of job parameters of the first set of one or more job parameters.
19 . A method, implemented in a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, the method comprising:
receiving first job failure information for a first execution instance of a first scheduled job, the first scheduled job being periodically reexecuted according to a first schedule, the job failure information comprising structured error data;
parsing the structured error data of the first job failure information to obtain a first set of values for a first set of one or more job parameters, the first set of job parameters including at least an error code comprising a machine-generated code or message emitted by a job execution component that identifies a failure mode of the first execution instance;
determining, prior to a next scheduled execution of the first scheduled job, that the first execution instance of the first scheduled job is to be resubmitted for execution, and responsive thereto, issuing a resubmission instruction to a job execution component, the determining being based at least in part on the error code and on one or more of:
a downstream job dependency, including parsing the job failure information to identify one or more jobs dependent on the failed job and prioritizing resubmission of the failed job to enable execution of the dependent jobs;
a tenant associated with the job, including extracting a tenant identifier from the job failure information and evaluating a workload of the tenant to adjust timing or prioritization of resubmission of the first execution instance; or
an availability of computing resources, including detecting from the error code a resource exhaustion condition and monitoring current resource usage of the job server to determine a threshold level below which the first execution instance is to be resubmitted; and
causing, by issuing the resubmission instruction to a job execution component, the first execution instance of the first scheduled job to be reexecuted, wherein the job execution component reexecutes the first execution instance responsive to the resubmission instruction.
20 . One or more non-transitory computer-readable storage media comprising:
computer-executable instructions that, when executed by a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, cause the computing system to receive first job failure information for a first execution instance of a first scheduled job, the first scheduled job being periodically reexecuted according to a first schedule, the job failure information comprising structured error data;
computer-executable instructions that, when executed by the computing system, cause the computing system to parse the structured error data of the first job failure information to obtain a first set of values for a first set of one or more job parameters, the first set of job parameters including at least an error code comprising a machine-generated code or message emitted by a job execution component that identifies a failure mode of the first execution instance;
computer-executable instructions that, when executed by the computing system, cause the computing system to determine, prior to a next scheduled execution of the first scheduled job, that the first execution instance of the first scheduled job is to be resubmitted for execution, and responsive thereto, issuing a resubmission instruction to a job execution component, the determining being based at least in part on the error code and on one or more of:
a downstream job dependency, including parsing the job failure information to identify one or more jobs dependent on the failed job and prioritizing resubmission of the failed job to enable execution of the dependent jobs;
a tenant associated with the job, including extracting a tenant identifier from the job failure information and evaluating a workload of the tenant to adjust timing or prioritization of resubmission of the first execution instance; or
an availability of computing resources, including detecting from the error code a resource exhaustion condition and monitoring current resource usage of the job server to determine a threshold level below which the first execution instance is to be resubmitted; and
computer-executable instructions that, when executed by the computing system, cause the computing system to cause, by issuing the resubmission instruction to a job execution component, the first execution instance of the first scheduled job to be reexecuted, wherein the job execution component reexecutes the first execution instance responsive to the resubmission instruction.