System and method for automated recovery after an error in a batch processing system caused by malformatted or unsupported data
A system and method for automated recovery of processing of a unit of work during an error in a batch processing system is disclosed. Generally, at least a portion of a unit of work and instructions of operations to perform to process the portion of the unit of work is sent to a worker data structure. A periodic heartbeat is received from the worker data structure indicating the worker data structure is processing the at least a portion of the unit of work. If an unexpected termination of the worker data structure is detected, a signal is sent to a crash handler data structure instructing the crash handler data structure to detect and store a current input location of the at a portion of the unit of work. The records from the current location at the crash are skipped during reprocessing of the unit of work to increase the chances of success during reprocessing.
1. A method for automated recovery of processing of a unit of work during an error in a batch processing system, comprising:
sending at least a portion of a unit of work and instructions of operations to perform to process the at least a portion of the unit of work to a worker data structure;
receiving a periodic heartbeat signal from the worker data structure indicating the worker data structure is processing the at least a portion of the unit of work;
detecting an unexpected interruption in the heartbeat signal from the worker data structure; and
sending a signal to a crash handler data structure instructing the crash handler data structure to detect and store a current input record location of the at least a portion of the unit of work.
2. The method of claim 1 , wherein a worker process failure caused the unexpected interruption in the heartbeat signal.
3. The method of claim 2 , wherein the current input record location is an identifier of a record the worker data structure was processing when the worker process failure occurred.
4. The method of claim 3 , wherein the crash handler data structure stores the current input record location in a task package of the unit of work.
5. The method of claim 4 , further comprising:
recording a number of records that have caused worker process failures in the task package;
comparing the number of records that have caused worker process failure to a predetermined threshold; and
determining whether to remove the unit of work from processing based on the comparison of the number of records that have caused worker process failures and the predetermined threshold.
6. The method of claim 5 , wherein determining whether to remove the unit of work from processing based on the comparison of the number of records that have caused worker process failures and the predetermined threshold comprise:
removing the unit of work from processing in response to determining the number of records that have caused worker process failures exceeds the predetermined threshold.
7. The method of claim 5 , wherein determining whether to remove the unit of work from processing based on the comparison of the number of records that have caused worker process failures and the predetermined threshold comprise:
resubmitting the unit of work for processing in response to determining the number of records that have caused worker process failures does not exceed the predetermined threshold.
8. The method of claim 7 , further comprising:
skipping each record identified in the task package as causing a worker process failure during subsequent processing of the unit of work.
9. The method of claim 5 , wherein the predetermined threshold comprises a maximum number of records that may be removed from a unit of work.
10. The method of claim 5 , wherein the predetermined threshold comprises a maximum percentage of records of the total number of records in a unit of work that may be removed from the unit of work.
11. An apparatus operative for automated recovery of processing a unit of work comprising:
a task agent data structure operative to receive a unit of work and determine one or more operations that need to be performed to process the unit of work;
a worker data structure operative to receive at least a portion of the unit of work and instructions for one or more operations to perform on the at least a portion of the unit of work from the task agent data structure, and to send a periodic heartbeat signal to the task agent data structure during processing of the at least a portion of the unit of work;
a crash handler data structure operative to receive a signal from the task agent data structure, and detect and store a current input location of the worker data structure in response to receiving the signal from the task agent;
wherein the task agent data structure sends the signal to the crash handler in response to detecting an unexpected interruption in the heartbeat signal from the worker data structure.
12. The apparatus of claim 11 , wherein the worker data structure is operative to interrupt the periodic heartbeat signal to the task agent data structure in response to experiencing a worker process failure.
13. The apparatus of claim 12 , wherein the current input record location is an identifier of a record the worker data structure was processing when the worker process failure occurred.
14. The apparatus of claim 13 , wherein the crash handler data structure is operative to store the current input location in a task package of the unit of work.
15. The apparatus of claim 13 , further comprising:
a task queue data structure operative to receive a signal from the task agent data structure indicating a worker process error in processing the unit of work, and in response to receiving the signal from the data structure, determine a number of records of the unit of work that have caused a worker process failure, compare the number of records of the unit of work that have caused a worker process failure to a predetermined threshold, and determine whether to remove the unit of work from processing based on the comparison of the number of records of the unit of work that have caused a worker process failure and the predetermined threshold.
16. The apparatus of claim 15 , wherein the task queue data structure is operative to remove the unit of work from processing in response to determining the number of records that have caused a worker process failure exceeds the predetermined threshold.
17. The apparatus of claim 15 , wherein the task queue data structure is operative to resubmit the unit of work for processing in response to determining the number of records that have caused a worker process failure does not exceed the predetermined threshold.
18. The apparatus of claim 15 , wherein the predetermined threshold comprises a maximum number of records that may be removed from the unit of work.
19. The apparatus of claim 15 , wherein the predetermined threshold comprises a maximum percentage of records of the total number of records in the unit of work that may be removed from the unit of work.
20. A computer-readable storage medium comprising a set of instructions for automated recovery of processing of a unit of work during an error in a batch processing system, the set of instructions to direct a computer system to perform acts of:
sending at least a portion of a unit of work and instructions of operations to perform to process the at least a portion of the unit of work to a worker data structure;
receiving a periodic heartbeat signal from the worker data structure indicating the worker data structure is processing the at least a portion of the unit of work;
detecting an unexpected interruption in the heartbeat signal from the worker data structure; and
sending a signal to a crash handler data structure instructing the crash handler data structure to detect and store a current input record location of the at least a portion of the unit of work.
21. The computer-readable storage medium of claim 20 , wherein a worker process failure caused the unexpected interruption in the heartbeat signal, the current input record location is an identifier of a record the worker data structure was processing when the worker process failure occurred, and the crash handler stores the current input record location in a task package of the unit of work.
22. The computer-readable storage medium of claim 21 , further comprising a set of instructions to direct the computer system to perform acts of:
recording a number of records that have caused worker process failures in the task package;
comparing the number of records that have caused worker process failure to a predetermined threshold; and
determining whether to remove the unit of work from processing based on the comparison of the number of records that have caused worker process failures and the predetermined threshold.
23. The computer-readable storage medium of claim 22 , wherein determining whether to remove the unit of work from processing based on the comparison of the number of records that have caused worker process failures and the predetermined threshold comprise:
removing the unit of work from processing in response to determining the number of records that have caused worker process failures exceeds the predetermined threshold.
24. The computer-readable storage medium of claim 23 , wherein determining whether to remove the unit of work from processing based on the comparison of the number of records that have caused worker process failures and the predetermined threshold comprises:
resubmitting the unit of work for processing in response to determining the number of records that have caused worker process failures does not exceed the predetermined threshold; and
wherein the computer-readable storage medium further comprises a set of instructions to direct the computer system to perform acts of:
skipping each record identified in the task package as causing a worker process failure during subsequent processing of the unit of work.