Error handling during asynchronous processing of sequential data blocks
A data analytics system stores a data file that includes an ordered set of data blocks. The data blocks can be parsed out of order. An error management module of the data analytics system detects a parse error occurring during parsing of a data block and generates an error message for the parse error. The error message includes unresolved location information indicating a location of the detected parse error in the data block. The error management module resolves the unresolved location information after determining that one or more additional data blocks preceding the data block in the ordered set have been parsed. The error management module generates resolved location information that indicates a location of the parse error in the data file. The error management module updates the error message with the resolved location information and outputs the updated error message.
1. A computer implemented method comprising:
detecting a parse error occurring during parsing of a data block of a sequence of data blocks, the sequence of data blocks including the data block and a preceding data block that precedes the data block in the sequence, wherein the parsing of the data block occurs out of order before parsing of the preceding data block;
generating unresolved location information for the parse error, the unresolved location information for the parse error indicating a relative location of the parse error within the data block, wherein the unresolved location information includes at least one of a relative line number in the data block, a relative record number in the data block, or a block number referencing the data block;
generating resolved location information for the parse error based on the unresolved location information for the parse error and a parsing result of the preceding data block, the resolved location information for the parse error indicating an absolute location of the parse error within the sequence of data blocks;
generating an error message that includes the resolved location information for the parse error; and
outputting the error message.
2. The computer implemented method of claim 1 , further comprising
parsing the data block, a parsing result of the data block including records in the data block,
wherein generating the unresolved location information for the parse error comprises determining a record number referencing one of the records in the data block based on the parsing result.
3. The computer implemented method of claim 1 , further comprising:
determining whether the preceding data block has been parsed; and
in response to determine that the preceding data block has been parsed, generating the resolved location information for the parse error.
4. The computer implemented method of claim 3 , wherein determining whether the preceding data block has been parsed comprises:
determining whether the data block is a lowest numbered data block in the sequence for which parsing has not been completed.
5. The computer implemented method of claim 3 , wherein determining whether the preceding data block has been parsed comprises:
retrieving information indicating whether the data blocks in the sequence have been parsed or not been parsed from a buffer; and
using the retrieved information to determine whether the preceding data block has been parsed.
6. The computer implemented method of claim 1 , wherein the parsing result of the preceding data block includes records produced during the parsing of the preceding data block and generating the resolved location information for the parse error based on the unresolved location information of the parse error and the parsing result of the preceding data block comprises:
determining a record number referencing an absolute location of a record within the sequence of data blocks by using the records produced during the parsing of the preceding data block, the detected parse error having occurred in the record.
7. The computer implemented method of claim 6 , wherein determining the record number referencing the location of the record within the sequence of data blocks where the detected parse error occurs by using the records produced during the parsing of the preceding data block comprises:
aggregating a number of the records produced during the parsing of the preceding data block with a record number referencing a location of the record within the data block where the detected parse error occurs; and
determining the record number referencing the location of the record within the sequence of data blocks based on the aggregated number.
8. A non-transitory computer-readable memory storing executable computer program instructions, the instructions executable to perform operations comprising:
detecting a parse error occurring during parsing of a data block of a sequence of data blocks, the sequence of data blocks including the data block and a preceding data block that precedes the data block in the sequence, wherein the parsing of the data block occurs out of order before parsing of the preceding data block;
generating unresolved location information for the parse error, the unresolved location information for the parse error indicating a relative location of the parse error within the data block, wherein the unresolved location information includes at least one of a relative line number in the data block, a relative record number in the data block, or a block number referencing the data block;
generating resolved location information for the parse error based on the unresolved location information for the parse error and a parsing result of the preceding data block, the resolved location information for the parse error indicating an absolute location of the parse error within the sequence of data blocks;
generating an error message that includes the resolved location information for the parse error; and
outputting the error message.
9. The non-transitory computer-readable memory of claim 8 , wherein the operations further comprise:
parsing the data block, a parsing result of the data block including records in the data block,
wherein generating the unresolved location information for the parse error comprises determining a record number referencing one of the records in the data block based on the parsing result.
10. The non-transitory computer-readable memory of claim 8 , wherein the operations further comprise:
determining whether the preceding data block has been parsed; and
in response to determine that the preceding data block has been parsed, generating the resolved location information for the parse error.
11. The non-transitory computer-readable memory of claim 10 , wherein determining whether the preceding data block has been parsed comprises:
determining whether the data block is a lowest numbered data block in the sequence for which parsing has not been completed.
12. The non-transitory computer-readable memory of claim 10 , wherein determining whether the preceding data block has been parsed comprises:
retrieving information indicating whether the data blocks in the sequence have been parsed or not been parsed from a buffer; and
using the retrieved information to determine whether the preceding data block has been parsed.
13. The non-transitory computer-readable memory of claim 8 , wherein the parsing result of the preceding data block includes records produced during the parsing of the preceding data block and generating the resolved location information for the parse error based on the unresolved location information of the parse error and the parsing result of the preceding data block comprises:
determining a record number referencing an absolute location of a record within the sequence of data blocks by using the records produced during the parsing of the preceding data block, the detected parse error having occurred in the record.
14. The non-transitory computer-readable memory of claim 13 , wherein determining the record number referencing the location of the record within the sequence of data blocks where the detected parse error occurs by using the records produced during the parsing of the preceding data block comprises:
aggregating a number of the records produced during the parsing of the preceding data block with a record number referencing a location of the record within the data block where the detected parse error occurs; and
determining the record number referencing the location of the record within the sequence of data blocks based on the aggregated number.
15. A computer system, comprising:
a computer processor for executing computer program instructions; and
a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising:
detecting a parse error occurring during parsing of a data block of a sequence of data blocks, the sequence of data blocks including the data block and a preceding data block that precedes the data block in the sequence, wherein the parsing of the data block occurs out of order before parsing of the preceding data block;
generating unresolved location information for the parse error, the unresolved location information for the parse error indicating a relative location of the parse error within the data block, wherein the unresolved location information includes at least one of a relative line number in the data block, a relative record number in the data block, or a block number referencing the data block;
generating resolved location information for the parse error based on the unresolved location information for the parse error and a parsing result of the preceding data block, the resolved location information for the parse error indicating an absolute location of the parse error within the sequence of data blocks;
generating an error message that includes the resolved location information for the parse error; and
outputting the error message.
16. The computer system of claim 15 , wherein the operations further comprise:
parsing the data block, a parsing result of the data block including records in the data block,
wherein generating the unresolved location information for the parse error comprises determining a record number referencing one of the records in the data block based on the parsing result.
17. The computer system of claim 15 , wherein the operations further comprise:
determining whether the preceding data block has been parsed; and
in response to determine that the preceding data block has been parsed, generating the resolved location information for the parse error.
18. The computer system of claim 17 , wherein determining whether the preceding data block has been parsed comprises:
retrieving information indicating whether the data blocks in the sequence have been parsed or not been parsed from a buffer; and
using the retrieved information to determine whether the preceding data block has been parsed.
19. The computer system of claim 17 , wherein determining whether the preceding data block has been parsed comprises:
determining whether the data block is a lowest numbered data block in the sequence for which parsing has not been completed.
20. The computer system of claim 15 , wherein the parsing result of the preceding data block includes records produced during the parsing of the preceding data block and generating the resolved location information for the parse error based on the unresolved location information of the parse error and the parsing result of the preceding data block comprises:
determining a record number referencing an absolute location of a record within the sequence of data blocks by using the records produced during the parsing of the preceding data block, the detected parse error having occurred in the record.