Error handling during asynchronous processing of sequential data blocks
A data analytics system stores a data file that includes an ordered set of data blocks. The data blocks can be parsed out of order. An error management module of the data analytics system detects a parse error occurring during parsing of a data block and generates an error message for the parse error. The error message includes unresolved location information indicating a location of the detected parse error in the data block. The error management module resolves the unresolved location information after determining that one or more additional data blocks preceding the data block in the ordered set have been parsed. The error management module generates resolved location information that indicates a location of the parse error in the data file. The error management module updates the error message with the resolved location information and outputs the updated error message.
1. A computer implemented method comprising:
parsing a data block of a data file, the data file comprising a sequence of data blocks that include the data block and one or more preceding data blocks that precede the data block in the sequence, wherein the parsing of the data block occurs before parsing of the one or more preceding data blocks;
detecting a parse error occurring during the parsing of the data block;
generating an error message for the detected parse error, the generated error message including unresolved location information indicating a location of the detected parse error in the data block;
determining whether the one or more preceding data blocks have been parsed;
in response to a determination that the one or more preceding data blocks have been parsed, generating resolved location information indicating the location of the detected parse error in the data file;
updating the stored error message with the resolved location information to produce an updated error message; and
outputting the updated error message including the resolved location information indicating the location of the detected parse error in the data file.
2. The method of claim 1 , wherein generating an error message for the detected parse error, the generated error message including unresolved location information representing a location of the detected parse error in the data block comprises:
determining a line number referencing a line in the data block where the detected parse error occurs.
3. The method of claim 1 , wherein determining whether the one or more preceding data blocks have been parsed comprises:
determining whether the data block is a lowest numbered data block in the sequence for which parsing has not been completed.
4. The method of claim 1 , wherein generating the resolved location information indicating the location of the detected parse error in the data file comprises:
generating the resolved location information based on the unresolved location information, result of the parsing of the data block, and result of the parsing of the one or more preceding data blocks.
5. The method of claim 1 , wherein outputting the updated error message including the resolved location information indicating the location of the detected parse error in the data file comprises:
outputting the updated error message and one or more other error messages for errors detected during parsing of one or more additional data blocks preceding the data block in the sequence.
6. A non-transitory computer-readable memory storing executable computer program instructions, the instructions executable to perform operations comprising:
parsing a data block of a data file, the data file comprising a sequence of data blocks that include the data block and one or more preceding data blocks that precede the data block in the sequence, wherein the parsing of the data block occurs before parsing of the one or more preceding data blocks;
detecting a parse error occurring during the parsing of the data block;
generating an error message for the detected parse error, the generated error message including unresolved location information indicating a location of the detected parse error in the data block;
determining whether the one or more preceding data blocks have been parsed;
in response to a determination that the one or more preceding data blocks have been parsed, generating resolved location information indicating the location of the detected parse error in the data file;
updating the stored error message with the resolved location information to produce an updated error message; and
outputting the updated error message including the resolved location information indicating the location of the detected parse error in the data file.
7. The non-transitory computer-readable memory of claim 6 , wherein generating an error message for the detected parse error, the generated error message including unresolved location information representing a location of the detected parse error in the data block comprises:
determining a line number referencing a line in the data block where the detected parse error occurs.
8. The non-transitory computer-readable memory of claim 6 , wherein determining whether the one or more preceding data blocks have been parsed comprises:
determining whether the data block is a lowest numbered data block in the sequence for which parsing has not been completed.
9. The non-transitory computer-readable memory of claim 6 , wherein generating the resolved location information indicating the location of the detected parse error in the data file comprises:
generating the resolved location information based on the unresolved location information, result of the parsing of the data block, and result of the parsing of the one or more preceding data blocks.
10. The non-transitory computer-readable memory of claim 6 , wherein outputting the updated error message including the resolved location information indicating the location of the detected parse error in the data file comprises:
outputting the updated error message and one or more other error messages for errors detected during parsing of one or more additional data blocks preceding the data block in the sequence.
11. A computer-implemented data analytics system for processing data blocks, comprising:
a computer processor for executing computer program instructions; and
a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising:
parsing a data block of a data file, the data file comprising a sequence of data blocks that include the data block and one or more preceding data blocks that precede the data block in the sequence, wherein the parsing of the data block occurs before parsing of the one or more preceding data blocks;
detecting a parse error occurring during the parsing of the data block;
generating an error message for the detected parse error, the generated error message including unresolved location information indicating a location of the detected parse error in the data block;
determining whether the one or more preceding data blocks have been parsed;
in response to a determination that the one or more preceding data blocks have been parsed, generating resolved location information indicating the location of the detected parse error in the data file;
updating the stored error message with the resolved location information to produce an updated error message; and
outputting the updated error message including the resolved location information indicating the location of the detected parse error in the data file.
12. The system of claim 11 , wherein generating an error message for the detected parse error, the generated error message including unresolved location information representing a location of the detected parse error in the data block comprises:
determining a line number referencing a line in the data block where the detected parse error occurs.
13. The system of claim 11 , wherein determining whether the one or more preceding data blocks have been parsed comprises:
determining whether the data block is a lowest numbered data block in the sequence for which parsing has not been completed.
14. The system of claim 11 , wherein generating the resolved location information indicating the location of the detected parse error in the data file comprises:
generating the resolved location information based on the unresolved location information, result of the parsing of the data block, and result of the parsing of the one or more preceding data blocks.