Checkpoints in batch file processing
Embodiments of the present disclosure may provide a dynamic query execution model with fault tolerance and failure recovery techniques. Embodiments of the present disclosure may utilize checkpoints to map processed output files to their corresponding input files. Therefore, if an error occurs in processing one or more files, the system may only need to reschedule processing of selected file(s).
1. A method comprising:
receiving, by one or more shared computing resources of a data system, an assignment of a batch from a query coordinator of the data system, the batch including a set of files;
performing, by the one or more shared computing resources, a first operator on the batch;
based on performing the first operator, generating a checkpoint signal;
determining a state condition of a second operator as being stateless;
based on the stateless condition of the second operator, performing, by the one or more shared computing resources, the second operator and passing the checkpoint signal to a third operator without implementing barriers between batches;
determining a state condition of the third operator as being non-stateless;
based on the non-stateless condition of the third operator, clearing, by the one or more shared computing resources, data associated with the third operator, implementing barriers between batches, and performing the third operator; and
generating an output file associated with the batch;
generating a materialized result file as the output file associated with the batch;
uploading the materialized result file to a storage area from where the materialized result file is accessible to be scanned by a query coordinator.
2. The method of claim 1 , further comprising:
uploading a listing of the materialized result to an output shared file queue.
3. The method of claim 1 , further comprising:
transmitting the output file associated with the batch;
generating a file registration request; and
deleting a unique batch ID and information related to the batch.
4. The method of claim 1 , wherein the first operator includes a table scan operator.
5. The method of claim 1 , wherein the third operator includes an insert operator.
6. The method of claim 1 , wherein the output file includes data corresponding only to the batch and no other batch.
7. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving, by one or more shared computing resources of a data system, an assignment of a batch from a query coordinator of the data system, the batch including a set of files;
performing, by the one or more shared computing resources, a first operator on the batch;
based on performing the first operator, generating a checkpoint signal;
determining a state condition of a second operator as being stateless;
based on the stateless condition of the second operator, performing, by the one or more shared computing resources, the second operator and passing the checkpoint signal to a third operator without implementing barriers between batches;
determining a state condition of the third operator as being non-stateless;
based on the non-stateless condition of the third operator, clearing, by the one or more shared computing resources, data associated with the third operator, implementing barriers between batches, and performing the third operator; and
generating an output file associated with the batch;
generating a materialized result file as the output file associated with the batch;
uploading the materialized result file to a storage area from where the materialized result file is accessible to be scanned by a query coordinator.
8. The system of claim 7 , further comprising:
uploading a listing of the materialized result to an output shared file queue.
9. The system of claim 7 , further comprising:
transmitting the output file associated with the batch;
generating a file registration request; and
deleting a unique batch ID and information related to the batch.
10. The system of claim 7 , wherein the first operator includes a table scan operator.
11. The system of claim 7 , wherein the third operator includes an insert operator.
12. The system of claim 7 , wherein the output file includes data corresponding only to the batch and no other batch.
13. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving, by one or more shared computing resources of a data system, an assignment of a batch from a query coordinator of the data system, the batch including a set of files;
performing, by the one or more shared computing resources, a first operator on the batch;
based on performing the first operator, generating a checkpoint signal;
determining a state condition of a second operator as being stateless;
based on the stateless condition of the second operator, performing, by the one or more shared computing resources, the second operator and passing the checkpoint signal to a third operator without implementing barriers between batches;
determining a state condition of the third operator as being non-stateless;
based on the non-stateless condition of the third operator, clearing, by the one or more shared computing resources, data associated with the third operator, implementing barriers between batches, and performing the third operator; and
generating an output file associated with the batch;
generating a materialized result file as the output file associated with the batch;
uploading the materialized result file to a storage area from where the materialized result file is accessible to be scanned by a query coordinator.
14. The machine-storage medium of claim 13 , further comprising:
uploading a listing of the materialized result to an output shared file queue.
15. The machine-storage medium of claim 13 , further comprising:
transmitting the output file associated with the batch;
generating a file registration request; and
deleting a unique batch ID and information related to the batch.
16. The machine-storage medium of claim 13 , wherein the first operator includes a table scan operator.
17. The machine-storage medium of claim 13 , wherein the third operator includes an insert operator.
18. The machine-storage medium of claim 13 , wherein the output file includes data corresponding only to the batch and no other batch.