Checkpoints in batch file processing

1. A method comprising:

receiving, by one or more processors, an assignment of a batch from a query coordinator, the batch including a group of a set of files from a shared file queue;

storing a unique batch ID and information related to the batch in a staging area;

performing a first operator on the batch;

based on performing the first operator, generating a checkpoint signal;

determining a state condition of a second operator as being stateless;

based on the stateless condition of the second operator, performing the second operator and passing the checkpoint signal to a third operator without implementing barriers between batches;

determining a state condition of the third operator as being non-stateless;

based on the non-stateless condition of the third operator, clearing data associated with the third operator, implementing barriers between batches, and performing the third operator;

finalizing the stored information in the staging area;

generating a materialized result file as an output file associated with the batch;

uploading the materialized result file to a storage area from where the materialized result file is accessible to be scanned by the query coordinator; and

uploading a listing of the materialized result to an output shared file queue.

2. The method of claim 1 , further comprising:

generating a file registration request;

transmitting the file registration request to the query coordinator including the unique batch ID indicating that the batch has been processed; and

deleting the unique batch ID and information related to the batch stored in the staging area.

3. The method of claim 1 , wherein the first operator includes a table scan operator.

4. The method of claim 1 , wherein the third operator includes an insert operator.

5. The method of claim 1 , wherein the output file includes data corresponding only to the batch and no other batch.

6. A system comprising:

one or more processors of a machine; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:

receiving an assignment of a batch an assignment of a batch from a query coordinator, the batch including a group of a set of files from a shared file queue;

storing a unique batch ID and information related to the batch in a staging area;;

performing a first operator on the batch;

based on performing the first operator, generating a checkpoint signal;

determining a state condition of a second operator as being stateless;

based on the stateless condition of the second operator, performing the second operator and passing the checkpoint signal to a third operator without implementing barriers between batches;

determining a state condition of the third operator as being non-stateless;

based on the non-stateless condition of the third operator, clearing data associated with the third operator, implementing barriers between batches, and performing the third operator;

finalizing the stored information in the staging area;

generating a materialized result file as an output file associated with the batch;

uploading the materialized result file to a storage area from where the materialized result file is accessible to be scanned by the query coordinator; and

uploading a listing of the materialized result to an output shared file queue.

7. The system of claim 6 , the operations further comprising:

transmitting the output file associated with the batch;

generating a file registration request; and

deleting the unique batch ID and information related to the batch.

8. The system of claim 6 , wherein the first operator includes a table scan operator.

9. The system of claim 6 , wherein the third operator includes an insert operator.

10. The system of claim 6 , wherein the output file includes data corresponding only to the batch and no other batch.

11. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

receiving, by one or more processors, an assignment of a batch from a query coordinator, the batch including a group of a set of files from a shared file queue;

storing a unique batch ID and information related to the batch in a staging area;;

performing a first operator on the batch;

based on performing the first operator, generating a checkpoint signal;

determining a state condition of a second operator as being stateless;

based on the stateless condition of the second operator, performing the second operator and passing the checkpoint signal to a third operator without implementing barriers between batches;

determining a state condition of the third operator as being non-stateless;

based on the non-stateless condition of the third operator, clearing data associated with the third operator, implementing barriers between batches, and performing the third operator;

generating a materialized result file as an output file associated with the batch;

uploading the materialized result file to a storage area from where the materialized result file is accessible to be scanned by the query coordinator; and

uploading a listing of the materialized result to an output shared file queue.

12. The machine-storage medium of claim 11 , further comprising:

transmitting the output file associated with the batch;

generating a file registration request; and

deleting the unique batch ID and information related to the batch.

13. The machine-storage medium of claim 11 , wherein the first operator includes a table scan operator.

14. The machine-storage medium of claim 11 , wherein the third operator includes an insert operator.

15. The machine-storage medium of claim 11 , wherein the output file includes data corresponding only to the batch and no other batch.