Parallel processing of data
A data parallel pipeline may specify multiple parallel data objects that contain multiple elements and multiple parallel operations that operate on the parallel data objects. Based on the data parallel pipeline, a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline may be generated and one or more graph transformations may be applied to the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations. The deferred, combined parallel operations may be executed to produce materialized parallel data objects corresponding to the deferred parallel data objects.
1. A computer-implemented method, comprising:
executing a deferred, combined parallel operation, which is included in a dataflow graph that comprises deferred parallel data objects and deferred, combined parallel operations corresponding to a data parallel pipeline, to produce materialized parallel data objects corresponding to deferred parallel data objects, wherein the executing comprises:
determining an estimated size of data associated with the deferred, combined parallel operation being executed;
determining that the estimated size of data associated with the deferred, combined parallel operation does not exceed a threshold size based at least on accessing annotations in the dataflow graph that represent an estimate of the size of the data associated with the deferred, combined parallel operation; and
in response to determining that the estimated size does not exceed the threshold size, executing the deferred, combined parallel operation as a local, sequential operation.
2. The method of claim 1 , wherein executing the deferred, combined parallel operation as a local, sequential operation comprises:
evaluating each of the deferred, combined parallel operations with a local sequential loop.
3. The method of claim 1 , wherein the deferred, combined parallel operation is a first deferred, combined parallel operation and the dataflow graph includes a second deferred, combined parallel operation, the method further comprising:
executing the second deferred, combined parallel operation to produce materialized data objects corresponding to the second deferred, combined parallel operation, wherein the executing comprises:
determining an estimated size of data associated with the second deferred, combined parallel operation;
determining that the estimated size associated with the deferred, combined parallel operation exceeds the threshold size; and
in response to determining that the estimated size exceeds the threshold size, executing the second deferred, combined parallel operation as a remote, parallel operation.
4. The method of claim 1 , wherein the materialized data objects are configured to be accessed during execution of a program corresponding to sequential programming language instructions associated with the data parallel pipeline.
5. The method of claim 4 , comprising determining the dataflow graph for the data parallel pipeline by analyzing the sequential programming language instructions associated with the data parallel pipeline.
6. The method of claim 1 , wherein the materialized data objects comprise one or more of: input data for the deferred, combined parallel operation, intermediary data produced by the deferred, combined parallel operation, or output data produced by the deferred, combined parallel operation.
7. The method of claim 1 , wherein:
the deferred, combined parallel operation is a generalized mapreduce operation comprising multiple, parallel map operations and multiple, parallel reduce operations and is translatable to a single mapreduce operation that comprises a single map function to implement the multiple, parallel map operations and a single reduce function to implement the multiple, parallel reduce operations; and
executing the deferred, combined parallel operation comprises translating the generalized mapreduce operation to the single mapreduce operation and executing the single mapreduce operation.
8. A system comprising:
one or more computers; and
memory storing instructions that, upon being executed by the one or more computers, cause the one or more computers to perform operations comprising:
executing a deferred, combined parallel operation, which is included in a dataflow graph that comprises deferred parallel data objects and deferred, combined parallel operations corresponding to a data parallel pipeline, to produce materialized parallel data objects corresponding to deferred parallel data objects, wherein the executing comprises:
determining an estimated size of data associated with the deferred, combined parallel operation being executed;
determining that the estimated size of data associated with the deferred, combined parallel operation does not exceed a threshold size based at least on accessing annotations in the dataflow graph that represent an estimate of the size of the data associated with the deferred, combined parallel operation; and
in response to determining that the estimated size does not exceed the threshold size, executing the deferred, combined parallel operation as a local, sequential operation.
9. The system of claim 8 , wherein executing the deferred, combined parallel operation as a local, sequential operation comprises:
evaluating each of the deferred, combined parallel operations with a local sequential loop.
10. The system of claim 8 , wherein the deferred, combined parallel operation is a first deferred, combined parallel operation and the dataflow graph includes a second deferred, combined parallel operation, the operations further comprising:
executing the second deferred, combined parallel operation to produce materialized data objects corresponding to the second deferred, combined parallel operation, wherein the executing comprises:
determining an estimated size of data associated with the second deferred, combined parallel operation;
determining that the estimated size associated with the deferred, combined parallel operation exceeds the threshold size; and
in response to determining that the estimated size exceeds the threshold size, executing the second deferred, combined parallel operation as a remote, parallel operation.
11. The system of claim 8 , wherein the materialized data objects are configured to be accessed during execution of a program corresponding to sequential programming language instructions associated with the data parallel pipeline.
12. The system of claim 11 , the operations comprising determining the dataflow graph for the data parallel pipeline by analyzing the sequential programming language instructions associated with the data parallel pipeline.
13. The system of claim 8 , wherein the materialized data objects comprise one or more of: input data for the deferred, combined parallel operation, intermediary data produced by the deferred, combined parallel operation, or output data produced by the deferred, combined parallel operation.
14. The system of claim 8 , wherein:
the deferred, combined parallel operation is a generalized mapreduce operation comprising multiple, parallel map operations and multiple, parallel reduce operations and is translatable to a single mapreduce operation that comprises a single map function to implement the multiple, parallel map operations and a single reduce function to implement the multiple, parallel reduce operations; and
executing the deferred, combined parallel operation comprises translating the generalized mapreduce operation to the single mapreduce operation and executing the single mapreduce operation.
15. A computer-readable storage device storing instructions that, upon execution by one or more processors, cause the one or more processors to perform operations comprising:
executing a deferred, combined parallel operation, which is included in a dataflow graph that comprises deferred parallel data objects and deferred, combined parallel operations corresponding to a data parallel pipeline, to produce materialized parallel data objects corresponding to deferred parallel data objects, wherein the executing comprises:
determining an estimated size of data associated with the deferred, combined parallel operation being executed;
determining that the estimated size of data associated with the deferred, combined parallel operation does not exceed a threshold size based at least on accessing annotations in the dataflow graph that represent an estimate of the size of the data associated with the deferred, combined parallel operation; and
in response to determining that the estimated size does not exceed the threshold size, executing the deferred, combined parallel operation as a local, sequential operation.
16. The computer-readable storage device of claim 15 , wherein executing the deferred, combined parallel operation as a local, sequential operation comprises:
evaluating each of the deferred, combined parallel operations with a local sequential loop.
17. The computer-readable storage device of claim 15 , wherein the deferred, combined parallel operation is a first deferred, combined parallel operation and the dataflow graph includes a second deferred, combined parallel operation, the operations further comprising:
executing the second deferred, combined parallel operation to produce materialized data objects corresponding to the second deferred, combined parallel operation, wherein the executing comprises:
determining an estimated size of data associated with the second deferred, combined parallel operation;
determining that the estimated size associated with the deferred, combined parallel operation exceeds the threshold size; and
in response to determining that the estimated size exceeds the threshold size, executing the second deferred, combined parallel operation as a remote, parallel operation.
18. The computer-readable storage device of claim 15 , wherein the materialized data objects are configured to be accessed during execution of a program corresponding to sequential programming language instructions associated with the data parallel pipeline.
19. The computer-readable storage device of claim 15 , wherein the materialized data objects comprise one or more of: input data for the deferred, combined parallel operation, intermediary data produced by the deferred, combined parallel operation, or output data produced by the deferred, combined parallel operation.
20. The computer-readable storage device of claim 15 , wherein
the deferred, combined parallel operation is a generalized mapreduce operation comprising multiple, parallel map operations and multiple, parallel reduce operations and is translatable to a single mapreduce operation that comprises a single map function to implement the multiple, parallel map operations and a single reduce function to implement the multiple, parallel reduce operations; and
executing the deferred, combined parallel operation comprises translating the generalized mapreduce operation to the single mapreduce operation and executing the single mapreduce operation.