Parallel processing of data
A data parallel pipeline may specify multiple parallel data objects that contain multiple elements and multiple parallel operations that operate on the parallel data objects. Based on the data parallel pipeline, a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline may be generated and one or more graph transformations may be applied to the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations. The deferred, combined parallel operations may be executed to produce materialized parallel data objects corresponding to the deferred parallel data objects.
1. A computer implemented method, comprising:
obtaining multiple, parallel data objects;
obtaining multiple, parallel data operations for processing the multiple, parallel data objects;
generating at least one generalized mapreduce operation that includes multiple, parallel map operations and multiple, parallel reduce operations;
translating the at least one generalized mapreduce operation into a single mapreduce operation that includes a single map function to implement the multiple, parallel map operations and a single reduce function to implement the multiple, parallel reduce operations; and
executing the single mapreduce operation to produce as output a set of materialized parallel data objects associated with the obtained multiple, parallel data objects.
2. The computer implemented method of claim 1 , wherein generating at least one generalized mapreduce operation comprises constructing an internal directed acyclic dataflow graph that includes unevaluated multiple, parallel data operations and unevaluated multiple, parallel data objects.
3. The computer implemented method of claim 2 , wherein the unevaluated multiple, parallel data objects comprise data structures that include one or more pointers to one or more of the unevaluated multiple, parallel data operations that operate on a given one of the unevaluated multiple, parallel data objects.
4. The computer implemented method of claim 2 , wherein translating the generalized mapreduce operation comprises combining subgraphs of parallel operations in the internal directed acyclic dataflow graph to produce a revised data flow graph.
5. The computer implemented method of claim 4 , wherein executing the single mapreduce operation comprises traversing operations identified in the revised data flow graph in a forward topological order.
6. The computer implemented method of claim 5 , wherein executing the single mapreduce operation comprises traversing operations identified in the revised data flow graph in turn.
7. The computer implemented method of claim 1 , wherein executing the single mapreduce operation comprises instantiating the single mapreduce operation on a plurality of machines in a datacenter.
8. The computer implemented method of claim 7 , wherein executing the single mapreduce operation comprises instantiating multiple reduce worker processes for each reduce operation associated with the single mapreduce operation on one or more of the plurality of machines.
9. The computer implemented method of claim 1 , wherein the set of materialized parallel data objects comprise a set of evaluated data objects.
10. A system, comprising:
one or more computing devices; and
a memory storing instructions that when executed by the one or more computing devices cause the one or more computing devices to:
obtain multiple, parallel data objects;
obtain multiple, parallel data operations for processing the multiple, parallel data objects;
generate at least one generalized mapreduce operation that includes multiple, parallel map operations and multiple, parallel reduce operations;
translate the generalized mapreduce operation into a single mapreduce operation that includes a single map function to implement the multiple, parallel map operations and a single reduce function to implement the multiple, parallel reduce operations; and
execute the single mapreduce operation to produce as output a set of materialized parallel data objects associated with the obtained multiple, parallel data objects.
11. The system of claim 10 , wherein generate at least one generalized mapreduce operation comprises causing the one or more computing devices to construct an internal directed acyclic dataflow graph that includes unevaluated multiple, parallel data operations and unevaluated multiple, parallel data objects.
12. The system of claim 11 , wherein the unevaluated multiple, parallel data objects comprise data structures that include one or more pointers to one or more of the unevaluated multiple, parallel data operations that operate on a given one of unevaluated multiple, parallel data objects.
13. The system of claim 12 , wherein translate the generalized mapreduce operation comprises causing the one or more computing devices to combine subgraphs of parallel operations in the internal directed acyclic dataflow graph to produce a revised data flow graph.
14. The system of claim 13 , wherein execute the single mapreduce operation comprises causing the one or more computing devices to traverse operations identified in the revised data flow graph in a forward topological order.
15. The system of claim 14 , wherein execute the single mapreduce operation comprises causing the one or more computing devices to traverse operations identified in the revised data flow graph in turn.
16. The system of claim 10 , wherein execute the single mapreduce operation comprises causing the one or more computing devices to instantiate the single mapreduce operation on a plurality of machines in a datacenter.
17. The system of claim 16 , wherein execute the single mapreduce operation comprises causing the one or more computing devices to instantiate multiple reduce worker processes for each reduce operation associated with the single mapreduce operation on one or more of the plurality of machines.
18. The system of claim 10 , wherein the set of materialized parallel data objects comprise a set of evaluated data objects.
19. A non-transitory computer medium having stored thereon programming instructions that when executed by one or more computing devices, cause the one or more computing devices to perform a process comprising:
obtaining multiple, parallel data objects;
obtaining multiple, parallel data operations for processing the multiple, parallel data objects;
generating at least one generalized mapreduce operation that includes multiple, parallel map operations and multiple, parallel reduce operations;
translating the generalized mapreduce operation into a single mapreduce operation that includes a single map function to implement the multiple, parallel map operations and a single reduce function to implement the multiple, parallel reduce operations; and
executing the single mapreduce operation to produce as output a set of materialized parallel data objects associated with the obtained multiple, parallel data objects.