Parallel processing of data
A data parallel pipeline may specify multiple parallel data objects that contain multiple elements and multiple parallel operations that operate on the parallel data objects. Based on the data parallel pipeline, a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline may be generated and one or more graph transformations may be applied to the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations. The deferred, combined parallel operations may be executed to produce materialized parallel data objects corresponding to the deferred parallel data objects.
1. A method of processing data, comprising:
receiving a plurality of data objects associated with a set of logical computations;
accessing a pipeline library to formulate the set of logical computations as multiple mapreduce operations based on one or more parallel data objects and one or more parallel operations associated with the pipeline library;
generating, using the pipeline library, a generalized mapreduce operation that includes one or more of the multiple mapreduce operations; and
implementing, using a mapreduce library, the generalized mapreduce operation.
2. The method of claim 1 , comprising translating the generalized mapreduce operation into a single mapreduce operation.
3. The method of claim 2 , comprising implementing the single mapreduce operation using the mapreduce library.
4. The method of claim 2 , wherein the multiple mapreduce operations is associated with a dataflow graph based on the one or more parallel data objects and one or more parallel operations.
5. The method of claim 4 , wherein translating the generalized mapreduce operation comprises applying one or more graph transformations to the dataflow graph to generate a revised data flow graph representing the single mapreduce operation.
6. The method of claim 5 , wherein the single mapreduce operation includes a single map function and a single reduce function.
7. The method of claim 1 , wherein the pipeline library includes a parallel data collection class including a immutable set of elements.
8. The method of claim 7 , wherein the parallel data collection class is defined as a PCollection<T>function in which the immutable set of elements are of type T.
9. The method of claim 1 , wherein implementing the generalized mapreduce operation comprises translating the generalized mapreduce operation into a single mapreduce operation and executing the single mapreduce operation.
10. The method of claim 9 , wherein the pipeline library includes an optimizer function for translating the generalized mapreduce operation into the single mapreduce operation and an executor function for executing the single mapreduce operation.
11. A data processing system, comprising:
a processing module having one or more processing units and a memory, the memory storing a mapreduce library and a pipeline library and instructions that cause the one or more processing units to:
receive a plurality of data objects associated with a set of logical computations;
access the pipeline library to formulate the set of logical computations as multiple mapreduce operations based on one or more parallel data objects and one or more parallel operations associated with the pipeline library;
generate, using the pipeline library, a generalized mapreduce operation that includes one or more of the multiple mapreduce operations; and
implement, using the mapreduce library, the generalized mapreduce operation.
12. The data processing system of claim 11 , wherein the instructions cause the one or more processing units to translate the generalized mapreduce operation into a single mapreduce operation.
13. The data processing system of claim 12 , wherein the instructions cause the one or more processing units to implement the single mapreduce operation using the mapreduce library.
14. The data processing system of claim 13 , wherein the multiple mapreduce operations is associated with a data flow graph based on the one or more parallel data objects and one or more parallel operations.
15. The data processing system of claim 14 , wherein to translate the generalized mapreduce operation comprises applying one or more graph transformations to the data flow graph to generate a revised data flow graph representing the single mapreduce operation.
16. The data processing system of claim 15 , wherein the single mapreduce operation includes a single map function and a single reduce function.
17. The data processing system of claim 11 , wherein the pipeline library includes a parallel data collection class including a immutable set of elements.
18. The data processing system of claim 17 , wherein the parallel data collection class is defined as a PCollection<T>function in which the immutable set of elements are of type T.
19. The data processing system of claim 11 , wherein to implement the generalized mapreduce operation comprises translating the generalized mapreduce operation into a single mapreduce operation and executing the single mapreduce operation.
20. The data processing system of claim 19 , wherein the pipeline library includes an optimizer function for translating the generalized mapreduce operation into the single mapreduce operation and an executor function for executing the single mapreduce operation.