Parallel processing of data
A data parallel pipeline may specify multiple parallel data objects that contain multiple elements and multiple parallel operations that operate on the parallel data objects. Based on the data parallel pipeline, a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline may be generated and one or more graph transformations may be applied to the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations. The deferred, combined parallel operations may be executed to produce materialized parallel data objects corresponding to the deferred parallel data objects.
1. A computer implemented method comprising:
receiving a plurality of parallel data objects;
receiving a plurality of parallel data operations associated with the plurality of parallel data objects;
generating a single mapreduce operation that includes a single map function to implement a plurality of multiple, parallel map operations and a single reduce function to implement a plurality of multiple, parallel reduce operations; and
executing the single mapreduce operation to produce as output a set of materialized parallel data objects associated with the received parallel data objects.
2. The computer implemented method of claim 1 , wherein generating the single mapreduce operation comprises generating the single mapreduce operation from a generalized mapreduce operation that includes multiple, parallel map operations and multiple, parallel reduce operations.
3. The computer implemented method of claim 1 , wherein the single map function comprises reading a collection of key, value pairs associated with the plurality of parallel data objects from an input source.
4. The computer implemented method of claim 3 , wherein the single map function comprises shuffling the collection of key, value pairs.
5. The computer implemented method of claim 4 , further comprising grouping together the key, value pairs with a first key to form a key-grouped data.
6. The computer implemented method of claim 5 , wherein the single reduce function comprises aggregating values associated with a given key.
7. The computer implemented method of claim 1 , wherein executing the single mapreduce operation comprises causing the single mapreduce operation to be copied and executed on multiple processing modules in one or more datacenters.
8. The computer implemented method of claim 1 , wherein executing the single mapreduce operation comprises causing the single mapreduce operation to be executed by multiple worker processes on multiple processing modules in a datacenter.
9. A system comprising:
one or more computing devices;
a memory storing instructions that when executed by the one or more computing devices caused the one or more computing devices to:
receive a plurality of parallel data objects;
receive a plurality of parallel data operations associated with the received plurality of parallel data objects;
generate a single mapreduce operation that includes a single map function to implement a plurality of multiple, parallel map operations and a single reduce function to implement a plurality of multiple, parallel reduce operations; and
execute the single mapreduce operation to produce as output a set of materialized parallel data objects associated with the received parallel data objects.
10. The system of claim 9 , wherein the instructions to generate the single mapreduce operation comprises generate the single mapreduce operation from a generalized mapreduce operation that includes multiple, parallel map operations and multiple, parallel reduce operations.
11. The system of claim 9 , wherein the single map function comprises a function to read a collection of key, value pairs associated with the plurality of parallel data objects from an input source.
12. The system of claim 11 , wherein the single map function comprises a function to shuffle the collection of key, value pairs.
13. The system of claim 12 , further wherein the instructions comprise instructions to group together the key, value pairs with a first key to form a key-grouped data.
14. The system of claim 13 , wherein the single reduce function comprises a function to aggregate values associated with a given key.
15. The system of claim 9 , wherein the instructions to execute the single mapreduce operation comprise instructions that cause the single mapreduce operation to be copied and executed on multiple processing modules in one or more datacenters.
16. The system of claim 9 , wherein the instructions to execute the single mapreduce operation comprise instructions that cause the single mapreduce operation to be executed by multiple worker processes on multiple processing modules in a datacenter.