Parallel processing of data
A data parallel pipeline may specify multiple parallel data objects that contain multiple elements and multiple parallel operations that operate on the parallel data objects. Based on the data parallel pipeline, a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline may be generated and one or more graph transformations may be applied to the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations. The deferred, combined parallel operations may be executed to produce materialized parallel data objects corresponding to the deferred parallel data objects.
1. A computer-implemented method comprising:
obtaining multiple, parallel map operations and multiple, parallel reduce operations;
determining indexes for each of multiple map workers and each of multiple reduce workers;
generating, based on the indexes, a single mapreduce operation that includes a map function that implements the multiple, parallel map operations and a reduce function that implements the multiple, parallel reduce operations,
wherein the map function specifies which map operation of the multiple, parallel map operations to perform by map workers based on the indexes of the map workers and specifies which reduce operation of the multiple, parallel map operations to perform by reduce workers based on the indexes of the reduce workers; and
executing the single mapreduce operation based on the map function, the reduce function, and the indexes.
2. The computer-implemented method of claim 1 , wherein executing the single mapreduce operation based on the map function, the reduce function, and the indexes comprises:
providing the map function to the map workers, wherein each of the map workers determines which of the map operations to perform based on the index of the map worker; and
providing the reduce function to the reduce workers, wherein each of the reduce workers determines which of the reduce operations to perform based on the index of the reduce worker.
3. The computer-implemented method of claim 1 , wherein determining indexes for each of multiple map workers and each of multiple reduce workers comprises:
for each of the map workers, assigning a unique number to the map worker.
4. The computer-implemented method of claim 1 , wherein generating, based on the indexes, a single mapreduce operation that includes a map function that implements the multiple, parallel map operations and a reduce function that implements the multiple, parallel reduce operations comprises:
generating the mapreduce operation such that the map function includes an if-then statement with the indexes as decision points to select one of the map operations.
5. The computer-implemented method of claim 4 , wherein indexes for two or more map workers correspond to a same map operation of the multiple, parallel map operations.
6. The computer-implemented method of claim 1 , wherein generating, based on the indexes, a single mapreduce operation that includes a map function that implements the multiple, parallel map operations and a reduce function that implements the multiple, parallel reduce operations comprises:
generating the mapreduce operation such that the reduce function includes an if-then statement with the indexes as decision points to select one of the reduce operations.
7. The computer-implemented method of claim 1 , wherein determining indexes for each of multiple map workers and each of multiple reduce workers comprises:
determining a number of the map workers to execute the single mapreduce operation based on a size of data associated with the single mapreduce operation.
8. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
obtaining multiple, parallel map operations and multiple, parallel reduce operations;
determining indexes for each of multiple map workers and each of multiple reduce workers;
generating, based on the indexes, a single mapreduce operation that includes a map function that implements the multiple, parallel map operations and a reduce function that implements the multiple, parallel reduce operations,
wherein the map function specifies which map operation of the multiple, parallel map operations to perform by map workers based on the indexes of the map workers and specifies which reduce operation of the multiple, parallel map operations to perform by reduce workers based on the indexes of the reduce workers; and
executing the single mapreduce operation based on the map function, the reduce function, and the indexes.
9. The system of claim 8 , wherein executing the single mapreduce operation based on the map function, the reduce function, and the indexes comprises:
providing the map function to the map workers, wherein each of the map workers determines which of the map operations to perform based on the index of the map worker; and
providing the reduce function to the reduce workers, wherein each of the reduce workers determines which of the reduce operations to perform based on the index of the reduce worker.
10. The system of claim 8 , wherein determining indexes for each of multiple map workers and each of multiple reduce workers comprises:
for each of the map workers, assigning a unique number to the map worker.
11. The system of claim 8 , wherein generating, based on the indexes, a single mapreduce operation that includes a map function that implements the multiple, parallel map operations and a reduce function that implements the multiple, parallel reduce operations comprises:
generating the mapreduce operation such that the map function includes an if-then statement with the indexes as decision points to select one of the map operations.
12. The system of claim 8 , wherein indexes for two or more map workers correspond to a same map operation of the multiple, parallel map operations.
13. The system of claim 8 , wherein generating, based on the indexes, a single mapreduce operation that includes a map function that implements the multiple, parallel map operations and a reduce function that implements the multiple, parallel reduce operations comprises:
generating the mapreduce operation such that the reduce function includes an if-then statement with the indexes as decision points to select one of the reduce operations.
14. The system of claim 8 , wherein determining indexes for each of multiple map workers and each of multiple reduce workers comprises:
determining a number of the map workers to execute the single mapreduce operation based on a size of data associated with the single mapreduce operation.
15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
obtaining multiple, parallel map operations and multiple, parallel reduce operations;
determining indexes for each of multiple map workers and each of multiple reduce workers;
generating, based on the indexes, a single mapreduce operation that includes a map function that implements the multiple, parallel map operations and a reduce function that implements the multiple, parallel reduce operations,
wherein the map function specifies which map operation of the multiple, parallel map operations to perform by map workers based on the indexes of the map workers and specifies which reduce operation of the multiple, parallel map operations to perform by reduce workers based on the indexes of the reduce workers; and
executing the single mapreduce operation based on the map function, the reduce function, and the indexes.
16. The medium of claim 15 , wherein executing the single mapreduce operation based on the map function, the reduce function, and the indexes comprises:
providing the map function to the map workers, wherein each of the map workers determines which of the map operations to perform based on the index of the map worker; and
providing the reduce function to the reduce workers, wherein each of the reduce workers determines which of the reduce operations to perform based on the index of the reduce worker.
17. The medium of claim 15 , wherein determining indexes for each of multiple map workers and each of multiple reduce workers comprises:
for each of the map workers, assigning a unique number to the map worker.
18. The medium of claim 15 , wherein generating, based on the indexes, a single mapreduce operation that includes a map function that implements the multiple, parallel map operations and a reduce function that implements the multiple, parallel reduce operations comprises:
generating the mapreduce operation such that the map function includes an if-then statement with the indexes as decision points to select one of the map operations.
19. The medium of claim 18 , wherein indexes for two or more map workers correspond to a same map operation of the multiple, parallel map operations.
20. The medium of claim 15 , wherein generating, based on the indexes, a single mapreduce operation that includes a map function that implements the multiple, parallel map operations and a reduce function that implements the multiple, parallel reduce operations comprises:
generating the mapreduce operation such that the reduce function includes an if-then statement with the indexes as decision points to select one of the reduce operations.