Parallel processing of data
A data parallel pipeline may specify multiple parallel data objects that contain multiple elements and multiple parallel operations that operate on the parallel data objects. Based on the data parallel pipeline, a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline may be generated and one or more graph transformations may be applied to the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations. The deferred, combined parallel operations may be executed to produce materialized parallel data objects corresponding to the deferred parallel data objects.
1. A method comprising:
receiving an untrusted application that includes a data parallel pipeline, wherein the data parallel pipeline specifies multiple parallel data objects and multiple parallel data operations;
instantiating a first secured processing environment in a native processing environment and on one or more processing modules;
executing the untrusted application to generate a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline;
communicating information representing the data-flow graph outside of the first secured processing environment;
applying, in the native processing environment, one or more graph transformations to the information representing the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations that are associated with the untrusted application; and
executing the deferred, combined parallel operations to produce materialized parallel data objects corresponding to the deferred parallel data objects.
2. The method of claim 1 , wherein applying one or more graph transformations comprise combining chains of parallel operations together into a smaller number of combined operations.
3. The method of claim 1 , wherein the first secured processing environment comprises a first virtual machine.
4. The method of claim 1 , wherein the deferred, combined parallel data operations includes at least one generalized mapreduce operation, the generalized mapreduce operation including multiple, parallel map operations and multiple, parallel reduce operations and being translatable to a single mapreduce operation that includes a single map function to implement the multiple, parallel map operations and a single reduce function to implement the multiple, parallel reduce operations, the single map function and the single reduce function including one or more untrusted functions associated with the untrusted application.
5. The method of claim 1 , wherein executing the untrusted application in the first secured processing environment comprises executing the untrusted application within a virtual machine in the first secured processing environment.
6. The method of claim 1 , wherein communicating information representing the data flow graph outside of the first secured processing environment comprises communicating information representing the data flow graph outside of the first secured processing environment using a remote procedure call.
7. The method of claim 1 , wherein a first deferred parallel data object of the deferred parallel data objects holds a pointer to a first deferred parallel operation that computes the first deferred parallel data object.
8. The method of claim 1 , wherein a first deferred parallel operation of the deferred parallel operations holds references to a first parallel data object that is an argument of at least one deferred parallel operation of the deferred parallel operations.
9. A system comprising:
one or more processing devices; and
one or more storage devices, the storage devices storing instructions that, when executed by the one or more processing devices, cause the one or more processing devices to:
receive an untrusted application that includes a data parallel pipeline, wherein the data parallel pipeline specifies multiple parallel data objects and multiple parallel data operations,
instantiate a first secured processing environment in a native processing environment and on one or more processing modules,
execute the untrusted application to generate a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline,
communicate information representing the dataflow graph outside of the first secured processing environment,
apply, in the native processing environment, one or more graph transformations to the information representing the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations that are associated with the untrusted application, and
execute the deferred, combined parallel operations to produce materialized parallel data objects corresponding to the deferred parallel data objects.
10. The system of claim 9 , wherein to apply one or more graph transformations comprise the instructions causing the one or processing devices to combine chains of parallel operations together into a smaller number of combined operations.
11. The system of claim 9 , wherein the first secured processing environment comprises a first virtual machine.
12. The system of claim 9 , wherein the deferred, combined parallel data operations includes at least one generalized mapreduce operation, the generalized mapreduce operation including multiple, parallel map operations and multiple, parallel reduce operations and being translatable to a single mapreduce operation that includes a single map function to implement the multiple, parallel map operations and a single reduce function to implement the multiple, parallel reduce operations, the single map function and the single reduce function including one or more untrusted functions associated with the untrusted application.
13. The system of claim 9 , wherein to execute the deferred, combined parallel operations comprises executing the untrusted application within a virtual machine in the first secured processing environment.
14. The system of claim 9 , wherein to communicate information representing the data flow graph outside of the first secured processing environment comprises communicating information representing the data flow graph outside of the first secured processing environment using a remote procedure call.
15. The system of claim 9 , wherein a first deferred parallel data object of the deferred parallel data objects holds a pointer to a first deferred parallel operation that computes the first deferred parallel data object.
16. The system of claim 9 , wherein a first deferred parallel operation of the deferred parallel operations hold references to a first parallel data object that is an argument of at least one deferred parallel operation of the deferred parallel operations.