Distributed computing in R

1. A method for distributed computing in R, the method performed by at least one processor and comprising:

identifying a distributed multivariate apply (dmapply) operation and an invocation of a distributed computing backend;

determining a function referenced in the dmapply operation;

translating, using a distributed backend driver associated with the invoked distributed computing backend, the determined function to a function native to an R application programming interface (API) of the invoked distributed computing backend; and

providing the translated function to the invoked distributed computing backend to perform the translated function on a distributed data set referenced in the dmapply operation.

2. The method of claim 1 , wherein providing the translated function to the invoked distributed computing backend includes:

providing the translated function to worker nodes of the invoked distributed computing backend to perform the translated function.

3. The method of claim 2 , wherein each worker node of the worker nodes performs the translated function on a partition of the distributed data set referenced in the dmapply operation.

4. The method of claim 1 , wherein the dmapply operation specifies a subset of partitions included in the distributed data set, and the method comprises:

providing the translated function to the invoked distributed computing backend to perform the translated function on the specified subset of partitions included in the distributed data set.

5. The method of claim 4 , wherein:

the distributed data set is a distributed array;

the dmapply operation specifies a subset of partitions included in the distributed array; and

and the method comprises providing the translated function to each distributed node of the invoked distributed computing backend to perform the translated function on a partition of the specified subset of partitions included in the distributed array.

6. The method of claim 5 , wherein the specified subset of partitions included in the distributed array are at least one of row partitions and block partitions.

7. The method of claim 1 , wherein the distributed data set includes at least one of a distributed array, a distributed data frame, and a distributed list.

8. The method of claim 1 , comprising:

providing the translated function to the invoked distributed computing backend to perform the translated function on a plurality of distributed data sets referenced in the dmapply operation.

9. The method of claim 1 , comprising:

providing instructions to the invoked distributed computing backend to perform the translated function on each element of the distributed data set referenced in the dmapply operation based on a type of distributed data structure associated with the distributed data set referenced in the dmapply operation.

10. The method of claim 9 , wherein:

the distributed data set referenced in the dmapply operation is a distributed list and the method includes providing instructions to the invoked distributed computing backend to perform the translated function on each row of the distributed data set referenced in the dmapply operation;

the distributed data set referenced in the dmapply operation is a distributed array and the method includes providing instructions to the invoked distributed computing backend to perform the translated function on each cell of the distributed data set referenced in the dmapply operation; or

the distributed data set referenced in the dmapply operation is a distributed fram and the method includes providing instructions to the invoked distributed computing backend to perform the translated function on each column of the distributed data set referenced in the dmapply operation.

11. A non-transitory machine-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to:

identify a distributed multivariate apply (dmapply) operation and an invocation of a distributed computing backend;

determine a function referenced in the dmapply operation;

determine a distributed backend driver associated with the invoked distributed computing backend among a plurality of distributed backend drivers;

translate, using the determined distributed backend driver, the determined function to a function native to an R application programming interface (API) of the invoked distributed computing backend;

determine worker nodes of the invoked distributed computing backend having partitions of a distributed data set referenced in the dmapply operation; and

provide the translated function to the determined worker nodes of the invoked computation backend to perform the translated function on the partitions of the distributed data set referenced in the dmapply operation.

12. The non-transitory machine-readable storage medium of claim 11 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to:

translate, using the determined distributed backend driver, the determined function to a function native to an R API of the invoked distributed computing backend based on a direct mapping between the determined function and the native function.

13. The non-transitory machine-readable storage medium of claim 11 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to:

provide the translated function and a global data set referenced in the dmapply operation to the determined worker nodes of the invoked computation backend to perform the translated function on the partitions of the distributed data set referenced in the dmapply operation using the global data set as input to the translated function.

14. The non-transitory machine-readable storage medium of claim 11 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to:

determine that a global data set referenced in the dmapply operation is a distributed data structure; and

in response to determining that the global data set referenced in the dmapply operation is a distributed data structure, determine worker nodes of the invoked distributed computing backend having partitions of the distributed data structure.

15. The non-transitory machine-readable storage medium of claim 14 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to:

provide instructions to the worker nodes determined to have partitions of the distributed data structure to provide the partitions of the distributed data structure to the worker nodes determined to have partitions of a distributed data set referenced in the dmapply operation to perform the translated function on the partitions of the distributed data set referenced in the dmapply operation using the provided partitions of the distributed data structure as input to the translated function.

16. A system, comprising:

a master node to:

identify a first distributed multivariate apply (dmapply) operation and an invocation of a distributed computing backend;

determine a first function referenced in the first dmapply operation;

translate, using a distributed backend driver associated with the invoked distributed computing backend, the first function to a first function native to an R application programming interface (API) of the invoked distributed computing backend; and

provide the translated first function to the invoked distributed computing backend to generate an intermediate distributed data set by performing the translated first function on an initial distributed data set referenced in the first dmapply operation;

identify a second dmapply operation;

determine a second function referenced in the second dmapply operation;

translate, using the distributed backend driver associated with the invoked distributed computing backend, the second function to a second function native to the R API of the invoked distributed computing backend; and

provide the translated second function to the invoked distributed computing backend to generate a final distributed data set by performing the translated second function on the intermediate distributed data set.

17. The system of claim 16 , wherein the master node is to:

determine a first global data set referenced in the first dmapply operation; and

determine a second global data set referenced in the second dmapply operation.

18. The system of claim 17 , wherein the master node is to:

provide the translated first function and the first global data set to the invoked distributed computing backend to generate the intermediate distributed data set by performing the translated first function on the initial distributed data set using the first global data set as input to the first translated function.

19. The system of claim 17 , wherein the first global data set is a vector and the second global data set is a list.

20. The system of claim 17 , wherein the first global data set includes a plurality of partitions of a first distributed data set and the second global data set includes an entire second distributed data set.