Distributed in-database vectorized operations using user defined table functions

1. A system comprising:

at least one hardware processor; and

a memory storing instructions that cause the at least one hardware processor to perform operations comprising:

receiving a set of values for a data set corresponding to a request for determining a Pearson correlation;

determining a number of rows from the data set;

determining a number of execution nodes to execute the request for determining the Pearson correlation;

determining a set of shards of rows from the data set based on the number of rows and the number of execution nodes;

for each shard from the set of shards, sending a particular user defined table function (UDTF), including a particular shard of rows, to a different execution node to perform a set of operations for determining the Pearson correlation;

providing a set of output values of each particular UDTF corresponding to each shard from the set of shards in a second UDTF;

sending the second UDTF to a particular execution node to perform an aggregate operation using the set of output values of each particular UDTF; and

receiving a value of the Pearson correlation from the particular execution node based on the aggregate operation.

2. The system of claim 1 , wherein determining the number of rows further comprises:

executing a query on the data set to determine a count of the number of rows.

3. The system of claim 2 , wherein the query comprises a count operation.

4. The system of claim 1 , wherein determining the number of execution nodes further comprises:

determining the number of execution nodes based on a configuration parameter that indicates a maximum number of execution nodes to use for determining the Pearson correlation.

5. The system of claim 1 , wherein determining the set of shards further comprises:

determining a shard size based on dividing the number of rows by the number of execution nodes, the shard size indicating a particular number of rows for including in the particular shard of rows.

6. The system of claim 1 , wherein sending the particular UDTF further comprises:

for each shard from the set of shards, sending a shard identifier to the different execution node.

7. The system of claim 1 , wherein the operations further comprise:

determining a dot product and a square root of a sum of squared values for the particular shard of rows based on the set of output values from the particular UDTF.

8. The system of claim 1 , wherein the aggregate operation includes:

determining a total sum of values and a total sum of squared values using the set of output values from each particular UDTF.

9. The system of claim 1 , wherein the operations further comprise:

storing the value of the Pearson correlation in a results table;

receiving a second request for reading the value of the Pearson correlation in the results table;

performing a read operation of the value of the Pearson correlation in the results table; and

determining whether to remove a set of features from the set of values for the data set based on the value of the Pearson correlation.

10. The system of claim 1 , wherein each execution node comprises a different server instance or is included in a different virtual warehouse.

11. A method comprising:

receiving a set of values for a data set corresponding to a request for determining a Pearson correlation;

determining a number of rows from the data set;

determining a number of execution nodes to execute the request for determining the Pearson correlation;

determining a set of shards of rows from the data set based on the number of rows and the number of execution nodes;

for each shard from the set of shards, sending a particular user defined table function (UDTF), including a particular shard of rows, to a different execution node to perform a set of operations for determining the Pearson correlation;

providing a set of output values of each particular UDTF corresponding to each shard from the set of shards in a second UDTF;

sending the second UDTF to a particular execution node to perform an aggregate operation using the set of output values of each particular UDTF; and

receiving a value of the Pearson correlation from the particular execution node based on the aggregate operation.

12. The method of claim 11 , wherein determining the number of rows further comprises:

executing a query on the data set to determine a count of the number of rows.

13. The method of claim 12 , wherein the query comprises a count operation.

14. The method of claim 11 , wherein determining the number of execution nodes further comprises:

determining the number of execution nodes based on a configuration parameter that indicates a maximum number of execution nodes to use for determining the Pearson correlation.

15. The method of claim 11 , wherein determining the set of shards further comprises:

determining a shard size based on dividing the number of rows by the number of execution nodes, the shard size indicating a particular number of rows for including in the particular shard of rows.

16. The method of claim 11 , wherein sending the particular UDTF further comprises:

for each shard from the set of shards, sending a shard identifier to the different execution node.

17. The method of claim 11 , further comprising:

determining a dot product and a square root of a sum of squared values for the particular shard of rows based on the set of output values from the particular UDTF.

18. The method of claim 11 , wherein the aggregate operation includes:

determining a total sum of values and a total sum of squared values using the set of output values from each particular UDTF.

19. The method of claim 11 , further comprising:

storing the value of the Pearson correlation in a results table;

receiving a second request for reading the value of the Pearson correlation in the results table;

performing a read operation of the value of the Pearson correlation in the results table; and

determining whether to remove a set of features from the set of values for the data set based on the value of the Pearson correlation.

20. The method of claim 11 , wherein each execution node comprises a different server instance or is included in a different virtual warehouse.

21. A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

receiving a set of values for a data set corresponding to a request for determining a Pearson correlation;

determining a number of rows from the data set;

determining a number of execution nodes to execute the request for determining the Pearson correlation;

determining a set of shards of rows from the data set based on the number of rows and the number of execution nodes;

for each shard from the set of shards, sending a particular user defined table function (UDTF), including a particular shard of rows, to a different execution node to perform a set of operations for determining the Pearson correlation;

providing a set of output values of each particular UDTF corresponding to each shard from the set of shards in a second UDTF;

sending the second UDTF to a particular execution node to perform an aggregate operation using the set of output values of each particular UDTF; and

receiving a value of the Pearson correlation from the particular execution node based on the aggregate operation.

22. The non-transitory computer-storage medium of claim 21 , wherein determining the number of rows further comprises:

executing a query on the data set to determine a count of the number of rows.

23. The non-transitory computer-storage medium of claim 22 , wherein the query comprises a count operation.

24. The non-transitory computer-storage medium of claim 21 , wherein determining the number of execution nodes further comprises:

determining the number of execution nodes based on a configuration parameter that indicates a maximum number of execution nodes to use for determining the Pearson correlation.

25. The non-transitory computer-storage medium of claim 21 , wherein determining the set of shards further comprises:

determining a shard size based on dividing the number of rows by the number of execution nodes, the shard size indicating a particular number of rows for including in the particular shard of rows.

26. The non-transitory computer-storage medium of claim 21 , wherein sending the particular UDTF further comprises:

for each shard from the set of shards, sending a shard identifier to the different execution node.

27. The non-transitory computer-storage medium of claim 21 , wherein the operations further comprise:

determining a dot product and a square root of a sum of squared values for the particular shard of rows based on the set of output values from the particular UDTF.

28. The non-transitory computer-storage medium of claim 21 , wherein the aggregate operation includes:

determining a total sum of values and a total sum of squared values using the set of output values from each particular UDTF.

29. The non-transitory computer-storage medium of claim 21 , wherein the operations further comprise:

storing the value of the Pearson correlation in a results table;

receiving a second request for reading the value of the Pearson correlation in the results table;

performing a read operation of the value of the Pearson correlation in the results table; and

determining whether to remove a set of features from the set of values for the data set based on the value of the Pearson correlation.

30. The non-transitory computer-storage medium of claim 21 , wherein each execution node comprises a different server instance or is included in a different virtual warehouse.