Distributed in-database vectorized operations using user defined table functions
The subject technology determines a set of shards of rows from a data set based on a number of rows and a number of execution nodes to execute a request for determining a correlation. For each shard from the set of shards, the subject technology sends a particular user defined table function (UDTF), including a particular shard of rows, to a different execution node to perform a set of operations for determining the correlation. The subject technology provides a set of output values of each particular UDTF corresponding to each shard from the set of shards in a second UDTF. The subject technology sends the second UDTF to a particular execution node to perform an aggregate operation using the set of output values of each particular UDTF. The subject technology receives a value of the correlation from the particular execution node based on the aggregate operation.
1 . A system comprising:
at least one hardware processor; and
a memory storing instructions that cause the at least one hardware processor to perform operations comprising:
determining a set of shards of rows from a data set based on a number of rows and a number of execution nodes to execute a request for determining a correlation;
for each shard from the set of shards, sending a particular user defined table function (UDTF), including a particular shard of rows, to a different execution node to perform a set of operations for determining the correlation;
providing a set of output values of each particular UDTF corresponding to each shard from the set of shards in a second UDTF;
sending the second UDTF to a particular execution node to perform an aggregate operation using the set of output values of each particular UDTF; and
receiving a value of the correlation from the particular execution node based on the aggregate operation.
2 . The system of claim 1 , wherein the operations further comprise:
prior to determining the set of shards of rows, receiving a set of values for a data set corresponding to a request for determining the correlation;
determining a number of rows from the data set; and
determining a number of execution nodes to execute the request for determining the correlation.
3 . The system of claim 1 , wherein determining the number of rows further comprises:
executing a query on the data set to determine a count of the number of rows.
4 . The system of claim 3 , wherein the query comprises a count operation.
5 . The system of claim 2 , wherein determining the number of execution nodes further comprises:
determining the number of execution nodes based on a configuration parameter that indicates a maximum number of execution nodes to use for determining the correlation.
6 . The system of claim 2 , wherein determining the set of shards further comprises:
determining a shard size based on dividing the number of rows by the number of execution nodes, the shard size indicating a particular number of rows for including in the particular shard of rows.
7 . The system of claim 1 , wherein sending the particular UDTF further comprises:
for each shard from the set of shards, sending a shard identifier to the different execution node.
8 . The system of claim 1 , wherein the operations further comprise:
determining a dot product and a square root of a sum of squared values for the particular shard of rows based on the set of output values from the particular UDTF.
9 . The system of claim 1 , wherein the aggregate operation includes:
determining a total sum of values and a total sum of squared values using the set of output values from each particular UDTF.
10 . The system of claim 1 , wherein the operations further comprise:
storing the value of the correlation in a results table;
receiving a second request for reading the value of the correlation in the results table;
performing a read operation of the value of the correlation in the results table; and
determining whether to remove a set of features from the set of values for the data set based on the value of the correlation.
11 . A method comprising:
determining a set of shards of rows from a data set based on a number of rows and a number of execution nodes to execute a request for determining a correlation;
for each shard from the set of shards, sending a particular user defined table function (UDTF), including a particular shard of rows, to a different execution node to perform a set of operations for determining the correlation;
providing a set of output values of each particular UDTF corresponding to each shard from the set of shards in a second UDTF;
sending the second UDTF to a particular execution node to perform an aggregate operation using the set of output values of each particular UDTF; and
receiving a value of the correlation from the particular execution node based on the aggregate operation.
12 . The method of claim 11 , further comprising:
prior to determining the set of shards of rows, receiving a set of values for a data set corresponding to a request for determining the correlation;
determining a number of rows from the data set; and
determining a number of execution nodes to execute the request for determining the correlation.
13 . The method of claim 11 , wherein determining the number of rows further comprises:
executing a query on the data set to determine a count of the number of rows.
14 . The method of claim 13 , wherein the query comprises a count operation.
15 . The method of claim 12 , wherein determining the number of execution nodes further comprises:
determining the number of execution nodes based on a configuration parameter that indicates a maximum number of execution nodes to use for determining the correlation.
16 . The method of claim 12 , wherein determining the set of shards further comprises:
determining a shard size based on dividing the number of rows by the number of execution nodes, the shard size indicating a particular number of rows for including in the particular shard of rows.
17 . The method of claim 11 , wherein sending the particular UDTF further comprises:
for each shard from the set of shards, sending a shard identifier to the different execution node.
18 . The method of claim 11 , further comprising:
determining a dot product and a square root of a sum of squared values for the particular shard of rows based on the set of output values from the particular UDTF.
19 . The method of claim 11 , wherein the aggregate operation includes:
determining a total sum of values and a total sum of squared values using the set of output values from each particular UDTF.
20 . A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
determining a set of shards of rows from a data set based on a number of rows and a number of execution nodes to execute a request for determining a correlation;
for each shard from the set of shards, sending a particular user defined table function (UDTF), including a particular shard of rows, to a different execution node to perform a set of operations for determining the correlation;
providing a set of output values of each particular UDTF corresponding to each shard from the set of shards in a second UDTF;
sending the second UDTF to a particular execution node to perform an aggregate operation using the set of output values of each particular UDTF; and
receiving a value of the correlation from the particular execution node based on the aggregate operation.