User defined data partitioning (UDP)—grouping of data based on computation model
View Patent ↗Methods, systems, and computer program products are provided for generating application-aware data partitioning to support parallel computing. A label for a user defined data partitioning (UDP) key is generated by a labeling process to configure data partitions of original data. The UDP is labeled by the labeling process to include at least one key property excluded from the original data. The data partitions are evenly distributed to co-locate and balance the data partitions and corresponding computations performed by computational servers. A data record of the data partitions is retrieved by performing an all-node parallel search of the computational servers using the UDP key.
1. A computer system for data partitioning, the computer system comprising:
a memory; and
a computer processor to:
generate a user defined data partitioning key to configure data partitions of original data, the user defined data partitioning key generated based upon a computational model applied to the original data, the user defined data partitioning key to include at least one key property of the computational model, the at least one key property is excluded from the original data, and the user defined data partitioning key is generated or learnt from the original data based on an application; and
allocate the data partitions to co-locate the data partitions and corresponding processing of computations associated with the computational model.
2. The system of claim 1 , the computer processor to distribute the data partitions in accordance with the computational model.
3. The system of claim 1 , the computer processor to compute the data partitions in accordance with a data dependency graph based parallel processing.
4. The system of claim 1 , the computer processor to retrieve a data record of the data partitions by performing an all-node parallel search of computational servers performing the computations.
5. The system of claim 1 , the computer processor to retrieve a data record of the data partitions by generating partition indices to perform a query as a two-step search.
6. The system of claim 1 , the computer processor to communicate information about the data partitions to each one of a cluster of computational servers performing the computations.
7. The system of claim 1 , the computer processor to balance a load level of computational servers performing the computations by evenly distributing the data partitions.
8. The system of claim 1 , the computer processor to apply a computation function to data objects included in the data partitions.
9. The system of claim 1 , the computer processor to execute an algorithm to generate a label used as the user defined partition key, the algorithm being configured in accordance with the computation model.
10. The system of claim 1 , the computer processor to mark rows of a table T for representing membership in a group, the group being accessed by the user defined partition key, the table T including at least a portion of the original data.
11. The system of claim 1 , the computer processor to map from a feature space X to a label space Y, the feature space X comprising vectors x 1 to x n , n being an integer, the label space Y comprising vectors y 1 to y m , m being an integer, y i being a vector in the label space X.
12. The system of claim 11 , the computer processor to include a probability measure in the map.