Clustering key selection based on machine-learned key selection models for data processing service
The disclosed configurations provide a method (and/or a computer-readable medium or system) for determining, from a table schema describing keys of a data table, one or more clustering keys that can be used to cluster data files of a data table. The method includes generating features for the data table, generating tokens from the features, generating a prediction for each token by applying to the token a machine-learned transformer model trained to predict a likelihood that the key associated with the token is a clustering key for the data table, determining clustering keys based on the predictions, and clustering data records of the data table into data files based on key-values for the clustering keys.
1. A computer-implemented method, comprising:
obtaining a table schema describing at least one or more keys of a data table, the data table stored at a storage system associated with a data processing service;
generating a set of features for the data table, wherein for each key of the data table, the set of features includes at least a name label for the key and a data type associated with the key;
generating a set of tokens from the set of features, wherein the set of tokens are numerical representations of features in the set of features;
generating a set of predictions by applying a machine-learned transformer model to the set of tokens, wherein the set of predictions include a prediction for each token indicating a likelihood that a key associated with the token is a clustering key for the data table;
determining one or more clustering keys based on the set of predictions, wherein the one or more clustering keys are a subset of the one or more keys; and
clustering data records of the data table into one or more data files based on key-values for the one or more clustering keys.
2. The computer-implemented method of claim 1 , wherein clustering data records of the data table comprises generating a data classifier tree by recursively dividing the data records of the data table according to key-values for each of the one or more clustering keys.
3. The computer-implemented method of claim 1 , further comprising training the machine-learned transformer model by:
retrieving a training data table clustered according to one or more clustering keys;
obtaining a table schema from the training data table, the table schema describing at least one or more keys of the training data table;
identifying a set of features for the training data table, wherein for each key of the training data table, the set of features includes at least a name label for the key and a data type associated with the key;
generating a set of tokens from the set of features, wherein the set of tokens are numerical representations of features in the set of features;
obtaining a set of labels for the set of tokens, wherein the set of labels includes a label for each token indicating whether a key associated with the token is one of the one or more clustering keys; and
training the machine-learned transformer model by repeatedly iterating between a forward pass step and a backpropagation step to reduce a loss function.
4. The computer-implemented method of claim 1 , wherein generating the set of predictions by applying the machine-learned transformer model to the set of tokens comprises:
generating one or more output embeddings for the set of tokens; and
inputting the one or more output embeddings to a machine-learned classifier to generate the set of predictions.
5. The computer-implemented method of claim 1 , wherein the set of tokens are first tokens for the name label of each feature of the set of features of the data table.
6. The computer-implemented method of claim 1 , wherein the machine-learned transformer model is a bi-directional encoding representation from transformers (BERT) model.
7. The computer-implemented method of claim 1 , wherein clustering data records of the data table into one or more data files based on key-values for the one or more clustering keys comprises clustering data records using a KD-classifier tree.
8. A non-transitory computer readable medium comprising stored instructions, the stored instructions when executed by at least one processor of one or more computing devices, cause the one or more computing devices to:
obtain a table schema describing at least one or more keys of a data table, the data table stored at a storage system associated with a data processing service;
generate a set of features for the data table, wherein for each key of the data table, the set of features includes at least a name label for the key and a data type associated with the key;
generate a set of tokens from the set of features, wherein the set of tokens are numerical representations of features in the set of features;
generate a set of predictions by applying a machine-learned transformer model to the set of tokens, wherein the set of predictions include a prediction for each token indicating a likelihood that a key associated with the token is a clustering key for the data table;
determine one or more clustering keys based on the set of predictions, wherein the one or more clustering keys are a subset of the one or more keys; and
cluster data records of the data table into one or more data files based on key-values for the one or more clustering keys.
9. The non-transitory computer readable medium of claim 8 , wherein the instructions for clustering data records of the data table comprise instructions causing the computing devices to generate a data classifier tree by recursively dividing the data records of the data table according to key-values for each of the one or more clustering keys.
10. The non-transitory computer readable medium of claim 8 , the instructions further causing the computing devices to:
retrieve a training data table clustered according to one or more clustering keys;
obtain a table schema from the training data table, the table schema describing at least one or more keys of the training data table;
identify a set of features for the training data table, wherein for each key of the training data table, the set of features includes at least a name label for the key and a data type associated with the key;
generate a set of tokens from the set of features, wherein the set of tokens are numerical representations of the features;
obtain a set of labels for the set of tokens, wherein the set of labels include a label for each token indicating whether a key associated with the token is one of the one or more clustering keys; and
train the machine-learned transformer model by repeatedly iterating between a forward pass step and a backpropagation step to reduce a loss function.
11. The non-transitory computer readable medium of claim 8 , wherein the instructions for generating the set of predictions by applying the machine-learned transformer model to the set of tokens comprise instructions causing the computing devices to:
generate one or more output embeddings for the set of tokens; and
input the one or more output embeddings to a machine-learned classifier to generate the set of predictions.
12. The non-transitory computer readable medium of claim 8 , wherein the set of tokens are first tokens for the name label of each feature of the set of features of the data table.
13. The non-transitory computer readable medium of claim 8 , wherein the machine-learned transformer model is a bi-directional encoding representation from transformers (BERT) model.
14. The non-transitory computer readable medium of claim 8 , wherein clustering data records of the data table into one or more data files based on key-values for the one or more clustering keys comprises clustering data records using a KD-classifier tree.
15. A computer system, comprising:
a computer processor; and
a non-transitory computer-readable storage medium comprising instructions that when executed by the computer processor, cause the computer system to:
obtain a table schema describing at least one or more keys of a data table, the data table stored at a storage system associated with a data processing service;
generate a set of features for the data table, wherein for each key of the data table, the set of features includes at least a name label for the key and a data type associated with the key;
generate a set of tokens from the set of features, wherein the set of tokens are numerical representations of features in the set of features;
generate a set of predictions by applying a machine-learned transformer model to the set of tokens, wherein the set of predictions include a prediction for each token indicating a likelihood that a key associated with the token is a clustering key for the data table;
determine one or more clustering keys based on the set of predictions, wherein the one or more clustering keys are a subset of the one or more keys; and
cluster data records of the data table into one or more data files based on key-values for the one or more clustering keys.
16. The computer system of claim 15 , wherein the instructions for clustering data records of the data table comprise instructions causing the computing system to generate a data classifier tree by recursively dividing the set of records of the data table according to key-values for each of the one or more clustering keys.
17. The computer system of claim 15 , the instructions further causing the computing system to:
retrieve a training data table clustered according to one or more clustering keys;
obtain a table schema from the training data table, the table schema describing at least one or more keys of the training data table;
identify a set of features for the training data table, wherein for each key of the training data table, the set of features includes at least a name label for the key and a data type associated with the key;
generate a set of tokens from the set of features, wherein the set of tokens are numerical representations of the features;
obtain a set of labels for the set of tokens, wherein the set of labels include a label for each token indicating whether a key associated with the token is one of the one or more clustering keys; and
train the machine-learned transformer model by repeatedly iterating between a forward pass step and a backpropagation step to reduce a loss function.
18. The computer system of claim 15 , wherein the instructions for generating the set of predictions by applying the machine-learned transformer model to the set of tokens comprise instructions causing the computing system to:
generate one or more output embeddings for the set of tokens; and
input the one or more output embeddings to a machine-learned classifier to generate the set of predictions.
19. The computer system of claim 15 , wherein the set of tokens are first tokens for the name label of each feature of the set of features of the data table.
20. The computer system of claim 15 , wherein the machine-learned transformer model is a bi-directional encoding representation from transformers (BERT) model.