Automated selection of principal component analysis variants for large-scale datasets
A method for principal component analysis includes receiving a principal component analysis (PCA) request from a user requesting data processing hardware to perform PCA on a dataset, the dataset including a plurality of input features. The method further includes training a PCA model on the plurality of input features of the dataset. The method includes determining, using the trained PCA model, one or more principal components of the dataset. The method also includes generating, based on the plurality of input features and the one or more principal components, one or more embedded features of the dataset. The method includes returning the one or more embedded features to the user.
1 . A computer-implemented method comprising:
receiving, by data processing hardware, a principal component analysis request from a user requesting the data processing hardware to perform principal component analysis on a dataset, the dataset comprising a plurality of input features;
determining, by the data processing hardware, whether a number of the plurality of input features satisfies a threshold number of input features, wherein the threshold number of input features corresponds to a number of the input features that fit into a memory of a single server that implements the data processing hardware;
responsive to determining that the number of the plurality of input features satisfies the threshold number of input features, selecting, by the data processing hardware, a basic principal component analysis variant as a selected principal component analysis variant;
responsive to determining that the number of the plurality of input features fails to satisfy the threshold number of input features, selecting, by the data processing hardware, a randomized principal component analysis variant as the selected principal component analysis variant;
training, by the data processing hardware and using the selected principal component analysis variant, a principal component analysis model on the plurality of input features of the dataset;
determining, by the data processing hardware and using the trained principal component analysis model, one or more principal components of the dataset;
generating, by the data processing hardware and based on the plurality of input features and the one or more principal components, one or more embedded features of the dataset; and
returning, by the data processing hardware, the one or more embedded features to the user.
2 . The method of claim 1 , wherein a number of the one or more embedded features of the dataset is less than a number of the plurality of input features of the dataset.
3 . The method of claim 1 , wherein the principal component analysis request comprises a single Structured Query Language query.
4 . The method of claim 1 , wherein determining, using the trained principal component analysis model, the one or more principal components of the dataset comprises using an economy sized quadrature remainder decomposition algorithm.
5 . The method of claim 1 , wherein determining, using the trained principal component analysis model, the one or more principal components of the dataset comprises determining, by the data processing hardware and using a quadratic programming with non-decreasing constraints algorithm, the one or more principal components of the dataset.
6 . The method of claim 1 , wherein the randomized principal component analysis variant is configured to process a matrix having a number of rows in a defined range of 100,000 to 1,000,000.
7 . The method of claim 1 , wherein the randomized principal component analysis variant is a transposed randomized principal component analysis algorithm.
8 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to:
receive a principal component analysis request from a user requesting the data processing hardware to perform principal component analysis on a dataset, the dataset comprising a plurality of input features;
determine whether a number of the plurality of input features satisfies a threshold number of input features, wherein the threshold number of input features corresponds to a number of the input features that fit into a memory of a single server that implements the data processing hardware;
responsive to determining that the number of the plurality of input features satisfies the threshold number of input features, select a basic principal component analysis variant as a selected principal component analysis variant;
responsive to determining that the number of the plurality of input features fails to satisfy the threshold number of input features, select a randomized principal component analysis variant as the selected principal component analysis variant;
train, using the selected principal component analysis variant, a principal component analysis model on the plurality of input features of the dataset;
determine, using the trained principal component analysis model, one or more principal components of the dataset;
generate, based on the plurality of input features and the one or more principal components, one or more embedded features of the dataset; and
return the one or more embedded features to the user.
9 . The system of claim 8 , wherein a number of the one or more embedded features of the dataset is less than a number of the plurality of input features of the dataset.
10 . The system of claim 8 , wherein the principal component analysis request comprises a single Structured Query Language query.
11 . The system of claim 8 , wherein the instructions cause the data processing hardware to determine the one or more principal components of the dataset using an economy sized quadrature remainder decomposition algorithm.
12 . The system of claim 8 , wherein the instructions cause the data processing hardware to determine the one or more principal components of the dataset using a quadratic programming with non-decreasing constraints algorithm.
13 . The system of claim 8 , wherein the randomized principal component analysis variant is configured to process a matrix having a number of rows in a defined range of 100,000 to 1,000,000.
14 . The system of claim 8 , wherein the randomized principal component analysis variant is a transposed randomized principal component analysis algorithm.