Method, system, and computer program product for embedding compression and regularization
A method, system, and computer program product is provided for embedding compression and reconstruction. The method includes receiving embedding vector data comprising a plurality of embedding vectors. A beta-variational autoencoder is trained based on the embedding vector data and a loss equation. The method includes determining a respective entropy of a respective mean and a respective variance of each respective dimension of a plurality of dimensions. A first subset of the plurality of dimensions is determined based on the respective entropy of the respective mean and the respective variance for each respective dimension of the plurality of dimensions. A second subset of the plurality of dimensions is discarded based on the respective entropy of the respective mean and the respective variance for each respective dimension of the plurality of dimensions. The method includes generating a compressed representation of the embedding vector data based on the first subset of dimensions.
1 . A computer-implemented method for generating a compressed representation of embedding vector data, comprising:
receiving, with at least one processor, the embedding vector data comprising a plurality of embedding vectors;
training, with at least one processor, a beta-variational autoencoder based on the embedding vector data and a loss equation, the beta-variational autoencoder comprising an encoder network, a latent layer, and a decoder network, the loss equation comprising a first term associated with reconstruction of an input by the beta-variational autoencoder, a second term associated with regularization of a latent space of the latent layer, and a hyperparameter weight associated with a ratio of the first term and the second term, wherein the latent space has a plurality of dimensions;
determining, with at least one processor, a respective mean of each respective dimension of the plurality of dimensions, wherein determining the respective mean of each respective dimension of the plurality of dimensions comprises determining a respective entropy of the respective mean of each respective dimension of the plurality of dimensions;
determining whether each respective dimension of the plurality of dimensions comprises a useful dimension or a deprecated dimension based on the respective mean for the respective dimension, wherein the respective entropy of the respective mean for each useful dimension is higher than a threshold, and wherein the respective entropy of the respective mean for each deprecated dimension is lower than the threshold;
determining, with at least one processor, a first subset of the plurality of dimensions based on the respective mean for each respective dimension of the plurality of dimensions, wherein the first subset of the plurality of dimensions comprises useful dimensions;
discarding, with at least one processor, a second subset of the plurality of dimensions based on the respective mean of each respective dimension of the plurality of dimensions, wherein the second subset of the plurality of dimensions is different than the first subset of the plurality of dimensions, wherein the second subset of the plurality of dimensions comprises deprecated dimensions, and wherein discarding the second subset of the plurality of dimensions reduces a dimensionality of the latent space;
generating, with at least one processor, the compressed representation of the embedding vector data based on the first subset of dimensions; and
inputting, with at least one processor, the compressed representation of the embedding vector data into at least one machine learning model.
2 . The computer-implemented method of claim 1 , wherein determining the respective mean of each respective dimension of the plurality of dimensions comprises:
determining a respective entropy of the respective mean of each respective dimension of the plurality of dimensions; and
wherein training the beta-variational autoencoder comprises:
iteratively adjusting the hyperparameter weight and repeating the training, the determining of the respective entropy of the respective mean, the determining of the first subset, the discarding of the second subset, and the generating of the compressed representation.
3 . The computer-implemented method of claim 1 , wherein the first subset of the plurality of dimensions comprises each useful dimension, and wherein the second subset of the plurality of dimensions comprises each deprecated dimension.
4 . The computer-implemented method of claim 1 , wherein determining the respective mean of each respective dimension of the plurality of dimensions comprises:
determining, with at least one processor, a respective entropy of the respective mean of each respective dimension of the plurality of dimensions; and
wherein determining the first subset of the plurality of dimensions comprises:
determining, with at least one processor, the first subset of the plurality of dimensions based on the respective entropy of the respective mean for each respective dimension of the plurality of dimensions.
5 . The computer-implemented method of claim 4 , wherein discarding the second subset of the plurality of dimensions comprises:
discarding the second subset of the plurality of dimensions based on the respective entropy of the respective mean of each respective dimension of the plurality of dimensions.
6 . The computer-implemented method of claim 1 , wherein the loss equation is:
ℒ
=
∑
i
=
1
n
(
x
i
-
x
^
i
)
2
+
β
∑
i
=
1
m
D
KL
(
𝒩
(
μ
i
,
σ
i
2
)
𝒩
(
0
,
1
)
)
;
wherein the first term associated with reconstruction of an input by the beta-variational autoencoder is:
∑
i
=
1
n
(
x
i
-
x
^
i
)
2
;
wherein the second term associated with regularization of the latent space of the latent layer is:
∑
i
=
1
m
D
KL
(
𝒩
(
μ
i
,
σ
i
2
)
𝒩
(
0
,
1
)
)
;
wherein x i is the input and {circumflex over (x)} i is a reconstruction of the input provided by the beta-variational autoencoder;
wherein D KL is a function for a Kullback-Leibler divergence;
wherein is a function for a normal distribution; and
wherein the hyperparameter weight associated with the ratio of the first term and the second term is β.
7 . A system for generating a compressed representation of embedding vector data comprising at least one processor programmed or configured to:
receive the embedding vector data comprising a plurality of embedding vectors;
train a beta-variational autoencoder based on the embedding vector data and a loss equation, the beta-variational autoencoder comprising an encoder network, a latent layer, and a decoder network, the loss equation comprising a first term associated with reconstruction of an input by the beta-variational autoencoder, a second term associated with regularization of a latent space of the latent layer, and a hyperparameter weight associated with a ratio of the first term and the second term, wherein the latent space has a plurality of dimensions;
determine a respective mean of each respective dimension of the plurality of dimensions, wherein, when determining the respective mean of each respective dimension of the plurality of dimensions, the at least one processor is programmed or configured to:
determine a respective entropy of the respective mean of each respective dimension of the plurality of dimensions;
determine whether each respective dimension of the plurality of dimensions comprises a useful dimension or a deprecated dimension based on the respective mean for the respective dimension, wherein the respective entropy of the respective mean for each useful dimension is higher than a threshold, and wherein the respective entropy of the respective mean for each deprecated dimension is lower than the threshold;
determine a first subset of the plurality of dimensions based on the respective mean for each respective dimension of the plurality of dimensions, wherein the first subset of the plurality of dimensions comprises useful dimensions;
discard a second subset of the plurality of dimensions based on the respective mean of each respective dimension of the plurality of dimensions;
generate the compressed representation of the embedding vector data based on the first subset of dimensions; and
input the compressed representation of the embedding vector data into at least one machine learning model.
8 . The system of claim 7 , wherein, when determining the respective mean of each respective dimension of the plurality of dimensions, the at least one processor is programmed or configured to:
determine a respective entropy of the respective mean of each respective dimension of the plurality of dimensions; and
wherein, when training the beta-variational autoencoder, the at least one processor is programmed or configured to:
iteratively adjust the hyperparameter weight and repeat the training, the determining of the respective entropy of the respective mean, the determining of the first subset, the discarding of the second subset, and the generating of the compressed representation.
9 . The system of claim 7 , wherein the first subset of the plurality of dimensions comprises each useful dimension, and wherein the second subset of the plurality of dimensions comprises each deprecated dimension.
10 . The system of claim 7 , wherein, when determining the respective mean of each respective dimension of the plurality of dimensions, the at least one processor is programmed or configured to:
determine a respective entropy of the respective mean of each respective dimension of the plurality of dimensions; and
wherein, when determining the first subset of the plurality of dimensions, the at least one processor is programmed or configured to:
determine the first subset of the plurality of dimensions based on the respective entropy of the respective mean for each respective dimension of the plurality of dimensions.
11 . The system of claim 10 , wherein, when discarding the second subset of the plurality of dimensions, the at least one processor is programmed or configured to:
discard the second subset of the plurality of dimensions based on the respective entropy of the respective mean of each respective dimension of the plurality of dimensions.
12 . The system of claim 7 , wherein the loss equation is:
ℒ
=
∑
i
=
1
n
(
x
i
-
x
^
i
)
2
+
β
∑
i
=
1
m
D
KL
(
𝒩
(
μ
i
,
σ
i
2
)
𝒩
(
0
,
1
)
)
wherein the first term associated with reconstruction of an input by the beta-variational autoencoder is:
∑
i
=
1
n
(
x
i
-
x
^
i
)
2
;
wherein the second term associated with regularization of the latent space of the latent layer is:
∑
i
=
1
m
D
KL
(
𝒩
(
μ
i
,
σ
i
2
)
𝒩
(
0
,
1
)
)
;
wherein x i is the input and {circumflex over (x)} i is a reconstruction of the input provided by the beta-variational autoencoder;
wherein D KL is a function for a Kullback-Leibler divergence;
wherein is a function for a normal distribution; and
wherein the hyperparameter weight associated with the ratio of the first term and the second term is β.
13 . A computer program product for generating a compressed representation of embedding vector data, the computer program product comprising at least one non-transitory computer readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to:
receive the embedding vector data comprising a plurality of embedding vectors;
train a beta-variational autoencoder based on the embedding vector data and a loss equation, the beta-variational autoencoder comprising an encoder network, a latent layer, and a decoder network, the loss equation comprising a first term associated with reconstruction of an input by the beta-variational autoencoder, a second term associated with regularization of a latent space of the latent layer, and a hyperparameter weight associated with a ratio of the first term and the second term, wherein the latent space has a plurality of dimensions;
determine a respective mean of each respective dimension of the plurality of dimensions, wherein, when determining the respective mean of each respective dimension of the plurality of dimensions, the one or more instructions cause the at least one processor to:
determine a respective entropy of the respective mean of each respective dimension of the plurality of dimensions;
determine whether each respective dimension of the plurality of dimensions comprises a useful dimension or a deprecated dimension based on the respective mean for the respective dimension, wherein the respective entropy of the respective mean for each useful dimension is higher than a threshold, and wherein the respective entropy of the respective mean for each deprecated dimension is lower than the threshold;
determine a first subset of the plurality of dimensions based on the respective mean for each respective dimension of the plurality of dimensions, wherein the first subset of the plurality of dimensions comprises useful dimensions;
discard a second subset of the plurality of dimensions based on the respective mean of each respective dimension of the plurality of dimensions, wherein the second subset of the plurality of dimensions is different than the first subset of the plurality of dimensions, wherein the second subset of the plurality of dimensions comprises deprecated dimensions, and wherein discarding the second subset of the plurality of dimensions reduces a dimensionality of the latent space;
generate the compressed representation of the embedding vector data based on the first subset of dimensions; and
input the compressed representation of the embedding vector data into at least one machine learning model.
14 . The computer program product of claim 13 , wherein, when determining the respective mean of each respective dimension of the plurality of dimensions, the one or more instructions cause the at least one processor to:
determine a respective entropy of the respective mean of each respective dimension of the plurality of dimensions; and
wherein, when training the beta-variational autoencoder, the one or more instructions further cause the at least one processor to:
iteratively adjust the hyperparameter weight and repeat the training, the determining of the respective entropy of the respective mean, the determining of the first subset, the discarding of the second subset, and the generating of the compressed representation.
15 . The computer program product of claim 13 , wherein the first subset of the plurality of dimensions comprises each useful dimension, and wherein the second subset of the plurality of dimensions comprises each deprecated dimension.
16 . The computer program product of claim 13 , wherein, when determining the respective mean of each respective dimension of the plurality of dimensions, the one or more instructions cause the at least one processor to:
determine a respective entropy of the respective mean of each respective dimension of the plurality of dimensions; and
wherein, when determining the first subset of the plurality of dimensions, the one or more instructions cause the at least one processor to:
determine the first subset of the plurality of dimensions based on the respective entropy of the respective mean for each respective dimension of the plurality of dimensions.
17 . The computer program product of claim 16 , wherein, when discarding the second subset of the plurality of dimensions, the one or more instructions cause the at least one processor to:
discard the second subset of the plurality of dimensions based on the respective entropy of the respective mean of each respective dimension of the plurality of dimensions.
18 . The computer program product of claim 13 , wherein the loss equation is:
ℒ
=
∑
i
=
1
n
(
x
i
-
x
^
i
)
2
+
β
∑
i
=
1
m
D
KL
(
𝒩
(
μ
i
,
σ
i
2
)
𝒩
(
0
,
1
)
)
wherein the first term associated with reconstruction of an input by the beta-variational autoencoder is:
∑
i
=
1
n
(
x
i
-
x
^
i
)
2
;
wherein the second term associated with regularization of the latent space of the latent layer is:
∑
i
=
1
m
D
KL
(
𝒩
(
μ
i
,
σ
i
2
)
𝒩
(
0
,
1
)
)
;
wherein x i is the input and {circumflex over (x)} i is a reconstruction of the input provided by the beta-variational autoencoder;
wherein D KL is a function for a Kullback-Leibler divergence;
wherein is a function for a normal distribution; and
wherein the hyperparameter weight associated with the ratio of the first term and the second term is β.