Training latent variable machine learning models using multi-sample objectives
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a machine learning model. One of the methods includes, for each training observation: determining a plurality of latent variable value configurations, each latent variable value configuration being a combination of latent variable values that includes a respective value for each of the latent variables; determining, for each of the plurality of latent variable value configurations, a respective local learning signal that is minimally dependent on each of the other latent variable value configurations in the plurality of latent variable value configurations; determining an unbiased estimate of a gradient of the objective function using the local learning signals; and updating current values of the parameters of the machine learning model using the unbiased estimate of the gradient.
1. A method of training a machine learning model having a plurality of parameters on a plurality of training observations by optimizing an objective function, wherein the machine learning model is configured to determine a model output by determining a respective value for each of a plurality of latent variables and determining the model output from the determined latent variable values, the method comprising, for each of the plurality of training observations:
generating a distribution over possible values for the plurality of latent variables by processing the training observation; and
sampling from the distribution a plurality of times to generate a plurality of latent variable value configurations for the training observation, each latent variable value configuration being a combination of latent variable values that includes a respective value for each of the latent variables;
determining, for each of the plurality of latent variable value configurations, a respective local learning signal that measures how well the corresponding latent variable value configuration explains the training observation with minimal impact from the other latent variable value configurations, wherein the local learning signal {circumflex over (L)}(h j |h −j ) for a latent variable value configuration h j satisfies:
L
^
(
h
j
|
h
-
j
)
=
L
^
(
h
1
:
K
)
-
log
1
K
(
∑
i
≠
j
f
(
x
,
h
i
)
+
f
^
(
x
,
h
j
)
)
,
where h j is the j-th latent variable value configuration, K is the total number of configurations, i ranges to cover all of the configurations except for the j-th configuration, x is the training observation, ƒ is a function of the training observation x and a latent variable configuration, {circumflex over (ƒ)} is the mean of the values of ƒ for all of the configurations except the j-th configuration, and {circumflex over (L)}(h 1:K ) is a global learning signal;
determining an unbiased estimate of a gradient of the objective function using the local learning signals; and
updating current values of the parameters of the machine learning model using the unbiased estimate of the gradient.
2. The method of claim 1 , wherein at least one of the latent variables is a discrete latent variable.
3. The method of claim 1 , wherein the distribution is a multivariate distribution and wherein the machine learning model is further configured to:
determine the respective value for each of the latent variables by sampling from the multivariate distribution over possible values for the latent variables.
4. The method of claim 3 , wherein determining the plurality of latent variable value configurations comprises, for each latent variable value configuration:
selecting the latent variable values in the latent variable value configuration by sampling from the multivariate distribution.
5. The method of claim 3 , wherein the machine learning model is further configured to:
process the training observation, a context for the training observation, or both to determine parameters for the multivariate distribution.
6. The method of claim 3 , wherein the unbiased gradient estimate satisfies:
Σ j {circumflex over (L)}(h j |h −j )∇ log Q(h j |x)+Σ j {tilde over (w)} j ∇ log ƒ(x,h j )
where h j is the j-th latent variable value configuration, {circumflex over (L)}(h j |h −j ) is a local learning signal for the latent variable value configuration h j , ƒ is a function of the training observation x and a latent variable configuration, {circumflex over (ƒ)} is the mean of the values of ƒ for all of the configurations except the j-th configuration, Q(h j |x) is the multivariate distribution from which the configurations are sampled and {tilde over (w)} j is equal to the value off for the j-th configuration divided by the sum of the values of ƒ for all of the configurations.
7. The method of claim 1 , wherein {circumflex over (L)}(h 1:K ) satisfies:
L
^
(
h
1
:
K
)
=
log
1
K
(
∑
i
f
(
x
,
h
i
)
)
.
8. The method of claim 1 , wherein ƒ is a function that measures how well an input latent variable value configuration explains an input observation.
9. The method of claim 1 , wherein the machine learning model comprises a first neural network configured to generate the model output from the values of the latent variables.
10. The method of claim 9 , wherein the machine learning model comprises a second neural network configured to process the training observation to generate an output that defines the parameters of the distribution over possible values for the latent variables.
11. The method of claim 1 , wherein the model output is an image.
12. The method of claim 11 , wherein each of the latent variables represents a feature of the image.
13. The method of claim 1 , wherein the function ƒ(x,h i ) satisfies:
f
(
x
,
h
i
)
=
p
(
x
,
h
i
)
q
(
h
i
|
x
)
,
where p(x,h i ) measures how well the latent variable configuration h i explains the training observation x under the machine learning model, and q(h i |x) is the probability of latent variable configuration h i according to a multi-variate distribution parameterized by distribution parameters generated by the machine learning model by processing the training observation x.
14. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a machine learning model having a plurality of parameters on a plurality of training observations by optimizing an objective function, wherein the machine learning model is configured to determine a model output by determining a respective value for each of a plurality of latent variables and determining the model output from the determined latent variable values, the method comprising, for each training observation:
generating a distribution over possible values for the plurality of latent variables by processing the training observation; and
sampling from the distribution a plurality of times to generate a plurality of latent variable value configurations for the training observation, each latent variable value configuration being a combination of latent variable values that includes a respective value for each of the latent variables;
determining, for each of the plurality of latent variable value configurations, a respective local learning signal that measures how well the corresponding latent variable value configuration explains the training observation with minimal impact from the other latent variable value configurations, wherein the local learning signal {circumflex over (L)}(h j |h −j ) for a latent variable value configuration h j satisfies:
L
^
(
h
j
|
h
-
j
)
=
L
^
(
h
1
:
K
)
-
log
1
K
(
∑
i
≠
j
f
(
x
,
h
i
)
+
f
^
(
x
,
h
j
)
)
,
where h j is the j-th latent variable value configuration, K is the total number of configurations, i ranges to cover all of the configurations except for the j-th configuration, x is the training observation, ƒ is a function of the training observation x and a latent variable configuration, {circumflex over (ƒ)} is the mean of the values of ƒ for all of the configurations except the j-th configuration, and {circumflex over (L)}(h 1:K ) is a global learning signal;
determining an unbiased estimate of a gradient of the objective function using the local learning signals; and
updating current values of the parameters of the machine learning model using the unbiased estimate of the gradient.
15. The system of claim 14 , wherein the distribution is a multivariate distribution and wherein the machine learning model is further configured to:
determine the respective value for each of the latent variables by sampling from the multivariate distribution over possible values for the latent variables.
16. The system of claim 15 , wherein determining the plurality of latent variable value configurations comprises, for each latent variable value configuration:
selecting the latent variable values in the latent variable value configuration by sampling from the multivariate distribution.
17. The system of claim 15 , wherein the machine learning model is further configured to:
process the training observation, a context for the training observation, or both to determine parameters for the multivariate distribution.
18. The system of claim 14 , wherein {circumflex over (L)}(h 1:K ) satisfies:
L
^
(
h
1
:
K
)
=
log
1
K
(
∑
i
f
(
x
,
h
i
)
)
.
19. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a machine learning model having a plurality of parameters on a plurality of training observations by optimizing an objective function, wherein the machine learning model is configured to determine a model output by determining a respective value for each of a plurality of latent variables and determining the model output from the determined latent variable values, the method comprising, for each training observation:
generating a distribution over possible values for the plurality of latent variables by processing the training observation; and
sampling from the distribution a plurality of times to generate a plurality of latent variable value configurations for the training observation, each latent variable value configuration being a combination of latent variable values that includes a respective value for each of the latent variables;
determining, for each of the plurality of latent variable value configurations, a respective local learning signal that measures how well the corresponding latent variable value configuration explains the training observation with minimal impact from the other latent variable value configurations, wherein the local learning signal {circumflex over (L)}(h j |h −j ) for a latent variable value configuration h j satisfies:
L
^
(
h
j
|
h
-
j
)
=
L
^
(
h
1
:
K
)
-
log
1
K
(
∑
i
≠
j
f
(
x
,
h
i
)
+
f
^
(
x
,
h
j
)
)
,
where h j is the j-th latent variable value configuration, K is the total number of configurations, i ranges to cover all of the configurations except for the j-th configuration, x is the training observation, ƒ is a function of the training observation x and a latent variable configuration, {circumflex over (ƒ)} is the mean of the values of ƒ for all of the configurations except the j-th configuration, and {circumflex over (L)}(h 1:K ) is a global learning signal;
determining an unbiased estimate of a gradient of the objective function using the local learning signals; and
updating current values of the parameters of the machine learning model using the unbiased estimate of the gradient.
20. The system of claim 14 , wherein the function ƒ(x,h i ) satisfies:
f
(
x
,
h
i
)
=
p
(
x
,
h
i
)
q
(
h
i
|
x
)
,
where p(x,h i ) measures how well the latent variable configuration h i explains the training observation x under the machine learning model, and q(h i |x) is the probability of latent variable configuration h i according to a multi-variate distribution parameterized by distribution parameters generated by the machine learning model by processing the training observation x.