Preventing overfitting of hyperparameters during training of network
Some embodiments provide a method for training a machine-trained (MT) network. The method uses a first set of inputs to train parameters of the MT network according to a set of hyperparameters that define aspects of the training. The method uses a second set of inputs to validate the MT network as trained by the first set of inputs. Based on the validation, the method modifies the hyperparameters for subsequent training of the MT network, wherein the hyperparameter modification is constrained to prevent overfitting of the modified hyperparameters to the second set of inputs.
1. A method for training a machine-trained (MT) network, the method comprising:
using a first set of inputs to train parameters of the MT network according to a set of hyperparameters that define aspects of the training by (i) computing a value of a first loss function based on propagation of the first set of inputs through the MT network and (ii) modifying the MT network parameters based on gradients of the first loss function with respect to the parameters at the computed value;
using a second set of inputs to validate the MT network as trained by the first set of inputs by:
propagating the second set of inputs through the MT network with the modified parameters to generate a second set of outputs; and
for each input of the second set of inputs, measuring a difference between (i) the output generated by propagating the input through the MT network with the modified parameters and (ii) an expected output for the input; and
based on the validation, modifying the hyperparameters for subsequent training of the MT network based on gradients of a description length score with respect to the hyperparameters, wherein the description length score constrains the hyperparameter modification to prevent overfitting of the modified hyperparameters to the second set of inputs by accounting for (i) the difference measurements for each input of the second set of inputs and (ii) the modifications to the hyperparameters.
2. The method of claim 1 , wherein the description length score (i) quantifies information provided to modify the hyperparameters and (ii) is minimized to constrain the hyperparameter modification.
3. The method of claim 2 , wherein the description length score further quantifies a measure of information required to provide data regarding new training inputs for the subsequent training of the MT network.
4. The method of claim 3 , wherein the new training inputs are part of the second set of inputs.
5. The method of claim 1 , wherein propagation of the first set of inputs through the MT network generates a first set of outputs and the computed value of the first loss function measures a difference, for each input of the first set of inputs, between the output generated by propagating the input through the MT network and an expected output for the input.
6. The method of claim 1 , wherein the description length score incorporates the first loss function to account for error due to modification of the hyperparameters.
7. The method of claim 6 , wherein gradients of the description length score with respect to the hyperparameters incorporate gradients of the first loss function with respect to the parameters accounting for modifications to the hyperparameters.
8. The method of claim 1 further comprising:
using a third set of inputs to further train the parameters of the MT network according to the modified set of hyperparameters;
using a fourth set of inputs to validate the MT network as trained by the third set of inputs; and
based on the validation with the fourth set of inputs, further modifying the hyperparameters for subsequent training of the MT network.
9. The method of claim 8 , wherein:
the third set of inputs comprises (i) the first set of inputs and (ii) a subset of the second set of inputs; and
the fourth set of inputs comprises the second set of inputs without the subset that is part of the third set of inputs.
10. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit trains a machine-trained (MT) network, the program comprising sets of instructions for:
using a first set of inputs to train parameters of the MT network according to a set of hyperparameters that define aspects of the training by i) computing a value of a first loss function based on propagation of the first set of inputs through the MT network and (ii) modifying the MT network parameters based on gradients of the first loss function with respect to the parameters at the computed value;
using a second set of inputs to validate the MT network as trained by the first set of inputs by:
propagating the second set of inputs through the MT network with the modified parameters to generate a second set of outputs; and
for each input of the second set of inputs, measuring a difference between (i) the output generated by propagating the input through the MT network with the modified parameters and (ii) an expected output for the input; and
based on the validation, modifying the hyperparameters for subsequent training of the MT network based on gradients of a description length score with respect to the hyperparameters, wherein the description length score constrains the hyperparameter modification to prevent overfitting of the modified hyperparameters to the second set of inputs by accounting for (i) the difference measurements for each input of the second set of inputs and (ii) the modifications to the hyperparameters.
11. The non-transitory machine-readable medium of claim 10 , wherein the description length score (i) quantifies information provided to modify the hyperparameters and (ii) is minimized to constrain the hyperparameter modification.
12. The non-transitory machine-readable medium of claim 11 , wherein the description length score further quantifies a measure of information required to provide data regarding new training inputs for the subsequent training of the MT network.
13. The non-transitory machine-readable medium of claim 12 , wherein the new training inputs are part of the second set of inputs.
14. The non-transitory machine-readable medium of claim 10 , wherein propagation of the first set of inputs through the MT network generates a first set of outputs and the computed value of the first loss function measures a difference, for each input of the first set of inputs, between the output generated by propagating the input through the MT network and an expected output for the input.
15. The non-transitory machine-readable medium of claim 10 , wherein the description length score incorporates the first loss function to account for error due to modification of the hyperparameters.
16. The non-transitory machine-readable medium of claim 15 , wherein gradients of the description length score with respect to the hyperparameters incorporate gradients of the first loss function with respect to the parameters accounting for modifications to the hyperparameters.
17. The non-transitory machine-readable medium of claim 10 , wherein the program further comprises sets of instructions for:
using a third set of inputs to further train the parameters of the MT network according to the modified set of hyperparameters;
using a fourth set of inputs to validate the MT network as trained by the third set of inputs; and
based on the validation with the fourth set of inputs, further modifying the hyperparameters for subsequent training of the MT network.
18. The non-transitory machine-readable medium of claim 17 , wherein:
the third set of inputs comprises (i) the first set of inputs and (ii) a subset of the second set of inputs; and
the fourth set of inputs comprises the second set of inputs without the subset that is part of the third set of inputs.