Techniques for machine learning model selection for domain generalization
A computing device may perform training of a set of machine learning models on a first data set associated with a first domain. In some examples, the training may include, for each machine learning model of the set of machine learning models, inputting, as values for a set of parameters of the respective sets of parameters and for an iteration of a set of iterations, a moving average of the set of parameters calculated over a threshold number of previous iterations. The computing device may select a set of model states that are generated during the training of the plurality of machine learning models based on a validation performance of the set of model states performed during the training. The computing device may then generate an ensembled machine learning model by aggregating the set of machine learning models corresponding to the set of selected model states.
1. A method for machine learning model training, comprising:
performing training of a plurality of machine learning models on a first data set associated with a first domain, wherein the plurality of machine learning models comprises respective sets of parameters that are updated across a plurality of iterations during the training, wherein the training comprises, for each machine learning model of the plurality of machine learning models, inputting, as values for a set of parameters of the respective sets of parameters and for an iteration of the plurality of iterations, a moving average of the set of parameters calculated over a threshold number of previous iterations;
selecting a plurality of model states that are generated during the training of the plurality of machine learning models, wherein the plurality of model states are selected based at least in part on a validation performance of the plurality of model states performed during the training;
generating an ensembled machine learning model by aggregating the plurality of machine learning models corresponding to the plurality of selected model states; and
performing a machine learning prediction using the ensembled machine learning model on a second data set associated with a second domain different from the first domain, wherein an output of the ensembled machine learning model is a dimension-wise average of respective outputs from the plurality of machine learning models in the ensembled machine learning model.
2. The method of claim 1 , further comprising:
determining, for one or more iterations of the plurality of iterations, a validation performance value associated with a current state of a machine learning model of the plurality of machine learning models; and
selecting a model state for one or more machine learning models of the plurality of machine learning models based on a highest validation performance value for the corresponding machine learning model across all iterations of the plurality of iterations.
3. The method of claim 1 , further comprising:
determining, for a first iteration following the threshold number of previous iterations, a first set of values for the set of parameters of a machine learning model of the plurality of machine learning models; and
determining, for a second iteration following the first iteration, a second set of values for the set of parameters of the machine learning model based at least in part on the moving average of the set of parameters calculated during the first iteration.
4. The method of claim 1 , further comprising:
starting a calculation of the moving average of the set of parameters after a configured number of iterations from a starting iteration of the plurality of iterations.
5. The method of claim 1 , further comprising:
determining, for a first iteration, a first validation performance value associated with a first state of a machine learning model of the plurality of machine learning models;
determining, for a second iteration following the first iteration, a second validation performance value associated with a second state of the machine learning model of the plurality of machine learning models; and
selecting the first state of the machine learning model for generation of the ensembled machine learning model based at least in part on determining that the second validation performance value is less than the first validation performance value.
6. The method of claim 1 , wherein the plurality of machine learning models are trained using a gradient based technique.
7. The method of claim 1 , wherein the respective outputs from the plurality of machine learning models comprises a respective vector for each machine learning model and the output of the ensembled machine learning model comprises the dimension-wise average of the respective vector.
8. An apparatus for machine learning model training, comprising:
a processor;
memory coupled with the processor; and
instructions stored in the memory and executable by the processor to cause the apparatus to:
perform training of a plurality of machine learning models on a first data set associated with a first domain, wherein the plurality of machine learning models comprises respective sets of parameters that are updated across a plurality of iterations during the training, wherein the training comprises, for each machine learning model of the plurality of machine learning models, inputting, as values for a set of parameters of the respective sets of parameters and for an iteration of the plurality of iterations, a moving average of the set of parameters calculated over a threshold number of previous iterations;
select a plurality of model states that are generated during the training of the plurality of machine learning models, wherein the plurality of model states are selected based at least in part on a validation performance of the plurality of model states performed during the training;
generate an ensembled machine learning model by aggregating the plurality of machine learning models corresponding to the plurality of selected model states; and
perform a machine learning prediction using the ensembled machine learning model on a second data set associated with a second domain different from the first domain, wherein an output of the ensembled machine learning model is a dimension-wise average of respective outputs from the plurality of machine learning models in the ensembled machine learning model.
9. The apparatus of claim 8 , wherein the instructions are further executable by the processor to cause the apparatus to:
determine, for one or more iterations of the plurality of iterations, a validation performance value associated with a current state of a machine learning model of the plurality of machine learning models; and
select a model state for one or more machine learning models of the plurality of machine learning models based on a highest validation performance value for the corresponding machine learning model across all iterations of the plurality of iterations.
10. The apparatus of claim 8 , wherein the instructions are further executable by the processor to cause the apparatus to:
determine, for a first iteration following the threshold number of previous iterations, a first set of values for the set of parameters of a machine learning model of the plurality of machine learning models; and
determine, for a second iteration following the first iteration, a second set of values for the set of parameters of the machine learning model based at least in part on the moving average of the set of parameters calculated during the first iteration.
11. The apparatus of claim 8 , wherein the instructions are further executable by the processor to cause the apparatus to:
start a calculation of the moving average of the set of parameters after a configured number of iterations from a starting iteration of the plurality of iterations.
12. The apparatus of claim 8 , wherein the instructions are further executable by the processor to cause the apparatus to:
determine, for a first iteration, a first validation performance value associated with a first state of a machine learning model of the plurality of machine learning models;
determine, for a second iteration following the first iteration, a second validation performance value associated with a second state of the machine learning model of the plurality of machine learning models; and
select the first state of the machine learning model for generation of the ensembled machine learning model based at least in part on determining that the second validation performance value is less than the first validation performance value.
13. The apparatus of claim 8 , wherein the plurality of machine learning models are trained using a gradient based technique.
14. The apparatus of claim 8 , wherein the respective outputs from the plurality of machine learning models comprises a respective vector for each machine learning model and the output of the ensembled machine learning model comprises the dimension-wise average of the respective vector.
15. A non-transitory computer-readable medium storing code for machine learning model training, the code comprising instructions executable by a processor to:
perform training of a plurality of machine learning models on a first data set associated with a first domain, wherein the plurality of machine learning models comprises respective sets of parameters that are updated across a plurality of iterations during the training, wherein the training comprises, for each machine learning model of the plurality of machine learning models, inputting, as values for a set of parameters of the respective sets of parameters and for an iteration of the plurality of iterations, a moving average of the set of parameters calculated over a threshold number of previous iterations;
select a plurality of model states that are generated during the training of the plurality of machine learning models, wherein the plurality of model states are selected based at least in part on a validation performance of the plurality of model states performed during the training;
generate an ensembled machine learning model by aggregating the plurality of machine learning models corresponding to the plurality of selected model states; and
perform a machine learning prediction using the ensembled machine learning model on a second data set associated with a second domain different from the first domain, wherein an output of the ensembled machine learning model is a dimension-wise average of respective outputs from the plurality of machine learning models in the ensembled machine learning model.
16. The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the processor to:
determine, for one or more iterations of the plurality of iterations, a validation performance value associated with a current state of a machine learning model of the plurality of machine learning models; and
select a model state for one or more machine learning models of the plurality of machine learning models based on a highest validation performance value for the corresponding machine learning model across all iterations of the plurality of iterations.
17. The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the processor to:
determine, for a first iteration following the threshold number of previous iterations, a first set of values for the set of parameters of a machine learning model of the plurality of machine learning models; and
determine, for a second iteration following the first iteration, a second set of values for the set of parameters of the machine learning model based at least in part on the moving average of the set of parameters calculated during the first iteration.
18. The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the processor to:
start a calculation of the moving average of the set of parameters after a configured number of iterations from a starting iteration of the plurality of iterations.
19. The non-transitory computer-readable medium of claim 15 , wherein the instructions are further executable by the processor to:
determine, for a first iteration, a first validation performance value associated with a first state of a machine learning model of the plurality of machine learning models;
determine, for a second iteration following the first iteration, a second validation performance value associated with a second state of the machine learning model of the plurality of machine learning models; and
select the first state of the machine learning model for generation of the ensembled machine learning model based at least in part on determining that the second validation performance value is less than the first validation performance value.
20. The non-transitory computer-readable medium of claim 15 , wherein the plurality of machine learning models are trained using a gradient based technique.