Method and system for adaptation of a trained object detection model to account for domain shift
The present disclosure provides a method and system for adapting a machine learning model, such as an object detection model, to account for domain shift. The method includes receiving a labeled data elements and target image samples and performing a plurality of model adaptation epochs. Each adaptation epoch includes: predicting for each of the target image samples, using the machine learning model configured by a current set of configuration parameters, a corresponding target class label for the respective target data object included in the target image sample; generating a plurality of labeled mixed data elements that each include: (i) a mixed image sample including a source data object from one of the source image samples and a target data object from one of the target image samples, and (ii) the corresponding source class label for the source data object and the corresponding target class label for the target data object. The method also includes adjusting the current set of configuration parameters to minimize a loss function for the machine learning model for the plurality of mixed data elements. The method results in adapted machine learning model that accounts for domain shift and that has improved performance at inference on new target image samples.
1. A computer implemented method for adapting a machine learning model that is configured by a learned set of configuration parameters, comprising:
receiving a plurality of labeled data elements that each include: (i) a source image sample including a respective source data object, and (ii) a corresponding source class label for the respective source data object;
receiving a plurality of target image samples each including a respective target data object;
performing a plurality of model adaptation epochs, wherein for an initial model adaptation epoch the learned set of configuration parameters is used as a current set of configuration parameters for the machine learning model, each model adaptation epoch comprising:
predicting for each of the plurality of targets image samples, using the machine learning model configured by the current set of configuration parameters, a corresponding target class label for the respective target data object included in the target image sample;
generating a plurality of labeled mixed data elements that each include: (i) a mixed image sample including a source data object from one of the source image samples and a target data object from one of the target image samples, and (ii) the corresponding source class label for the source data object and the corresponding target class label for the target data object; and
adjusting the current set of configuration parameters to minimize a loss function for the machine learning model for the plurality of labeled mixed data elements;
and
when performing the plurality of model adaptation epochs is completed, outputting the current set of configuration parameters as a final set of adapted configuration parameters for the machine learning model.
2. The method of claim 1 wherein the machine learning model comprises a plurality of first type layers performing a first type of computation and a plurality of second type layers performing a second type of computation, the first type layers being configured by a first group of the configuration parameters and the second type layers being configured by a second group of the configuration parameters, wherein for an initial set of the plurality of model adaptation epochs, adjusting the current set of configuration parameters comprises adjusting the second group of the configuration parameters while maintaining constant values for the first group of the configuration parameters; and for a further set of the plurality of model adaptation epochs, adjusting the current set of configuration parameters comprises adjusting the both first group of the configuration parameters and the second group of the configuration parameters.
3. The method of claim 2 wherein the first type layers perform matrix multiplication computations, the first group of the configuration parameters including weights for the matrix multiplication computations; and the second type layers perform batch normalization computations, the second group of configuration parameters including batch normalization beta and gamma parameters.
4. The method of claim 2 wherein the machine learning model is a deep learning neural network model comprising multiple network computational blocks that each include one of the first type of layers and one of the second type of layers.
5. The method of claim 1 wherein:
the data elements each include a source bounding box definition associated with each source class label to indicate a location of the respective source data object within the source image sample;
the mixed image samples are each composite image samples that include image portions from one of the source image samples and from one of the target image samples; and
for each model adaptation epoch,
predicting for each of the plurality of target image samples also comprises predicting a target bounding box definition associated with the corresponding target class label, the target bounding box indicating a location of the respective target data object within the target image sample;
the plurality of labeled mixed data elements each include: (i) a corresponding source bounding box definition that indicates a location of the at least one source data object within the mixed image sample of the labeled mixed data element; (ii) a corresponding target bounding box definition that indicates a location of the at least one target data object within the mixed image sample of the labeled mixed data element.
6. The method of claim 1 wherein at least some of the mixed image samples of the plurality of mixed data elements include a plurality of source data objects from one or more of the source image samples, and a plurality of target data objects from one or more of the target image samples, and the mixed data elements each include corresponding source class labels for each source data object included in the mixed image sample of the mixed data element and corresponding target class labels for each target data object included in the mixed image sample of the mixed data element.
7. The method of claim 1 wherein adjusting the current set of configuration parameters to minimize a loss function for the machine learning model is performed over a set of iterations that are performed for a respective batch of the of labeled mixed data elements.
8. The method of claim 1 further comprising adapting a second machine learning model that is configured by a learned set of second configuration parameters by:
using the machine learning model with the final set of adapted configuration parameters to predict, for each of the plurality of targets image samples, a corresponding target class label for the respective target data object included in the target image sample;
generating a further plurality of labeled mixed data elements that each include: (i) a mixed image sample including a source data object from one of the source image samples and a target data object from one of the target image samples, and (ii) the corresponding source class label for the source data object and the corresponding target class label for the target data object; and
adjusting the set of second configuration parameters to minimize a loss function for the second machine learning model for the further plurality of labeled mixed data elements.
9. The method of claim 8 wherein the second machine learning model has a smaller neural network size than the machine learning model with the final set of adapted configuration parameters.
10. The method of claim 1 further comprising:
receiving a request through a network from computing device to adapt the machine learning model that is configured by a learned set of configuration parameters; and
returning the final set of adapted configuration parameters for the machine learning model to the requesting computer device.
11. The method of claim 1 wherein generating the plurality of labeled mixed data elements includes downscaling a source image portion that includes the source data object.
12. A processing system for adapting a machine learning model that is configured by a learned set of configuration parameters, the processing system, including one or more processing circuits and one or more non-volatile memories storing computer software instructions that, when executed by the one or more processing circuits, configure the processing system to:
receive a plurality of labeled data elements that each include: (i) a source image sample including a respective source data object, and (ii) a corresponding source class label for the respective source data object;
receive a plurality of target image samples each including a respective target data object;
perform a plurality of model adaptation epochs, wherein for an initial model adaptation epoch the learned set of configuration parameters is used as a current set of configuration parameters for the machine learning model, each model adaptation epoch comprising:
predicting for each of the plurality of targets image samples, using the machine learning model configured by the current set of configuration parameters, a corresponding target class label for the respective target data object included in the target image sample;
generating a plurality of labeled mixed data elements that each include: (i) a mixed image sample including a source data object from one of the source image samples and a target data object from one of the target image samples, and (ii) the corresponding source class label for the source data object and the corresponding target class label for the target data object; and
adjusting the current set of configuration parameters to minimize a loss function for the machine learning model for the plurality of labeled mixed data elements;
and
when the plurality of model adaptation epochs are completed, output the current set of configuration parameters as a final set of adapted configuration parameters for the machine learning model.
13. The system of claim 12 wherein the machine learning model comprises a plurality of first type layers performing a first type of computation and a plurality of second type layers performing a second type of computation, the first type layers being configured by a first group of the configuration parameters and the second type layers being configured by a second group of the configuration parameters, wherein for an initial set of the plurality of model adaptation epochs, adjusting the current set of configuration parameters comprises adjusting the second group of the configuration parameters while maintaining constant values for the first group of the configuration parameters; and for a further set of the plurality of model adaptation epochs, adjusting the current set of configuration parameters comprises adjusting the both first group of the configuration parameters and the second group of the configuration parameters.
14. The system of claim 13 wherein the first type layers perform matrix multiplication computations, the first group of the configuration parameters including weights for the matrix multiplication computations; and the second type layers perform batch normalization computations, the second group of configuration parameters including batch normalization beta and gamma parameters.
15. The system of claim 13 wherein the machine learning model is a deep learning neural network model comprising multiple network computational blocks that each include one of the first type of layers and one of the second type of layers.
16. The system of claim 12 wherein:
the data elements each include a source bounding box definition associated with each source class label to indicate a location of the respective source data object within the image sample;
the mixed image samples are each composite image samples that include image portions from one of the source image samples and from one of the target image samples; and
for each model adaptation epoch,
predicting for each of the plurality of target image samples also comprises predicting a target bounding box definition associated with the corresponding target class label, the target bounding box indicating a location of the respective target data object within the target image sample;
the plurality of labeled mixed data elements each include: (i) a corresponding source bounding box definition that indicates a location of the at least one source data object within the mixed image sample of the labeled mixed data element; (ii) a corresponding target bounding box definition that indicates a location of the at least one target data object within the mixed image sample of the labeled mixed data element.
17. The system of claim 12 wherein at least some of the mixed image samples of the plurality of mixed data elements include a plurality of source data objects from one or more of the source image samples, and a plurality of target data objects from one or more of the target image samples, and the mixed data elements each include corresponding source class labels for each source data object included in the mixed image sample of the mixed data element and corresponding target class labels for each target data object included in the mixed image sample of the mixed data element.
18. The system of claim 12 wherein the computer software instructions, when executed by the one or more processing circuits, further configure the processing system to: adapt a second machine learning model that is configured by a learned set of second configuration parameters by:
using the machine learning model with the final set of adapted configuration parameters to predict, for each of the plurality of targets image samples, a corresponding target class label for the respective target data object included in the target image sample;
generating a further plurality of labeled mixed data elements that each include: (i) a mixed image sample including a source data object from one of the source image samples and a target data object from one of the target image samples, and (ii) the corresponding source class label for the source data object and the corresponding target class label for the target data object; and
adjusting the set of second configuration parameters to minimize a loss function for the second machine learning model for the further plurality of labeled mixed data elements,
wherein the second machine learning model has a smaller neural network size than the machine learning model with the final set of adapted configuration parameters.
19. The system of claim 12 wherein the computer software instructions, when executed by the one or more processing circuits, further configure the processing system to:
receive a request through a network from computing device to adapt the machine learning model that is configured by a learned set of configuration parameters; and
return the final set of adapted configuration parameters for the machine learning model to the requesting computer device.
20. A non-transitory computer readable medium storing computer software instructions that, when executed by the one or more processing circuits of a processing system, cause the processing system to perform a method for adapting a machine learning model that is configured by a learned set of configuration parameters, the method comprising:
receiving a plurality of labeled data elements that each include: (i) a source image sample including a respective source data object, and (ii) a corresponding source class label for the respective source data object;
receiving a plurality of target image samples each including a respective target data object;
performing a plurality of model adaptation epochs, wherein for an initial model adaptation epoch the learned set of configuration parameters is used as a current set of configuration parameters for the machine learning model, each model adaptation epoch comprising:
predicting for each of the plurality of targets image samples, using the machine learning model configured by the current set of configuration parameters, a corresponding target class label for the respective target data object included in the target image sample;
generating a plurality of labeled mixed data elements that each include: (i) a mixed image sample including a source data object from one of the source image samples and a target data object from one of the target image samples, and (ii) the corresponding source class label for the source data object and the corresponding target class label for the target data object; and
adjusting the current set of configuration parameters to minimize a loss function for the machine learning model for the plurality of labeled mixed data elements;
and
when performing the plurality of model adaptation epochs is completed, outputting the current set of configuration parameters as a final set of adapted configuration parameters for the machine learning model.