IP Library › Granted Patent US 12,561,556
Granted Patent B2
US 12,561,556 · App. 17/475,003 · Granted Feb 24, 2026

Devices, systems, methods, and media for domain adaptation using hybrid learning

Inventors: Eduardo R Corral-Soto (Toronto, CA); Bingbing Liu (Markham, CA)
Assignee: HUAWEI TECHNOLOGIES CO., LTD.
G06N3/08G06N3/044
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,561,556
App. No.
17/475,003
Granted
Feb 24, 2026
Kind
B2
Abstract

Devices, systems, methods, and media are disclosed for domain adaptation of a trained machine learning model using hybrid learning. A hybrid approach to domain adaptation is disclosed that combines aspects of discrepancy-based, adversarial, and reconstruction-based approaches to achieve an end-to-end trained model for performing a prediction task (such as semantic segmentation) on a sparsely labeled dataset in a target domain, by leveraging a richly-labeled dataset in the source domain. Some embodiments may also provide a trained domain translation model for generating synthetic data samples in a first domain based on input data samples from a second domain.

Claims (173)

1 . A method for training a target-domain model, comprising:

obtaining a target-domain dataset comprising a plurality of target-domain data samples;

obtaining a source-domain dataset comprising a plurality of source-domain data samples, each source-domain data sample being associated with respective label information;

performing self-supervised learning to train a domain translation neural network, comprising a target-domain encoder subnetwork, a target-domain decoder subnetwork, a source-domain encoder subnetwork, and a source-domain decoder subnetwork, by:

training the target-domain encoder subnetwork to encode target-domain data, and training the target-domain decoder subnetwork to reconstruct target-domain data samples from encoded target-domain data, using one or more target-domain data samples of the target-domain dataset; and

training the source-domain encoder subnetwork to encode source-domain data, and training the source-domain decoder subnetwork to reconstruct source-domain data samples from encoded source-domain data, using one or more source-domain data samples of the source-domain dataset;

performing adversarial learning to train the domain translation neural network by:

training the target-domain encoder subnetwork to encode target-domain data, and training the source-domain decoder subnetwork to synthesize source-domain data samples from encoded target-domain data, using one or more target-domain data samples of the target-domain dataset; and

training the source-domain encoder subnetwork to encode source-domain data, and training the target-domain decoder subnetwork to synthesize target-domain data samples from encoded source-domain data, using one or more source-domain data samples of the source-domain dataset;

processing one or more source-domain data samples of the plurality of source-domain data samples, using the source-domain encoder subnetwork and the target-domain decoder subnetwork, to generate a respective one or more synthetic target domain data samples; and

training the target-domain model to perform a prediction task on target domain data using:

the one or more synthetic target-domain data samples; and

the respective label information of the one or more source-domain target samples.

2 . The method of claim 1 , wherein:

performing self-supervised learning to train the domain translation neural network comprises:

enabling one or more intra-target-domain skip connections between one or more intermediate layers of the target-domain encoder subnetwork and one or more intermediate layers of the target-domain decoder subnetwork;

repeating one or more times:

processing a source-domain data sample of the source-domain dataset using the target-domain encoder subnetwork and the target-domain decoder subnetwork to generate a reconstructed target-domain data sample;

generating a target data identity loss based on a comparison of the target-domain data sample and the reconstructed target-domain data sample; and

adjusting a plurality of learned parameters of the target-domain encoder subnetwork and the target-domain decoder subnetwork based on the target data identity loss;

enabling one or more intra-source-domain skip connections between one or more intermediate layers of the source-domain encoder subnetwork and one or more intermediate layers of the source-domain decoder subnetwork; and

repeating one or more times:

processing a source-domain data sample of the target-domain dataset using the target-domain encoder subnetwork and the source-domain decoder subnetwork to generate a reconstructed source-domain data sample;

generating a source data identity loss based on a comparison of the source-domain data sample and the reconstructed source-domain data sample; and

adjusting a plurality of learned parameters of the source-domain encoder subnetwork and the source-domain decoder subnetwork based on the source data identity loss; and

performing adversarial learning to train the domain translation neural network comprises:

training a target-domain discriminator model to identify target-domain data samples using one or more target-domain data samples of the target-domain dataset;

disabling the intra-source-domain skip connections;

enabling one or more cross-domain skip connections between one or more intermediate layers of the source-domain encoder subnetwork and one or more intermediate layers of the target-domain decoder subnetwork; and

repeating one or more times:

processing a source-domain data sample of the source-domain dataset using the source-domain encoder subnetwork and the target-domain decoder subnetwork to generate a synthetic target-domain data sample;

generating an adversarial loss, using the target-domain discriminator model, based on the synthetic target-domain data sample; and

adjusting a plurality of learned parameters of the source-domain encoder subnetwork and the target-domain decoder subnetwork based on the adversarial loss.

3 . The method of claim 2 , further comprising:

computing mean information and covariance information of the target-domain dataset; and

after each repetition of processing a source-domain data sample of the source-domain dataset using the source-domain encoder subnetwork and the target-domain decoder subnetwork to generate a synthetic target-domain data sample:

generating a statistical loss based on a comparison between:

mean information of the synthetic target-domain data sample and the mean information of the target-domain dataset; and

covariance information of the synthetic target-domain data sample and the covariance information of the target-domain dataset; and

wherein the plurality of learned parameters of the source-domain encoder subnetwork and the target-domain decoder subnetwork are adjusted based on the adversarial loss and the statistical loss.

4 . The method of claim 2 ,

further comprising obtaining a source-domain model trained to perform the prediction task on source-domain data;

wherein training the target-domain model to perform the prediction task on target domain data comprises performing unsupervised learning to train the target-domain model by:

repeating one or more times:

selecting a selected source-domain data sample from the source-domain dataset;

processing the selected source-domain data sample, using the source-domain model, to generate source-domain model prediction information;

processing the selected source-domain data sample, using the source-domain encoder subnetwork of the domain translation neural network, to generate a synthetic target-domain data sample;

processing the synthetic target-domain data sample, using the target-domain model, to generate target-domain model prediction information;

generating a task consistency loss based on a comparison of the source-domain model prediction information and the target-domain model prediction information;

generating a task loss based on a comparison of the target-domain model prediction information and the label information of the selected source-domain data sample; and

adjusting a plurality of learned parameters of the target-domain model, the source-domain encoder subnetwork, and the target-domain decoder subnetwork based on the task consistency loss and the task loss.

5 . The method of claim 4 , wherein:

each of one or more target-domain data samples of the target-domain dataset is associated with respective label information; and

training the target-domain model to perform the prediction task on target domain data further comprises, prior to performing unsupervised learning stage to train the target-domain model, performing semi-supervised learning to train the target-domain model by repeating one or more times:

selecting a selected target-domain data sample associated with respective label information from the target-domain dataset;

processing the selected target-domain data sample, using the target-domain model, to generate target-domain model prediction information;

generating a task loss based on a comparison of the target-domain model prediction information and the label information of the selected target-domain data sample; and

adjusting a plurality of learned parameters of the target-domain model based on the task loss.

6 . The method of claim 5 , wherein:

the target-domain model comprises an encoder subnetwork and a decoder subnetwork;

training the target-domain model to perform the prediction task on target domain data further comprises, prior to performing semi-supervised learning to train the target-domain model, performing self-supervised learning to train the target-domain model by:

providing an auxiliary decoder subnetwork; and

repeating one or more times:

selecting a selected target-domain data sample from the target-domain dataset;

processing the selected target-domain data sample, using the encoder subnetwork and the auxiliary decoder subnetwork, to generate a reconstructed selected target-domain data sample;

generating an identity loss based on a comparison of the reconstructed selected target-domain data sample and the selected target-domain data sample; and

adjusting a plurality of learned parameters of the encoder subnetwork and the auxiliary decoder subnetwork based on the identity loss.

7 . The method of claim 4 , wherein obtaining the source-domain model comprises training the source-domain model, using supervised learning, to perform the prediction task on source-domain data by:

repeating one or more times:

selecting a selected source-domain data sample from the target-domain dataset;

processing the selected source-domain data sample, using the source-domain model, to generate source-domain model prediction information;

generating a task loss based on a comparison of the source-domain model prediction information and the respective label information of the selected source-domain data sample; and

adjusting a plurality of learned parameters of the source-domain model based on the task loss.

8 . The method of claim 2 , wherein:

the plurality of target-domain data samples comprises a plurality of imaging data samples obtained in a first context, each imaging data sample comprising a two-dimensional matrix of data points, each data point having one or more channels;

the plurality of source-domain data samples comprises a plurality of imaging data samples obtained in a second context, each imaging data sample comprising a two-dimensional matrix of data points, each data point having one or more channels; and

the prediction task is a data point classification task for classifying data points of an imaging data sample.

9 . The method of claim 8 , wherein:

the plurality of imaging data samples obtained in the first context comprise a plurality of LIDAR point cloud frames obtained in the first context;

the plurality of imaging data samples obtained in the second context comprise a plurality of LIDAR point cloud frames obtained in the second context;

the data points of the imaging data samples are points of the point clouds; and

the prediction task is a semantic segmentation task for classifying each point of a point cloud frame into one of a plurality of classes.

10 . The method of claim 9 , wherein:

obtaining the target-domain dataset comprises preprocessing a plurality of raw target-domain data samples to generate the plurality of target-domain data samples;

obtaining the source-domain dataset comprises preprocessing a plurality of raw source-domain data samples to generate the plurality of source-domain data samples; and

preprocessing a data sample comprises performing spherical projection normalization on the data sample.

11 . A system for training a target-domain model, the system comprising:

a processor device; and

a memory storing:

a target-domain dataset comprising a plurality of target-domain data samples;

a source-domain dataset comprising a plurality of source-domain data samples, each source-domain data sample being associated with respective label information; and

machine-executable instructions which, when executed by the processor device, cause the system to:

perform self-supervised learning to train a domain translation neural network, comprising a target-domain encoder subnetwork, a target-domain decoder subnetwork, a source-domain encoder subnetwork, and a source-domain decoder subnetwork, by:

training the target-domain encoder subnetwork to encode target-domain data, and training the target-domain decoder subnetwork to reconstruct target-domain data samples from encoded target-domain data, using one or more target-domain data samples of the target-domain dataset; and

training the source-domain encoder subnetwork to encode source-domain data, and training the source-domain decoder subnetwork to reconstruct source-domain data samples from encoded source-domain data, using one or more source-domain data samples of the source-domain dataset;

perform adversarial learning to train the domain translation neural network by:

training the target-domain encoder subnetwork to encode target-domain data, and training the source-domain decoder subnetwork to synthesize source-domain data samples from encoded target-domain data, using one or more target-domain data samples of the target-domain dataset; and

training the source-domain encoder subnetwork to encode source-domain data, and training the target-domain decoder subnetwork to synthesize target-domain data samples from encoded source-domain data, using one or more source-domain data samples of the source-domain dataset;

process one or more source-domain data samples of the plurality of source-domain data samples, using the source-domain encoder subnetwork and the target-domain decoder subnetwork, to generate a respective one or more synthetic target domain data samples; and

train the target-domain model to perform a prediction task on target domain data using:

the one or more synthetic target-domain data samples; and

the respective label information of the one or more source-domain target samples.

12 . The system of claim 11 , wherein:

performing self-supervised learning to train the domain translation neural network comprises:

enabling one or more intra-target-domain skip connections between one or more intermediate layers of the target-domain encoder subnetwork and one or more intermediate layers of the target-domain decoder subnetwork;

repeating one or more times:

processing a source-domain data sample of the source-domain dataset using the target-domain encoder subnetwork and the target-domain decoder subnetwork to generate a reconstructed target-domain data sample;

generating a target data identity loss based on a comparison of the target-domain data sample and the reconstructed target-domain data sample; and

adjusting a plurality of learned parameters of the target-domain encoder subnetwork and the target-domain decoder subnetwork based on the target data identity loss;

enabling one or more intra-source-domain skip connections between one or more intermediate layers of the source-domain encoder subnetwork and one or more intermediate layers of the source-domain decoder subnetwork; and

repeating one or more times:

processing a source-domain data sample of the target-domain dataset using the target-domain encoder subnetwork and the source-domain decoder subnetwork to generate a reconstructed source-domain data sample;

generating a source data identity loss based on a comparison of the source-domain data sample and the reconstructed source-domain data sample; and

adjusting a plurality of learned parameters of the source-domain encoder subnetwork and the source-domain decoder subnetwork based on the source data identity loss; and

performing adversarial learning to train the domain translation neural network comprises:

training a target-domain discriminator model to identify target-domain data samples using one or more target-domain data samples of the target-domain dataset;

disabling the intra-source-domain skip connections;

enabling one or more cross-domain skip connections between one or more intermediate layers of the source-domain encoder subnetwork and one or more intermediate layers of the target-domain decoder subnetwork; and

repeating one or more times:

processing a source-domain data sample of the source-domain dataset using the source-domain encoder subnetwork and the target-domain decoder subnetwork to generate a synthetic target-domain data sample;

generating an adversarial loss, using the target-domain discriminator model, based on the synthetic target-domain data sample; and

adjusting a plurality of learned parameters of the source-domain encoder subnetwork and the target-domain decoder subnetwork based on the adversarial loss.

13 . The system of claim 12 , wherein the machine-executable instructions, when executed by the processor device, further cause the system to:

compute mean information and covariance information of the target-domain dataset; and

after each repetition of processing a source-domain data sample of the source-domain dataset using the source-domain encoder subnetwork and the target-domain decoder subnetwork to generate a synthetic target-domain data sample:

generate a statistical loss based on a comparison between:

mean information of the synthetic target-domain data sample and the mean information of the target-domain dataset; and

covariance information of the synthetic target-domain data sample and the covariance information of the target-domain dataset; and

wherein the plurality of learned parameters of the source-domain encoder subnetwork and the target-domain decoder subnetwork are adjusted based on the adversarial loss and the statistical loss.

14 . The system of claim 12 , wherein:

the machine-executable instructions, when executed by the processor device, further cause the system to obtain a source-domain model trained to perform the prediction task on source-domain data; and

training the target-domain model to perform the prediction task on target domain data comprises performing unsupervised learning to train the target-domain model by:

repeating one or more times:

selecting a selected source-domain data sample from the source-domain dataset;

processing the selected source-domain data sample, using the source-domain model, to generate source-domain model prediction information;

processing the selected source-domain data sample, using the source-domain encoder subnetwork of the domain translation neural network, to generate a synthetic target-domain data sample;

processing the synthetic target-domain data sample, using the target-domain model, to generate target-domain model prediction information;

generating a task consistency loss based on a comparison of the source-domain model prediction information and the target-domain model prediction information;

generating a task loss based on a comparison of the target-domain model prediction information and the label information of the selected source-domain data sample; and

adjusting a plurality of learned parameters of the target-domain model, the source-domain encoder subnetwork, and the target-domain decoder subnetwork based on the task consistency loss and the task loss.

15 . The system of claim 14 , wherein:

each of one or more target-domain data samples of the target-domain dataset is associated with respective label information; and

training the target-domain model to perform the prediction task on target domain data further comprises, prior to performing unsupervised learning stage to train the target-domain model, performing semi-supervised learning to train the target-domain model by repeating one or more times:

selecting a selected target-domain data sample associated with respective label information from the target-domain dataset;

processing the selected target-domain data sample, using the target-domain model, to generate target-domain model prediction information;

generating a task loss based on a comparison of the target-domain model prediction information and the label information of the selected target-domain data sample; and

adjusting a plurality of learned parameters of the target-domain model based on the task loss.

16 . The system of claim 15 , wherein:

the target-domain model comprises an encoder subnetwork and a decoder subnetwork;

training the target-domain model to perform the prediction task on target domain data further comprises, prior to performing semi-supervised learning to train the target-domain model, performing self-supervised learning to train the target-domain model by:

providing an auxiliary decoder subnetwork; and

repeating one or more times:

selecting a selected target-domain data sample from the target-domain dataset;

processing the selected target-domain data sample, using the encoder subnetwork and the auxiliary decoder subnetwork, to generate a reconstructed selected target-domain data sample;

generating an identity loss based on a comparison of the reconstructed selected target-domain data sample and the selected target-domain data sample; and

adjusting a plurality of learned parameters of the encoder subnetwork and the auxiliary decoder subnetwork based on the identity loss.

17 . The system of claim 12 , wherein:

the plurality of target-domain data samples comprises a plurality of LIDAR point cloud frames obtained in a first context;

the plurality of source-domain data samples comprises a plurality of LIDAR point cloud frames obtained in a second context; and

the prediction task is a semantic segmentation task for classifying each point of a point cloud frame into one of a plurality of classes.

18 . A non-transitory processor-readable medium having machine-executable instructions stored thereon which, when executed by a processor of a device, cause the device to:

obtain a target-domain dataset comprising a plurality of target-domain data samples;

obtain a source-domain dataset comprising a plurality of source-domain data samples, each source-domain data sample being associated with respective label information;

perform self-supervised learning to train a domain translation neural network, comprising a target-domain encoder subnetwork, a target-domain decoder subnetwork, a source-domain encoder subnetwork, and a source-domain decoder subnetwork, by:

training the target-domain encoder subnetwork to encode target-domain data, and training the target-domain decoder subnetwork to reconstruct target-domain data samples from encoded target-domain data, using one or more target-domain data samples of the target-domain dataset; and

training the source-domain encoder subnetwork to encode source-domain data, and training the source-domain decoder subnetwork to reconstruct source-domain data samples from encoded source-domain data, using one or more source-domain data samples of the source-domain dataset;

perform adversarial learning to train the domain translation neural network by:

training the target-domain encoder subnetwork to encode target-domain data, and training the source-domain decoder subnetwork to synthesize source-domain data samples from encoded target-domain data, using one or more target-domain data samples of the target-domain dataset; and

training the source-domain encoder subnetwork to encode source-domain data, and training the target-domain decoder subnetwork to synthesize target-domain data samples from encoded source-domain data, using one or more source-domain data samples of the source-domain dataset;

process one or more source-domain data samples of the plurality of source-domain data samples, using the source-domain encoder subnetwork and the target-domain decoder subnetwork, to generate a respective one or more synthetic target domain data samples; and

train a target-domain model to perform a prediction task on target domain data using:

the one or more synthetic target-domain data samples; and

the respective label information of the one or more source-domain target samples.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 19, 2021
From: CORRAL-SOTO, EDUARDO R.; LIU, BINGBING
To: HUAWEI TECHNOLOGIES CO., LTD.
Reel/Frame 057829/0434 →
Continuity (1)
Related Publication 20230082899A1 · Mar 16, 2023
References Cited (18)
US 20170147944A1 · Csurka et al. · 2017 [cited by applicant]
US 20200193269A1 · Park et al. · 2020 [cited by applicant]
US 20210201152A1 · Perez et al. · 2021 [cited by applicant]
US 20210216818A1 · Umeda et al. · 2021 [cited by applicant]
Han, X., Qi, L., Yu, Q., Zhou, Z., Zheng, Y., Shi, Y., & Gao, Y. (Jan. 18, 2021). Deep symmetric adaptation network for cross-modality medical image segmentation. arXiv.org. https://arxiv.org/abs/2101.06853 (Year: 2021). [cited by examiner]
Jenni, S., & Favaro, P. (Jun. 13, 2018). Self-supervised feature learning by learning to spot artifacts. arXiv.org. https://arxiv.org/abs/1806.05024v1 (Year: 2018). [cited by examiner]
Li, L., Gao, K., Cao, J., Huang, Z., Weng, Y., Mi, X., Yu, Z., Li, X., & xia, B. (Mar. 30, 2021). Progressive Domain Expansion Network for single domain generalization. arXiv.org. https://arxiv.org/abs/2103.16050 (Year:… [cited by examiner]
Gretton et al., A kernel method for the two-sample problem, JMLR 2008. [cited by applicant]
Hoffman et al, Cycada: Cycle-consistent adversarial domain adaptation. ICML Dec. 29, 2017. [cited by applicant]
Zhu et al., Unpaired Image-to-Image Translation using Cycle-Consitent Adversarial Networks 2017. [cited by applicant]
Baochen and Saenko. Deep coral: Correlation alignment for deep domain adaptation. ECCV 2016. [cited by applicant]
Wang, Mei, and Weihong Deng. “Deep visual domain adaptation: A survey.” Neurocomputing 2018. [cited by applicant]
Corral-Soto et al. LiDAR few-shot domain adaptation via integrated CycleGAN and 3D object detector with joint learning delay, ICRA 2021. [cited by applicant]
Triess et al., “A Survey on Deep Domain Adaptation for LiDAR Perception”. IV Workshops. Jun. 7, 2021. [cited by applicant]
Cortinhal et al., “SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving”. Jul. 9, 2020. [cited by applicant]
Sun, Yu, et al. “Test-time training with self-supervision for generalization under distribution shifts.” ICML 2020. [cited by applicant]
Jiang et al., Tsit: A simple and versatile framework for image-to-image translation. ECCV 2020. [cited by applicant]
Anonymous Author(s). “Unsupervised Domain Adaptation in LiDAR Semantic Segmentation with Self-Supervision and Gated Adapters”. 35th Conference on Neural Information Processing Systems (NeurIPS 2021). 2021. [cited by applicant]