System and method for cross-modal knowledge transfer without task-relevant source data
A cross-modality knowledge transfer system is provided for adapting one or more source model networks to one or more target model networks. The system is configured to perform steps of providing the TI paired datasets through the source feature encoders of the one or more source model networks, extracting TI source features and TI source moments from the TI paired data by the BN layers of the one or more source model networks, providing the TI paired datasets and the unlabeled TR datasets through the one or more target model networks to extract TI target features and TR target moments, training jointly all the feature encoders of the one or more target model networks by matching the extracted TI target features and TR target moments with the TI source features and TI source moments along with mixing weights, and forming a final target model network by combining the trained one or more target model networks.
1 . A cross-modality knowledge transfer system for adapting one or more source model networks to one or more target model networks, comprising:
a memory configured to store task-irrelevant (TI) paired datasets, unlabeled task-relevant (TR) datasets, the one or more source model networks including batch normalization (BN) layers, feature encoders, convolutional neural network layers (CNN layers) and classifiers, the one or more target model networks including the BN layers, the feature encoders, the CNN layers and the classifiers, and a computer-implemented cross-modality knowledge transfer method having instructions;
at least one processor configured to perform steps of the computer-implemented cross-modality knowledge transfer method according to the instructions, wherein the steps comprise:
extracting TI source features from the one or more source model networks by passing the TI source paired datasets through the one or more source model networks, and extracting TR source moments from the BN layers of the one or more source model networks, wherein the CNN layers and the classifiers of the one or more source model networks are frozen;
extracting batch-wise TI target features and TR target moments from one or more target model networks by passing the TI paired datasets and the unlabeled TR datasets through the one or more target model networks, wherein the classifiers of the one or more target model networks are frozen;
computing modality-agnostic loss functions based on the extracted TR target features of the one or more target model networks;
computing modality-specific loss functions by computing distances between the extracted TI target features and TR target moments between the TI source features and TI source moments respectively;
training jointly the feature encoders of the one or more target model networks along with mixing weights by minimizing the computed modality-agnostic loss functions and modality-specific loss functions; and
generating a final target model network by combining the trained one or more target model networks.
2 . The cross-modality knowledge transfer system of claim 1 , further comprising a TI/TR dataset interface configured to receive the TI paired datasets and the unlabeled TR datasets via a communication network, wherein the TI/TR dataset interface is configured to store the received TI paired datasets and unlabeled TR datasets into the memory.
3 . The cross-modality knowledge transfer system of claim 1 , wherein the at least one processor further performs transmitting model parameters of the generated final target model network to another untrained one or more target model networks via a network.
4 . The cross-modality knowledge transfer system of claim 1 , wherein pairs of the task-irrelevant (TI) datasets and the unlabeled task-relevant (TR) datasets are RGB images and depth images, RGB images and infrared images, or RGB images and LIDAR point clouds.
5 . The cross-modality knowledge transfer system of claim 4 , wherein the unlabeled task-relevant (TR) datasets are used for computer vision tasks including image recognition, object recognition and scene recognition.
6 . The cross-modality knowledge transfer system of claim 1 , wherein the TI source features are extracted from the feature encoders of the one or more source model networks, wherein the TR source moments are extracted from the BN layers of the one or more source model networks, wherein the batch-wise TI target features are extracted from the feature encoders of the one or more target model networks, wherein the TR target moments are extracted from the BN layers of the one or more target model networks.
7 . The cross-modality knowledge transfer system of claim 1 , wherein the batch-wise TI target features and the TR target moments extracted from the one or more target model networks are respectively combined using mixing weights.
8 . The cross-modality knowledge transfer system of claim 1 , wherein the modality-agnostic and modality-specific loss functions are used to bridge a gap between the one or more source model networks and the one or more target model networks in feature space.
9 . The cross-modality knowledge transfer system of claim 8 , wherein a combination of the modality-agnostic loss functions includes entropy, pseudo-labeling and diversity, wherein the entropy, pseudo-labeling and diversity are defined on outputs of the one or more source model networks.
10 . The cross-modality knowledge transfer system of claim 8 , wherein a combination of the modality-specific loss functions includes feature distances between the extracted TI source features and TI target features, and distances between the TI source moments and TR target moments.
11 . A computer-implemented cross-modality knowledge transfer method having instructions using at least one processor and at least one memory, wherein the instructions comprising:
extracting Task-irrelevant (TI) source features from the one or more source model networks by passing the TI source paired datasets through the one or more source model networks, and extracting Task-relevant (TR) source moments from the BN layers of the one or more source model networks, wherein the CNN layers and the classifiers of the one or more source model networks are frozen;
extracting batch-wise TI target features and TR target moments from one or more target model networks by passing the TI paired datasets and the unlabeled TR datasets through the one or more target model networks, wherein the classifiers of the one or more target model networks are frozen;
computing modality-agnostic loss functions based on the extracted TR target features of the one or more target model networks;
computing modality-specific loss functions by computing distances between the extracted TI target features and TR target moments between the TI source features and TI source moments respectively;
training jointly the feature encoders of the one or more target model networks along with mixing weights by minimizing the computed modality-agnostic and modality-specific loss functions; and
generating a final target model network by combining the trained one or more target model networks.
12 . The computer-implemented cross-modality knowledge transfer method of claim 11 , further receiving, using a TI/TR dataset interface, the TI paired datasets and the unlabeled TR datasets via a communication network, wherein the TI/TR dataset interface is configured to store the received TI paired datasets and unlabeled TR datasets into the at least one memory.
13 . The computer-implemented cross-modality knowledge transfer method of claim 11 , wherein the at least one processor further performs transmitting model parameters of the generated final target model network to another untrained one or more target model networks via a network.
14 . The computer-implemented cross-modality knowledge transfer method of claim 11 , wherein pairs of the task-irrelevant (TI) datasets and the unlabeled task-relevant (TR) datasets are RGB images and depth images, RGB images and infrared images, or RGB images and LIDAR point clouds.
15 . The computer-implemented cross-modality knowledge transfer method of claim 14 , wherein the unlabeled task-relevant (TR) datasets are used for computer vision tasks including image recognition, object recognition and scene recognition.
16 . The computer-implemented cross-modality knowledge transfer method of claim 11 , wherein the TI source features are extracted from the feature encoders of the one or more source model networks, wherein the TR source moments are extracted from the BN layers of the one or more source model networks, wherein the batch-wise TI target features are extracted from the feature encoders of the one or more target model networks, wherein the TR target moments are extracted from the BN layers of the one or more target model networks.
17 . The computer-implemented cross-modality knowledge transfer method of claim 11 , wherein the batch-wise TI target features and the TR target moments extracted from the one or more target model networks are respectively combined using mixing weights.
18 . The computer-implemented cross-modality knowledge transfer method of claim 11 , wherein the modality-agnostic and modality-specific loss functions are used to bridge a gap between the one or more source model networks and the one or more target model networks in feature space.
19 . The computer-implemented cross-modality knowledge transfer method of claim 18 , wherein a combination of the modality-agnostic loss functions includes entropy, pseudo-labeling and diversity, wherein the entropy, pseudo-labeling and diversity are defined on outputs of the one or more source model networks.
20 . The computer-implemented cross-modality knowledge transfer method of claim 18 , wherein a combination of the modality-agnostic loss functions includes feature distances between the extracted TI source features and TI target features, and distances between the TI source moments and TR target moments.