Domain adaptation for structured output via disentangled representations
Systems and methods for domain adaptation for structured output via disentangled representations are provided. The system receives a ground truth of a source domain. The ground truth is used in a task loss function for a first convolutional neural network that predicts at least one output based on inputs from the source domain and a target domain. The system clusters the ground truth of the source domain into a predetermined number of clusters, and predicts, via a second convolutional neural network, a structure of label patches. The structure includes an assignment of each of the at least one output of the first convolutional neural network to the predetermined number of clusters. A cluster loss is computed for the predicted structure of label patches, and an adversarial loss function is applied to the predicted structure of label patches to align the source domain and the target domain on a structural level.
1. A method for domain adaptation for structured output via disentangled representations, comprising:
receiving a ground truth of a source domain, wherein the ground truth is used in a task loss function for a first convolutional neural network that predicts at least one output based on inputs from the source domain and a target domain;
clustering the ground truth of the source domain into a predetermined number of clusters;
predicting, via a second convolutional neural network whose inputs are connected to outputs of the first convolutional neural network, a structure of label patches, wherein the structure includes an assignment of each of the at least one output of the first convolutional neural network to the predetermined number of clusters;
computing, by a processor device, a cluster loss fhr the predicted structure of label patches, wherein the second convolutional neural network is trained by a training signal based on the cluster loss and is unused subsequent to training of a system framework including the first and second convolutional neural networks; and
applying an adversarial loss function to the predicted structure of label patches to align the source domain and the target domain on a structural level.
2. The method as recited in claim 1 , further comprising:
processing images from the target domain based on knowledge acquired in the source domain.
3. The method as recited in claim 1 , further comprising:
processing images from the target domain based on annotations acquired in the source domain.
4. The method as recited in claim 1 , wherein the predetermined number is K, wherein K is a hyperparameter chosen by a user.
5. The method as recited in claim 1 , wherein clustering the ground truth of the source domain into the predetermined number of clusters further comprises:
assigning a synthetic category to each of the label patches.
6. The method as recited in claim 1 , wherein clustering the ground truth of the source domain into the predetermined number of clusters further comprises:
extracting and clustering the label patches for semantic segmentation.
7. The method as recited in claim 1 , wherein clustering the ground truth of the source domain into the predetermined number of clusters provides supervision for the first convolutional network and enables an additional adversarial regularization between the source domain and the target domain.
8. The method as recited in claim 1 , wherein the ground truth is only available for the source domain.
9. The method as recited in claim 1 , further comprising:
implementing back-propagation based on the adversarial loss to modify parameters associated with the first convolutional network.
10. The method as recited in claim 1 , further comprising:
implementing back-propagation based on the adversarial loss to modify parameters associated with the second convolutional network.
11. A computer system for domain adaptation for structured output via disentangled representations, comprising:
a processor device operatively coupled to a memory device, the processor device being configured to:
receive a ground truth of a source domain, wherein the ground truth is used in a task loss function for a first convolutional neural network that predicts at least one output based on inputs from the source domain and a target domain;
cluster the ground truth of the source domain into a predetermined number of clusters;
predict, via a second convolutional neural network whose inputs are connected to outputs of the first convolutional neural network, a structure of label patches, wherein the structure includes an assignment of each of the at least one output of the first convolutional neural network to the predetermined number of clusters;
compute a cluster loss for the predicted structure of label patches. wherein the second convolutional neural network is trained by a training sigual based on the cluster loss and unused subsequent to training of a system framework including the first and second convolutional neural networks; and
apply an adversarial loss function to the predicted structure of label patches to align the source domain and the target domain on a structural level.
12. The system as recited in claim 11 , wherein the predetermined number is K, wherein K is a hyperparameter chosen by a user.
13. The system as recited in claim 11 , wherein, when clustering the ground truth of the source domain into the predetermined number of clusters, the processor device is further configured to:
assign a synthetic category to each of the label patches.
14. The system as recited in claim 11 , wherein, when clustering the ground truth of the source domain into the predetermined number of clusters, the processor device is further configured to:
extract and cluster the label patches for semantic segmentation.
15. The system as recited in claim 11 , wherein, when clustering the ground truth of the source domain into the predetermined number of clusters, the processor device is further configured to:
provide supervision for the first convolutional network and enable an additional adversarial regularization between the source domain and the target domain.
16. A computer program product for domain adaptation for structured output via disentangled representations, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to perform the method comprising:
receiving a ground. truth of a source domain, wherein the ground truth is used in a task loss function for a first convolutional neural network that predicts at least one output based on inputs from the source domain and a target domain;
clustering the ground truth of the source domain into a predetermined number of clusters;
predicting, via a second convolutional neural network whose inputs are connected to outputs of the first convolutional neural network, a structure of label patches, wherein the structure includes an assignment of each of the at least one output of the first convolutional neural network to the predetermined number of clusters;
computing, by a processor device, a cluster loss for the predicted structure of label patches, wherein the second convolutional neural network is trained by a training signal based on the cluster loss and is unused subsequent to training of system framework including the first and second convolutional neural networks; and
applying an adversarial loss function to the predicted structure of label patches to align the source domain and the target domain on a structural level.