Model disentanglement for domain adaptation
Certain aspects of the present disclosure provide techniques for improved domain adaptation in machine learning. A feature tensor is generated by processing input data using a feature extractor. A first set of logits is generated by processing the feature tensor using a domain-agnostic classifier, and a second set of logits is generated by processing the feature tensor using a domain-specific classifier. A loss is computed based at least in part on the first set of logits and the second set of logits, where the loss includes a divergence loss component. The feature extractor, the domain-agnostic classifier, and the domain-specific classifier are refined using the loss.
1. A computer-implemented method, comprising:
receiving first input data;
generating a first feature tensor by processing the first input data using a feature extractor;
generating a first set of logits by processing the first feature tensor using a domain-agnostic classifier;
generating a second set of logits by processing the first feature tensor using a domain-specific classifier;
computing a first loss based at least in part on the first set of logits and the second set of logits, wherein the first loss includes a first divergence loss component;
refining the feature extractor, the domain-agnostic classifier, and the domain-specific classifier using the first loss;
generating a second feature tensor by processing second input data using the feature extractor;
generating a third set of logits by processing the second feature tensor using the domain-agnostic classifier;
generating a fourth set of logits by processing the second feature tensor using the domain-specific classifier;
computing a second loss based on the third set of logits and the fourth set of logits, wherein the second loss includes a second divergence loss component; and
fine-tuning at least one of the feature extractor or the domain-specific classifier based on the second loss, wherein the domain-agnostic classifier is fixed during the fine-tuning, wherein the fine-tuning comprises at least one of:
upon determining that a number of labeled fine-tuning samples is less than a defined threshold, refining only the domain-specific classifier, wherein the feature extractor is fixed during the fine-tuning; and
upon determining that only unlabeled fine-tuning samples are available, refining only the domain-specific classifier, wherein the feature extractor is fixed during the fine-tuning.
2. The method of claim 1 , wherein the first loss further includes a first cross-entropy loss computed based on an aggregated set of logits including the first set of logits and the second set of logits.
3. The method of claim 2 , wherein the aggregated set of logits is generated by at least one of:
summing the first set of logits and the second set of logits;
concatenating the first set of logits and the second set of logits; or
combining the first set of logits and the second set of logits using an attention mechanism.
4. The method of claim 2 , wherein the first loss further includes a second cross-entropy loss computed based on the first set of logits.
5. The method of claim 1 , wherein the first input data corresponds to a source domain, the method further comprising fine-tuning at least one of the feature extractor or the domain-specific classifier for a target domain, wherein the domain-agnostic classifier is fixed during the fine-tuning.
6. The method of claim 5 , wherein the fine-tuning is performed as part of an enrollment process for a new user.
7. The method of claim 6 , wherein:
the source domain comprises biometric authentication for a first set of one or more users, and
the target domain comprises biometric authentication for the new user.
8. The method of claim 6 , wherein:
the source domain comprises voice verification for a first set of one or more users, and
the target domain comprises voice verification for the new user.
9. The method of claim 6 , wherein:
the source domain comprises attention verification for a first set of one or more users, and
the target domain comprises attention verification for the new user.
10. The method of claim 1 , wherein the first divergence loss component comprises an orthonormality loss.
11. The method of claim 1 , wherein the second loss comprises:
a first cross-entropy loss computed based on an aggregated set of logits including the third set of logits and the fourth set of logits; and
an orthonormality loss.
12. The method of claim 11 , wherein the second loss further comprises an information maximization loss.
13. The method of claim 12 , wherein the information maximization loss is defined based at least in part on pseudo labels for the second input data, and wherein the pseudo-labels are obtained by:
using soft probability outputs as weights to compute a cluster center of each of a plurality of classes; and
applying a nearest-neighbor approach to each respective sample in the second input data to identify a respective nearest cluster center, wherein the respective nearest cluster center is used as a respective pseudo-label for the respective sample.
14. The method of claim 1 , wherein the second loss comprises a third cross-entropy loss computed based on an aggregated set of logits including the third set of logits and the fourth set of logits.
15. The method of claim 1 , further comprising generating an inference, comprising:
generating an aggregated set of logits based on the third set of logits and the fourth set of logits; and
identifying a class with a largest logit in the aggregated set of logits.
16. A method, comprising:
generating a first feature tensor by processing first input data using a feature extractor;
generating a first set of logits by processing the first feature tensor using a domain-agnostic classifier;
generating a second set of logits by processing the first feature tensor using a domain-specific classifier;
computing a first loss based on the first set of logits and the second set of logits, wherein the first loss includes a first divergence loss component; and
fine-tuning at least one of the feature extractor or the domain-specific classifier based on the first loss, wherein the domain-agnostic classifier is fixed during the fine-tuning, and wherein the feature extractor, domain-agnostic classifier, and domain-specific classifier were pre-trained using at least a second input data, wherein the fine-tuning comprises at least one of:
upon determining that a number of labeled fine-tuning samples is less than a defined threshold, refining only the domain-specific classifier, wherein the feature extractor is fixed during the fine-tuning; and
upon determining that only unlabeled fine-tuning samples are available, refining only the domain-specific classifier, wherein the feature extractor is fixed during the fine-tuning.
17. The method of claim 16 , wherein, prior to the fine-tuning, the feature extractor, domain-agnostic classifier, and domain-specific classifier were pre-trained by:
generating a second feature tensor by processing the second input data using the feature extractor;
generating a third set of logits by processing the second feature tensor using the domain-agnostic classifier;
generating a fourth set of logits by processing the second feature tensor using the domain-specific classifier;
computing a second loss based at least in part on the third set of logits and the fourth set of logits, wherein the second loss includes a second divergence loss component; and
training the feature extractor, the domain-agnostic classifier, and the domain-specific classifier using the first loss.
18. The method of claim 16 , wherein the first loss further includes a first cross-entropy loss computed based on an aggregated set of logits including the first set of logits and the second set of logits.
19. The method of claim 18 , wherein the aggregated set of logits is generated by at least one of:
summing the first set of logits and the second set of logits;
concatenating the first set of logits and the second set of logits; or
combining the first set of logits and the second set of logits using an attention mechanism.
20. The method of claim 16 , wherein the first divergence loss component comprises an orthonormality loss.
21. The method of claim 16 , further comprising: upon determining that the first input data is not associated with a label, generating a pseudo-label for the first input data, wherein the first loss is based at least in part on the pseudo-label.
22. The method of claim 21 , wherein generating the pseudo label comprises:
using soft probability outputs as weights to compute a cluster center of each of a plurality of classes; and
applying a nearest-neighbor approach to the first input data to identify a nearest cluster center, wherein the nearest cluster center is used as the pseudo-label for the first input data.
23. A system, comprising:
a memory comprising computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions and cause the system to perform an operation comprising:
generating a first feature tensor by processing first input data using a feature extractor;
generating a first set of logits by processing the first feature tensor using a domain-agnostic classifier;
generating a second set of logits by processing the first feature tensor using a domain-specific classifier;
computing a first loss based at least in part on the first set of logits and the second set of logits, wherein the first loss includes a first divergence loss component;
refining the feature extractor, the domain-agnostic classifier, and the domain-specific classifier using the first loss;
generating a second feature tensor by processing second input data using the feature extractor;
generating a third set of logits by processing the second feature tensor using the domain-agnostic classifier;
generating a fourth set of logits by processing the second feature tensor using the domain-specific classifier;
computing a second loss based on the third set of logits and the fourth set of logits, wherein the second loss includes a second divergence loss component; and
fine-tuning at least one of the feature extractor or the domain-specific classifier based on the second loss, wherein the domain-agnostic classifier is fixed during the fine-tuning, wherein the fine-tuning comprises at least one of:
upon determining that a number of labeled fine-tuning samples is less than a defined threshold, refining only the domain-specific classifier, wherein the feature extractor is fixed during the fine-tuning; and
upon determining that only unlabeled fine-tuning samples are available, refining only the domain-specific classifier, wherein the feature extractor is fixed during the fine-tuning.
24. A system, comprising:
means for generating a first feature tensor by processing first input data using a feature extractor;
means for generating a first set of logits by processing the first feature tensor using a domain-agnostic classifier;
means for generating a second set of logits by processing the first feature tensor using a domain-specific classifier;
means for computing a first loss based at least in part on the first set of logits and the second set of logits, wherein the first loss includes a first divergence loss component;
means for refining the feature extractor, the domain-agnostic classifier, and the domain-specific classifier using the first loss;
means for generating a second feature tensor by processing second input data using the feature extractor;
means for generating a third set of logits by processing the second feature tensor using the domain-agnostic classifier;
means for generating a fourth set of logits by processing the second feature tensor using the domain-specific classifier;
means for computing a second loss based on the third set of logits and the fourth set of logits, wherein the second loss includes a second divergence loss component; and
means for fine-tuning at least one of the feature extractor or the domain-specific classifier based on the second loss, wherein the domain-agnostic classifier is fixed during the fine-tuning, wherein the means for fine-tuning is configured to at least one of:
upon determining that a number of labeled fine-tuning samples is less than a defined threshold, refine only the domain-specific classifier, wherein the feature extractor is fixed during the fine-tuning; and
upon determining that only unlabeled fine-tuning samples are available, refine only the domain-specific classifier, wherein the feature extractor is fixed during the fine-tuning.