System and method for contrastive affinity learning via auxiliary prompts for generalized novel category discovery
A system and method of fine-grained image category discovery with few human annotations includes a camera and a trained machine learning model, which predicts a label for an object in a captured image and outputs the predicted label. The machine learning model is trained by contrastive affinity learning, including retrieving images having an object, a warm-up stage in which semi-supervised contrastive learning is performed based on projected features of a class token and an ensembled prompt, respectively. In a contrastive affinity learning stage, a student model and an exponentially moving averaged teacher model are forwarded with different augmented views of the retrieved images. Teacher embeddings are enqueued into a token-specific memory. A semi-supervised contrastive loss is computed on a current batch and a contrastive affinity learning loss for student embeddings and the teacher embeddings with pseudo-labels from a affinity graph dynamically generated by semi-supervised affinity generation.
1 . A method of fine-grained image categorization, comprising:
capturing an image by a camera;
predicting, via a trained machine learning model, a label for at least one object in the captured image;
outputting the predicted label as a category of the captured image,
wherein the trained machine learning model is trained by contrastive affinity learning by processing circuitry configured to
retrieve, from an image database, a plurality of images having at least one object;
perform, in a warm-up stage, semi-supervised contrastive learning based on projected features of a class token and an ensembled prompt, respectively;
at each iteration, in a contrastive affinity learning stage,
forward a student model and an exponentially moving averaged (EMA) teacher model with different augmented views of the retrieved images;
enqueue teacher embeddings output from the teacher model into a corresponding token-specific memory; and
compute semi-supervised contrastive loss on a current batch and a contrastive affinity learning (CAL) loss for student embeddings output from the student model and the teacher embeddings with pseudo-labels from a affinity graph dynamically generated by semi-supervised affinity generation.
2 . The method of claim 1 , wherein the semi-supervised affinity generation includes:
generating, by the processing circuitry, affinity graphs for a class embedding and a prompt embedding respectively via affinity propagation with label constraints on corresponding consensus K nearest neighbor (KNN) graphs.
3 . The method of claim 1 , wherein both the warm-up stage and the contrastive affinity learning stage include calculating a discriminative prompt regularization (DPR) loss.
4 . The method of claim 1 , wherein the contrastive affinity learning stage includes dynamically constructing a sub-graph sub-sampled from the affinity graph supported by an extra embedding memory bank M and the EMA teacher model.
5 . The method of claim 1 , further including
gradually calibrating, based on the CAL loss, a semantic representation by learning from generated affinity constraints in the affinity graph.
6 . The method of claim 1 , wherein the retrieved plurality of images have a label ratio of fifty percent.
7 . The method of claim 1 , wherein the predicting adopts a class embedding as a final label prediction.
8 . A mobile device system, comprising:
a camera for capturing an image;
mobile device processing circuitry configured to
predict, using a trained machine learning model, a label for at least one object in the captured image;
display the predicted label in conjunction with the image;
machine learning processing circuitry configured to train the machine learning model by contrastive affinity learning including
retrieving, from an image database, a plurality of images having at least one object;
performing, in a warm-up stage, semi-supervised contrastive learning based on projected features of a class token and an ensembled prompt, respectively;
at each iteration, in a contrastive affinity learning stage,
forwarding a student model and an exponentially moving averaged (EMA) teacher model with different augmented views of the retrieved images;
enqueuing teacher embeddings output from the teacher model into a corresponding token-specific memory; and
computing semi-supervised contrastive loss on a current batch and a contrastive affinity learning (CAL) loss for student embeddings output from the student model and the teacher embeddings with pseudo-labels from a affinity graph dynamically generated by semi-supervised affinity generation.
9 . The mobile device system of claim 8 , wherein the semi-supervised affinity generation includes generating, by the machine learning processing circuitry, affinity graphs for a class embedding and a prompt embedding respectively via affinity propagation with label constraints on corresponding consensus K nearest neighbor (KNN) graphs.
10 . The mobile device system of claim 8 , wherein in both the warm-up stage and the contrastive affinity learning stage, the machine learning processing circuitry is configured to calculate a discriminative prompt regularization (DPR) loss.
11 . The mobile device system of claim 8 , wherein in the contrastive affinity learning stage, the machine learning processing circuitry is configured to dynamically construct a sub-graph sub-sampled from the affinity graph supported by an extra embedding memory bank M and the EMA teacher model.
12 . The mobile device system of claim 8 , wherein the machine learning processing circuitry is further configured to gradually calibrate, based on the CAL loss, a semantic representation by learning from generated affinity constraints in the affinity graph.
13 . The mobile device system of claim 8 , wherein the retrieved plurality of images have a label ratio of fifty percent.
14 . The mobile device system of claim 8 , wherein the mobile device processing circuitry is further configured to adopt a class embedding as a final label prediction.
15 . A system, comprising:
a plurality of camera equipped devices for capturing images of an object;
an image database for storing the captured images;
processing circuitry configured to train a machine learning model by contrastive affinity learning including
retrieving, from the image database, a plurality of the captured images;
performing, in a warm-up stage, semi-supervised contrastive learning based on projected features of a class token and an ensembled prompt, respectively;
at each iteration, in a contrastive affinity learning stage,
forwarding a student model and an exponentially moving averaged (EMA) teacher model with different augmented views of the retrieved images;
enqueuing teacher embeddings output from the teacher model into a corresponding token-specific memory; and
computing semi-supervised contrastive loss on a current batch and a contrastive affinity learning (CAL) loss for student embeddings output from the student model and the teacher embeddings with pseudo-labels from a affinity graph dynamically generated by semi-supervised affinity generation; and
processing circuitry configured to
predict, using the trained machine learning model, a label for at least one object in the captured images;
output the predicted label in conjunction with the captured images.
16 . The system of claim 15 , wherein in the semi-supervised affinity generation, the processing circuitry is configured to generate affinity graphs for a class embedding and a prompt embedding respectively via affinity propagation with label constraints on corresponding consensus K nearest neighbor (KNN) graphs.
17 . The system of claim 15 , wherein in both the warm-up stage and the contrastive affinity learning stage, the processing circuitry is configured to calculate a discriminative prompt regularization (DPR) loss.
18 . The system of claim 15 , wherein in the contrastive affinity learning stage, the processing circuitry is configured to dynamically construct a sub-graph sub-sampled from the affinity graph supported by an extra embedding memory bank M and the EMA teacher model.
19 . The system of claim 15 , wherein the processing circuitry is further configured to gradually calibrate, based on the CAL loss, a semantic representation by learning from generated affinity constraints in the affinity graph.
20 . The system of claim 15 , wherein the retrieved plurality of captured images have a label ratio of fifty percent.