Systems and methods for face annotation

1 . A method for face annotation, comprising:

receiving an annotated image and a caption corresponding to the annotated image, wherein the caption includes a plurality of phrases;

generating a face image by cropping the annotated image based on a face detection algorithm;

extracting a phrase of the plurality of phrases from the caption;

computing a phrase similarity score by comparing an image embedding of the face image to a phrase embedding of the phrase in a multi-modal embedding space; and

adding the face image with the phrase to a set of annotated face images based on the phrase similarity score.

2 . The method of claim 1 , wherein generating the face image comprises:

identifying a bounding box for a face in the annotated image using the face detection algorithm; and

cropping the annotated image based on the bounding box.

3 . The method of claim 1 , further comprising:

embedding the face image using a multi-modal encoder to obtain the image embedding; and

embedding the phrase using the multi-modal encoder to obtain the phrase embedding.

4 . The method of claim 1 , wherein computing the phrase similarity score comprises:

computing a cosine similarity between the image embedding and the phrase embedding, wherein the phrase similarity score is based on the cosine similarity.

5 . The method of claim 1 , wherein adding the face image with the phrase to the set of annotated face images comprises:

identifying a phrase similarity threshold; and

determining that the phrase similarity score exceeds the phrase similarity threshold.

6 . The method of claim 1 , further comprising:

generating an additional face image based on an additional annotated image corresponding to an additional caption;

determining that the additional caption includes a predetermined word that indicates the presence of a face; and

adding the additional face image and the additional caption to the set of annotated face images based on the determination.

7 . The method of claim 1 , further comprising:

generating an additional face image based on an additional annotated image corresponding to an additional caption;

computing an additional phrase similarity score by comparing the additional face image to the phrase; and

adding the additional face image with the phrase to the set of annotated face images based on the additional phrase similarity score.

8 . The method of claim 7 , further comprising:

embedding the additional face image to obtain an additional image embedding,

wherein the additional phrase similarity score is computed based on the additional image embedding and the phrase embedding.

9 . The method of claim 7 , further comprising:

comparing the additional face image to the caption to obtain a caption similarity score; and

determining that the caption similarity score does not exceed a caption similarity threshold, wherein the extraction is based on the determination.

10 . The method of claim 1 , further comprising:

adding a pseudo-phrase to the phrase, where the phrase similarity score is computed based on the pseudo-phrase.

11 . An apparatus for face annotation, comprising:

a face detection network configured to generate a face image by cropping an annotated image based on a face detection algorithm;

a comparison network configured to compute a phrase similarity score by comparing an image embedding of the face image to a phrase embedding of a phrase of a plurality of phrases extracted from a caption corresponding to the annotated image in a multi-modal embedding space; and

a filtering component configured to add the face image with the phrase to a set of annotated face images based on the phrase similarity score.

12 . The apparatus of claim 11 , wherein:

the face detection network includes a convolutional neural network (CNN).

13 . The apparatus of claim 11 , wherein:

the comparison network includes a multi-modal encoder.

14 . The apparatus of claim 13 , wherein:

the multi-modal encoder is a Contrastive Language-Image Pre-training (CLIP) network.

15 . The apparatus of claim 11 , further comprising:

a training component configured to update a machine learning model using the set of annotated face images.

16 . A non-transitory computer readable medium storing code for face annotation, the code comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

receiving an annotated image and a caption corresponding to the annotated image, wherein the caption includes a plurality of phrases;

generating a face image by cropping the annotated image based on a face detection algorithm;

extracting a phrase of the plurality of phrases from the caption;

computing a phrase similarity score by comparing an image embedding of the face image to a phrase embedding of the phrase in a multi-modal embedding space; and

adding the face image with the phrase to a set of annotated face images based on the phrase similarity score.

17 . The non-transitory computer readable medium of claim 16 , wherein adding the face image with the phrase to the set of annotated face images comprises:

identifying a phrase similarity threshold; and

determining that the phrase similarity score exceeds the phrase similarity threshold.

18 . The non-transitory computer readable medium of claim 16 , the code further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

generating an additional face image based on an additional annotated image corresponding to an additional caption;

computing an additional phrase similarity score by comparing the additional face image to the phrase; and

adding the additional face image with the phrase to the set of annotated face images based on the additional phrase similarity score.

19 . The non-transitory computer readable medium of claim 18 , wherein comparing the additional face image to the phrase comprises:

embedding the additional face image to obtain an additional image embedding,

wherein the additional phrase similarity score is computed based on the additional image embedding and the phrase embedding.