Multi-lingual tagging for digital images
Introduced here is an approach to translating tags assigned to digital images. As an example, embeddings may be extracted from a tag to be translated and the digital image with which the tag is associated by a multimodal model. These embeddings can be compared to embeddings extracted from a set of target tags associated with a target language by the multimodal model. Such an approach allows similarity to be established along two dimensions, which ensures the obstacles associated with direct translation can be avoided.
1. A method comprising:
obtaining a source tag in a source language for an object located in a digital image;
determining that the source tag is to be translated from the source language into a target language;
extracting:
an image embedding from the digital image using an image encoder;
a source text embedding from the source tag in the source language using a multimodal model; and
a set of target text embeddings from a set of target tags in the target language using the multimodal model;
generating a first set of similarity scores representing measures of similarity between the digital image and each target tag in the set of target tags by comparing the image embedding with each of the target text embeddings in the set of target text embeddings;
generating a second set of similarity scores representing measures of similarity between the source tag and the set of target tags by comparing the source text embedding and each of the target text embeddings in the set of target text embeddings; and
selecting, from the set of target tags, an appropriate target tag as a translation of the source tag by combining the first set of similarity scores and the second set of similarity scores.
2. The method of claim 1 , wherein extracting the image embedding, the source text embedding, and the set of target text embeddings comprises generating vector representations of the image embedding, the source text embedding, and the set of target text embeddings in a multimodal cross lingual metric space.
3. The method of claim 1 ,
wherein extracting the image embedding comprises utilizing the image encoder to generate a real-valued feature vector from the digital image;
wherein extracting the source text embedding comprises utilizing the multimodal model to generate a real-valued feature vector from the source tag; and
wherein extracting the set of target text embeddings comprises utilizing the multimodal model to generate a real-valued feature vector from the set of target tags.
4. The method of claim 1 , wherein the set of target tags comprises a vocabulary of tags that are available in the target language.
5. The method of claim 1 , wherein selecting the appropriate target tag comprises:
calculating, for each target tag in the set of target tags, a weighted sum of a first similarity score from the first set of similarity scores and a second similarity score from the second set of similarity scores; and
selecting a target tag with a highest weighted sum as the appropriate target tag.
6. The method of claim 1 , further comprising:
causing display of at least a portion of the set of target tags on an interface accessible via a computing device, wherein the at least a portion of the set of target tags is presented in descending order of respective combinations of similarity scores from the first set of similarity scores and the second set of similarity scores.
7. The method of claim 1 , wherein selecting the appropriate target tag comprises:
determining a mapping between the source tag and a target tag in the set of target tags by calculating, for the target tag, a weighted sum of an identity function and similarity scores in the first set of similarity scores and the second set of similarity scores corresponding to the target tag; and
selecting the target tag as the appropriate target tag based on the mapping.
8. The method of claim 1 , wherein the object is one of multiple objects in the digital image, and wherein each of the multiple objects is associated with a different source tag to be translated into the target language.
9. The method of claim 1 , wherein selecting the appropriate target tag comprises:
determining a first target tag from the set of target tags identified as a translation for another source tag associated with another object within the digital image; and
based on identifying the first target tag as the translation for another source tag, selecting a second target tag from the set of target tags as the appropriate target tag.
10. The method of claim 1 , wherein obtaining the source tag comprises:
applying a tagging model to the digital image that generates the source tag from the object located in the digital image.
11. A non-transitory computer readable medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising:
obtaining a source tag in a source language for an object located in a digital image;
determining that the source tag is to be translated from the source language into a target language;
extracting:
an image embedding from the digital image using an image encoder;
a source text embedding from the source tag in the source language using a multimodal model; and
a set of target text embeddings from a set of target tags in the target language using the multimodal model;
generating a first set of similarity scores representing measures of similarity between the digital image and each target tag in the set of target tags by comparing the image embedding with each of the target text embeddings in the set of target text embeddings;
generating a second set of similarity scores representing measures of similarity between the source tag and the set of target tags by comparing the source text embedding and each of the target text embeddings in the set of target text embeddings; and
selecting, from the set of target tags, an appropriate target tag as a translation of the source tag by combining the first set of similarity scores and the second set of similarity scores.
12. The non-transitory computer readable medium of claim 11 , wherein extracting the image embedding, the source text embedding, and the set of target text embeddings comprises generating vector representations of the image embedding, the source text embedding, and the set of target text embeddings in a multimodal cross lingual metric space.
13. The non-transitory computer readable medium of claim 11 , wherein:
extracting the image embedding comprises utilizing the image encoder to generate a real-valued feature vector from the digital image;
extracting the source text embedding comprises utilizing the multimodal model to generate a real-valued feature vector from the source tag; and
extracting the set of target text embeddings comprises utilizing the multimodal model to generate a real-valued feature vector from the set of target tags.
14. The non-transitory computer readable medium of claim 11 , wherein the set of target tags comprises a vocabulary of tags that are available in the target language.
15. The non-transitory computer readable medium of claim 11 , wherein selecting the appropriate target tag comprises:
calculating, for each target tag in the set of target tags, a weighted sum of a first similarity score from the first set of similarity scores and a second similarity score from the second set of similarity scores; and
selecting a target tag with a highest weighted sum as the appropriate target tag.
16. A system comprising:
one or more memory devices comprising a digital image and a set of target tags in a target language; and
one or more processors configured to cause the system to:
obtain a source tag in a source language for an object located in the digital image;
determine that the source tag is to be translated from the source language into the target language;
extract:
an image embedding from the digital image using an image encoder;
a source text embedding from the source tag in the source language using a multimodal model; and
a set of target text embeddings from a set of target tags in the target language using the multimodal model;
generate a first set of similarity scores representing measures of similarity between the digital image and each target tag in the set of target tags by comparing the image embedding with each of the target text embeddings in the set of target text embeddings;
generate a second set of similarity scores representing measures of similarity between the source tag and the set of target tags by comparing the source text embedding and each of the target text embeddings in the set of target text embeddings; and
select, from the set of target tags, an appropriate target tag as a translation of the source tag by combining the first set of similarity scores and the second set of similarity scores.
17. The system of claim 16 , wherein the one or more processors are further configured to select the appropriate target tag by:
determining a mapping between the source tag and a target tag in the set of target tags by calculating, for the target tag, a weighted sum of an identity function and similarity scores in the first set of similarity scores and the second set of similarity scores corresponding to the target tag; and
selecting the target tag as the appropriate target tag based on the mapping.
18. The system of claim 16 , wherein the object is one of multiple objects in the digital image, and wherein each of the multiple objects is associated with a different source tag to be translated into the target language.
19. The system of claim 16 , wherein the one or more processors are further configured to select the appropriate target tag by:
determining a first target tag from the set of target tags identified as a translation for another source tag associated with another object within the digital image; and
based on identifying the first target tag as the translation for another source tag, selecting a second target tag from the set of target tags as the appropriate target tag.
20. The system of claim 16 , wherein the one or more processors are further configured to obtain the source tag by:
applying a tagging model to the digital image that generates the source tag from the object located in the digital image.