Language-agnostic OCR extraction
Technologies for language agnostic OCR extraction include identifying a word region of an image using optical character recognition, applying a language agnostic machine learning model to the word region, where the language agnostic machine learning model is trained on training data including a set of image-text pairs and a set of multilingual text translation pairs, receiving, from the language agnostic machine learning model, a word region embedding that is associated with the word region, searching a multilingual index for a text embedding that matches the word region embedding, receiving, from the multilingual index, text associated with the text embedding; and outputting at least one of the text or the text embedding to at least one downstream process, application, system, component, or network.
1. A method, comprising:
identifying a word region of an image using optical character recognition;
the word region comprises a set of bounding box coordinates;
applying a language agnostic machine learning model to the word region;
the language agnostic machine learning model is trained on training data comprising a set of image-text pairs and a set of multilingual text translation pairs;
receiving, from the language agnostic machine learning model, a word region embedding that is associated with the word region;
searching a multilingual index for a text embedding that matches the word region embedding;
receiving, from the multilingual index, text associated with the text embedding; and
using at least one of the text or the text embedding to determine whether or how to display the image via a user interface at a device.
2. The method of claim 1 , further comprising outputting the at least one of the text or the text embedding to at least one of:
a content classification model;
a content ranking model;
a data storage device; or
an output device.
3. The method of claim 2 , further comprising:
based on the text, generating and outputting a caption for the image.
4. The method of claim 1 , further comprising creating the multilingual index by:
applying the language agnostic machine learning model to a multilingual vocabulary;
the multilingual vocabulary comprises a plurality of different words in a plurality of different languages;
receiving, from the language agnostic machine learning model, a set of word embeddings;
the set of word embeddings comprises a word embedding for each word in the multilingual vocabulary; and
indexing the set of word embeddings.
5. The method of claim 4 , further comprising:
using a nearest neighbor algorithm to index the set of word embeddings.
6. The method of claim 4 , further comprising:
creating the multilingual vocabulary based on words extracted from a particular online system.
7. The method of claim 1 , wherein the language agnostic machine learning model embeds both natural language texts and images in a same latent space.
8. The method of claim 1 , wherein the language agnostic machine learning model comprises at least one of a multimodal representation model or a Turing Bletchley model.
9. The method of claim 1 , wherein the word region embedding comprises an image embedding, the image embedding is used to perform a search of the multilingual index, and the text embedding is returned by the search.
10. The method of claim 1 , wherein the word region embedding comprises an image embedding generated based on the image, and the method further comprises:
computing geo-relation data between the image embedding and a text embedding determined based on the image embedding; and
generating a caption for the image based on the geo-relation data.
11. A system comprising:
at least one memory; and
at least one processor coupled to the at least one memory;
wherein the at least one memory comprises instructions that, when executed by the at least one processor cause the at least one processor to perform operations comprising:
identifying a word region of an image using optical character recognition;
the word region comprises a set of bounding box coordinates;
applying a language agnostic machine learning model to the word region;
the language agnostic machine learning model is trained on training data comprising a set of image-text pairs and a set of multilingual text translation pairs;
receiving, from the language agnostic machine learning model, a word region embedding that is associated with the word region;
searching a multilingual index for a text embedding that matches the word region embedding;
receiving, from the multilingual index, text associated with the text embedding; and
using at least one of the text or the text embedding to determine whether or how to display the image via a user interface at a device.
12. The system of claim 11 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform operations further comprising:
outputting the at least one of the text or the text embedding to at least one of: a content classification model; a content ranking model; a data storage device; or an output device.
13. The system of claim 12 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform operations further comprising:
based on the text, generating and outputting a caption for the image.
14. The system of claim 11 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform operations further comprising creating the multilingual index by:
applying the language agnostic machine learning model to a multilingual vocabulary;
the multilingual vocabulary comprises a plurality of different words in a plurality of different languages;
receiving, from the language agnostic machine learning model, a set of word embeddings;
the set of word embeddings comprises a word embedding for each word in the multilingual vocabulary; and
indexing the set of word embeddings.
15. The system of claim 14 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform operations further comprising:
using a nearest neighbor algorithm to index the set of word embeddings.
16. The system of claim 14 , wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform operations further comprising:
creating the multilingual vocabulary based on words extracted from a particular online system.
17. The system of claim 11 , wherein the language agnostic machine learning model embeds both natural language texts and images in a same latent space.
18. The system of claim 11 , wherein the language agnostic machine learning model comprises at least one of a multimodal representation model or a Turing Bletchley model.
19. The system of claim 11 , wherein the word region embedding comprises an image embedding, the image embedding is used to perform a search of the multilingual index, and the text embedding is returned by the search.
20. The system of claim 11 , wherein the word region embedding comprises an image embedding generated based on the image, and the instructions, when executed by the at least one processor, cause the at least one processor to perform operations further comprising:
computing geo-relation data between the image embedding and a text embedding determined based on the image embedding; and
generating a caption for the image based on the geo-relation data.