Context aware back-transliteration and translation of names and common phrases using web resources
Described within are systems and methods for transliterating and translating source non-Romanized language text strings from a plurality of electronic sources to Romanized target language text strings by converting the source non-Romanized language text strings to a standard document encoding format, splitting the source non-Romanized language text strings into smaller units, transforming the smaller units into entity profiles, processing the entities profiles with data from external databases, translating the entities in the entity profiles into a Romanized target language, and outputting the entities into a plurality of data formats for external systems.
1. A system for transforming content of an electronic document from a non-Romanized native language to a Romanized language, the system comprising one or more processors and memory to store program code that, when executed by the one or more processors, causes the one or more processors to:
identify within the native language document one or more references, in the native language, to entities having Romanized language names;
generate a preprocessed version of the native language document in which the identified references to entities having Romanized language names are replaced by the Romanized language names and corresponding tags that specify the Romanized language names as named entities; and
translate the preprocessed version of the native language document into a Romanized-language output document, maintaining the Romanized language names in the output document and applying the Romanized language names and corresponding tags as context for disambiguating remaining native language expressions in the preprocessed version of the native language document.
2. The system of claim 1 , wherein the program code that, when executed by the one or more processors, causes the one or more processors to identify the one or more references to entities having Romanized language names comprises program code that, when executed, causes the one or more processors to transliterate the native language document into a Romanized language output.
3. The system of claim 2 , wherein the program code that, when executed by the one or more processors, causes the one or more processors to identify the one or more references to entities having Romanized language names comprises program code that, when executed, causes the one or more processors to compare contents of the Romanized language output with Romanized language names within one or more databases.
4. The system of claim 3 , wherein the one or more databases are selected from a group consisting of websites, internet databases, dictionaries, lexicons and native language classification databases and combinations thereof.
5. The system of claim 1 , wherein the program code that, when executed by the one or more processors, causes the one or more processors to identify the one or more references to entities having Romanized language names comprises program code that, when executed, causes the one or more processors to split contents of the native language document into segments.
6. The system of claim 1 , wherein the native language of the native language document falls within a group consisting of Arabic, Armenian, Belarusian, Bengali, Bulgarian, Chinese, Croatian, Czech, Devanagari, Divehi, Farsi, Georgian, Greek, Gujarati, Hebrew, Hindi, Japanese, Kannada, Kazakh, Korean, Kyrgyz, Latvian, Lithuanian, Macedonian, Malayalam, Marathi, Mongolian, Pashtun, Punjabi, Romanian, Russian, Serbian, Syriac, Tamil, Telugu, That, Turkish, Ukrainian, Urdu, Uzbek, Vietnamese, other non-Romanized languages and combinations thereof.
7. The system of claim 1 , wherein the program code that, when executed by the one or more processors, causes the one or more processors to identify the one or more references to entities having Romanized language names comprises program code that, when executed, causes the one or more processors to generate a confidence score for each of the one or more references based on at least one of a probability that letters in the native language document are equivalent to letters in the Romanized language or a probability that a transliterated string within the native language document corresponds to a named entity in the Romanized language.
8. The system of claim 1 wherein the tags that specify the Romanized language names as named entities include information indicating an entity type.
9. The system of claim 8 wherein the information indicating the entity type specifies, as the entity type, at least one of a person, place, organization, product name, event, or meme.
10. A method, executed within one or more computer processors, of transforming content of an electronic document from a non-Romanized native language to a Romanized language the method comprising:
identifying within the native language document one or more references, in the native language, to entities having Romanized language names;
generating a preprocessed version of the native language document in which the identified references to entities having Romanized language names are replaced by the Romanized language names and corresponding tags that specify the Romanized language names as named entities; and
translating the preprocessed version of the native language document into Romanized-language output document, maintaining the Romanized language names in the output document and applying the Romanized language names and corresponding tags as context for disambiguating remaining native language expressions in the preprocessed version of the native language document.
11. The method of claim 10 , wherein identifying the one or more references to entities having Romanized language names comprises transliterating the native language document into a Romanized language output.
12. The method of claim 11 , wherein identifying the one or more references to entities having Romanized language names further comprises comparing contents of the Romanized language output with Romanized language names within one or more databases.
13. The method of claim 12 , wherein the one or more databases comprise external databases selected from a group consisting of websites, internet databases, dictionaries, lexicons and native language classification databases and combinations thereof.
14. The method of claim 10 , wherein identifying one or more references to entities having Romanized language names within the native language document comprises splitting contents of the native language document into segments.
15. The method of claim 10 , wherein identifying one or more references to entities having Romanized language names within the native language document comprises generating a confidence score for each of the one or more references based on at least one of a probability that letters in the native language document are equivalent to letters in the Romanized language or probability that a transliterated string within the native language document corresponds to a named entity in the Romanized language.
16. The method of claim 10 , wherein the native language of the native language document falls within a group consisting of Arabic, Armenian, Belarusian, Bengali, Bulgarian, Chinese, Croatian, Czech, Devanagari, Divehi, Farsi, Georgian, Greek, Gujarati, Hebrew, Hindi, Japanese, Kannada, Kazakh, Korean, Kyrgyz, Latvian, Lithuanian, Macedonian, Malayalam, Marathi, Mongolian, Pashtun, Punjabi, Romanian, Russian, Serbian, Syriac, Tamil, Telugu, That, Turkish, Ukrainian, Urdu, Uzbek, Vietnamese, other non-Romanized languages and combinations thereof.
17. The method of claim 10 wherein the tags that specify the Romanized language names as named entities include information indicating an entity type.
18. The method of claim 17 wherein the information indicating the entity type specifies, as the entity type, at least one of a person, place, organization, product name, event, or meme.
19. A non-transitory computer-readable medium having one or more sequences of instructions embodied therein which, when executed by one or more processors, cause the one or more processors to:
identify within the native language document one or more references, in the native language, to entities having Romanized language names;
generate a preprocessed version of the native language document in which the identified references to entities having Romanized language names are replaced by the Romanized language names and corresponding tags that specify the Romanized language names as named entities; and
translate the preprocessed version of the native language document into Romanized-language output document, maintaining the Romanized language names in the output document and applying the Romanized language names and corresponding tags as context for disambiguating remaining native language expressions in the preprocessed version of the native language document.