Language identification in multilingual text
Methods, systems, and media are provided for identifying languages in multilingual text. A document is decoded into a universal representative coding for easier tag manipulation, then broken into plain-text content sections. The sections are identified and assigned a weight, wherein more informative sections are given a higher weight and less informative sections are given a lesser weight. A language likelihood score is determined for each word, phrase, or character n-gram in a section. The language likelihood scores within a section are combined for each language. The combined section scores are then summed together to obtain a total document score for each language. This results in a document score for each language, which can be ranked to determine the primary language for the document.
1. A computer-implemented system for identifying multilingual text in a document using computer processor, memory, and data storage subsystems, the computer-implemented system comprising:
a code-page conversion component to identify the character encoding used by a document and to decode said encoding into a universal representative encoding via the processor;
a section breaking and classification component to divide plain-text content of the document into one or more weighted sections, the sections are weighed according to their relative importance in conveying information regarding a content of the document and comprise logical areas of a layout of the document including one or more of title, heading, article text, header, and footer;
a language scoring component to calculate a document score for each language present in the document by:
(1) obtaining a language likelihood scores of each word, phrase, or character n-gram in the one or more weighted sections, to
(2) generating a sum of the obtained language likelihood scores within the one or more weighted sections for each language present,
(3) for each section, generating a summation score for each language evaluated by multiplying each language's likelihood score sum by a weight associated with the section, and
(4) calculating the document score by summing the summation scores within the document; and
an output language selection component to select a primary language for the document, wherein the primary language has the highest document score.
2. The computer-implemented system of claim 1 , wherein the universal representative coding comprises Unicode.
3. The computer-implemented system of claim 1 , wherein the one or more weighted sections comprise a title with a higher weight than a footer.
4. The computer-implemented system of claim 1 , wherein the plain-text content is parsed into sections based upon HTML tags, visual layout, structure, and semantic content of the document.
5. The computer-implemented system of claim 1 , wherein the language likelihood score comprises a likelihood of each word, phrase, or character n-gram belonging to one or more languages.
6. The computer-implemented system of claim 1 , wherein the output language selection component ranks results of the combined language likelihood scores for each language.
7. A computer-implemented method of identifying multilingual text in a document using a computing system having processor, memory, and data storage subsystems, the computer-implemented method comprising:
isolating one or more regions of plain-text content in a document;
disjoining the plain-text content into sections according to semantic and syntactic categories, the sections comprise logical areas of a layout of the document including one or more of title, heading, article text, header, and footer;
assigning a weight to each of the sections according to relative importance in conveying information regarding a content of the document;
calculating a language likelihood score for each word, phrase, or character n-gram in each of the sections;
computing a combined language likelihood score for each of the sections for each language by summing the language likelihood scores for each word, phrase, or character n-gram in each of the sections and multiplying by the weight assigned to the section;
computing a document score for each language by summing the language likelihood scores of the document for each language; and
outputting a language having a highest ranked document score as a primary language of the document.
8. The computer-implemented method of claim 7 , further comprising:
identifying an encoding used with the document; and
decoding into a universal representative code.
9. The computer-implemented method of claim 7 , wherein said calculating comprises:
calculating the language likelihood scores for each word, phrase, or character n-gram within a section multiplied by a weight of the associated section.
10. The computer-implemented method of claim 7 , further comprising: dividing each of the sections of plain-text content into segments according to a writing script used.
11. The computer-implemented method of claim 10 , wherein the assigning comprises: assigning a weight to each of the segments.
12. One or more computer-readable storage device containing computer-readable instructions embodied thereon that, when executed by a computing device, perform a method of selecting a primary language of a multilingual document, the method comprising:
dividing plain-text content of a document into one or more weighted sections, the sections are weighed according to their relative importance in conveying information regarding a content of the document and comprise logical areas of a layout of the document including one or more of title, heading, article text, header, and footer;
determining a likelihood score for each word, phrase, or character n-gram belonging to one or more languages for each of the weighted sections;
summing all of the likelihood scores from each word, phrase, or character n-gram in each of the sections and multiplying a sum by the weight associated with the section for each individual language to obtain one or more section language summations;
combining all of the section language summations for each individual language to obtain a document score for each individual language;
ranking all of the document scores; and
selecting a primary document language from the highest document score.
13. The device of claim 12 , further comprising:
converting an encoding of the document into a universal representative coding.
14. The device of claim 12 , wherein the dividing is implemented using HTML tags.
15. The device of claim 12 , wherein the likelihood scores for each word, phrase, or character n-gram is obtained from a dictionary via a word-breaker.
16. The device of claim 12 , further comprising:
dividing each of the one or more weighted sections into one or more weighted language sections.
17. The device of claim 12 , further comprising:
selecting additional languages that cover alternative alphabets or scripts that are not already covered by the primary language output.
18. The device of claim 12 , wherein the one or more weighted sections are based upon importance of each section and a popularity of each language.