Text language identification
After prestoring first character strings that occur frequently in words of languages and second character strings that are a typical therein, a device for automatically identifying the language of a text from a plurality of languages extracts words from the text and constructs all of the character strings contained in each extracted word. Each string in an extracted word is compared to the first and second strings of a particular language. If the word contains a first string, a score of the language is increased by a coefficient depending in particular on the position of the first string in the word. If the word contains a second string, the score is decreased by a coefficient associated with the second string. The highest of the scores corresponding to the predetermined languages identifies the language of the text.
1. A device for automatically identifying the language of a digital text, comprising:
means for prestoring first character strings, including prefixes, suffixes and infixes, of different lengths from words of a plurality of predetermined languages, that occur frequently anywhere respectively in said words of said plurality of predetermined languages,
means for prestoring second character strings of different lengths, that are A typical anywhere respectively in said words of said predetermined languages,
means for analyzing words extracted from said digital text, thereby constructing for each extracted word all the character strings contained in said extracted word, including all the prefixes, suffixes and infixes in said extracted word, with overlap and different lengths lying between one character and the number of characters in said extracted word,
means for comparing each of said character strings contained in each said extracted word to said first prestored character strings and second prestored character strings of said predetermined languages,
means for calculating scores respectively associated with said predetermined languages, a score associated with one determined language being calculated by adding to said score a first coefficient whenever a prestored first character string of said one determined language is found in said extracted word, said first coefficient depending on the position of said found prestored first character string of said one determined language in said extracted word, and, by subtracting from said score a second coefficient whenever a prestored second character string of said one determined language is found in said extracted word, said second coefficient increasing as the probability of said found prestored second character string in said one determined language decreases, and
means for comparing said scores for said text associated with said predetermined languages in order to determine the highest of said scores, which identities the language of said text.
2. The device claimed in claim 1 , wherein said first coefficient of a first character string in said extracted word depends on the frequency of said character string in said determined language.
3. The device claimed in claim 1 , wherein said first coefficient of a first character string in said extracted word depends on the length of said character string.
4. The device claimed in claim 1 , wherein said first coefficient of a first character string in said extracted word is equal to:
PO (FR+LON),
where
PO is a coefficient depending on the position of said first character string in said extracted word,
FR is a coefficient depending on the frequency of said first character string in a determined language, and LON is a coefficient depending on the length of said first character string.
5. The device claimed in claim 1 , comprising comparator means for comparing each of said extracted words from said text with frequent words in said determined language and initially listed in storage means so that whenever a frequent word is found in said text, said score for said determined language is increased only by a coefficient depending on the frequency of said extracted word in said determined language.
6. The device claimed in claim 1 , comprising comparator means for comparing each of said extracted words from said text with frequent words in said determined language and initially listed in storage means so that whenever a frequent word is found in said text, said score for said determined language is increased only by a coefficient depending on the length of said frequent word.
7. A method of automatically identifying the language of a digital text, the method being performed with a computer arrangement including a storing arrangement and a processor arrangement,
the storing arrangement prestoring (a) first character strings, including prefixes, suffixes and infixes, of different lengths from words of a plurality of predetermined languages, that occur frequently anywhere respectively in said words of said plurality of predetermined languages, and (b) second character strings of different lengths, that are a typical anywhere respectively in said words of said predetermined languages,
the method comprising:
in the processor arrangement: (a) analyzing words extracted from said digital text, thereby constructing for each extracted word all the character strings contained in said extracted word, including all the prefixes, suffixes and infixes in said extracted word, with overlap and different lengths lying between one character and the number of characters in said extracted word, (b) comparing each of said character strings contained in each said extracted word with said first prestored character strings and second prestored character strings of said predetermined languages, (c) calculating scores respectively associated with said predetermined languages, the calculation of a score associated with one determined language being performed (i) by adding to said score a first coefficient whenever a prestored first character string of said one determined language is found in said extracted word, said first coefficient depending on the position of said found prestored first character string of said one determined language in said extracted word, and (ii) subtracting from said score a second coefficient whenever a prestored second character string of said one determined language is found in said extracted word, said second coefficient thereby increasing as the probability of said found prestored second character string in said one determined language decreases, and (d) comparing said scores for said text associated with said predetermined languages in order to determine the highest of said scores, the highest of the scores identifying the language of said text.
8. A device for automatically identifying the language of a digital text, comprising:
means for prestoring first character strings that occur frequently anywhere respectively in words of a plurality of predetermined languages and characterize said predetermined languages,
means for prestoring second character strings that are a typical anywhere respectively in words of said predetermined languages
means for analyzing words extracted from said digital text, thereby constructing for each extracted word all character strings contained in said extracted word and having lengths lying between one character and the number of characters in said extracted word,
means for comparing character strings contained in extracted words to prestored character strings in order to determine scores associated with said predetermined languages,
means for individually Comparing each of all character strings contained in each said extracted word to said first prestored character strings and said second prestored character strings of each determined language so that whenever a prestored first character string is found in said extracted word, a score associated with said each determined language is increased by a first coefficient depending on the position of said first character string found in said extracted word, and, whenever a prestored second character string is found in said extracted word, said score is decreased by a respective second coefficient that is associated with said found second character string, said respective second coefficient increasing as the probability of said found second character string in said each determined language decreases, and
means for comparing said scores for said text associated with said predetermined languages in order to determine the highest of said scores, which identifies the language of said text, wherein said first coefficient of a first character string in said extracted word is equal to:
PO (FR+LON),
where
PO is a coefficient depending on the position of said first character string in said extracted word,
FR is a coefficient depending on the frequency of said first character string in a determined language, and
LON is a coefficient depending on the length of said first character string.
9. The method of claim 7 , wherein said first coefficient of a first character string in said extracted word is equal to:
PO (FR+LON),
where
PO is a coefficient depending on the position of said first character string in said extracted word,
FR Is a coefficient depending on the frequency of said first character string in a determined language, and
LON is a coefficient depending on the length of said first character string.