Method, apparatus and computer program product for providing flexible text based language identification

1. A method comprising:

receiving an entry in a computer readable text format;

determining an alphabet score of the entry for each of a plurality of languages;

determining an n-gram frequency score of the entry for each of the plurality of languages; and

determining, via a processor, a language associated with the entry based on a composite score comprising a combination of the alphabet score and the n-gram frequency score.

2. A method according to claim 1 , wherein determining the alphabet score comprises comparing characters associated with the entry to an alphabet of each of the plurality of languages and generating a score for each of the plurality of languages, the score for each of the plurality of languages being based at least in part on a failure of one or more of the characters to be represented in the corresponding alphabet of a respective one of the plurality of languages being scored.

3. A method according to claim 1 , wherein determining the n-gram frequency score comprises, for each one of the plurality of languages, comparing the entry to n-gram statistics for the one of the plurality of languages.

4. A method according to claim 3 , wherein the entry comprises n-characters and wherein comparing the entry to n-gram statistics comprises determining a conditional probability of the nth character of the entry given the preceding n-1 characters.

5. A method according to claim 3 , further comprising assigning a start character and a stop character to a first character and a last character of the entry, respectively, for use in correlation with corresponding start characters and stop characters associated with each n-gram probability of the n-gram statistics.

6. A method according to claim 1 , further comprising comparing the alphabet score for each of the plurality of languages to a threshold.

7. A method according to claim 6 , wherein determining the n-gram frequency score is only performed for ones of the plurality of languages for which the alphabet score meets the threshold.

8. A method according to claim 6 , further comprising assigning the entry a predetermined alphabet score in response to the alphabet score failing to meet the threshold.

9. A method according to claim 1 , wherein determining the language associated with the entry comprises selecting the language associated with the entry based on a predefined order of languages in response to the combination of the alphabet score and the n-gram frequency score providing an indeterminate result with respect to the language associated with the entry.

10. A method according to claim 1 , wherein the entry is one of a plurality of segments comprising an input text and wherein the determining the language further comprises determining the language associated with the input text based on a cumulative combination of alphabet scores and n-gram frequency scores of each one of the plurality of segments.

11. A computer program product comprising at least one computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for receiving an entry in a computer readable text format;

a second executable portion for determining an alphabet score of the entry for each of a plurality of languages;

a third executable portion for determining an n-gram frequency score of the entry for each of the plurality of languages; and

a fourth executable portion for determining a language associated with the entry based on a composite score comprising a combination of the alphabet score and the n-gram frequency score.

12. A computer program product according to claim 11 , wherein the second executable portion includes instructions for comparing characters associated with the entry to an alphabet of each of the plurality of languages and generating a score for each of the plurality of languages, the score for each of the plurality of languages being based at least in part on a failure of one or more of the characters to be represented in the corresponding alphabet of a respective one of the plurality of languages being scored.

13. A computer program product according to claim 11 , wherein the third executable portion includes instructions for comparing the entry to n-gram statistics for the one of the plurality of languages for each one of the plurality of languages.

14. A computer program product according to claim 13 , wherein the entry comprises n-characters and wherein the third executable portion further includes instructions for determining a conditional probability of the nth character of the entry given the preceding n-1 characters.

15. A computer program product according to claim 13 , further comprising a fifth executable portion for assigning a start character and a stop character to a first character and a last character of the entry, respectively, for use in correlation with corresponding start characters and stop characters associated with each n-gram probability of the n-gram statistics.

16. A computer program product according to claim 11 , further comprising a fifth executable portion for comparing the alphabet score for each of the plurality of languages to a threshold.

17. A computer program product according to claim 16 , wherein the third executable portion is only executed for ones of the plurality of languages for which the alphabet score meets the threshold.

18. A computer program product according to claim 16 , further comprising a sixth executable portion for assigning the entry a predetermined alphabet score in response to the alphabet score failing to meet the threshold.

19. A computer program product according to claim 11 , wherein the fourth executable portion includes instructions for selecting the language associated with the entry based on a predefined order of languages in response to the combination of the alphabet score and the n-gram frequency score providing an indeterminate result with respect to the language associated with the entry.

20. A computer program product according to claim 11 , wherein the entry is one of a plurality of segments comprising an input text and wherein the fourth executable portion includes instructions for determining the language associated with the input text based on a cumulative combination of alphabet scores and n-gram frequency scores of each one of the plurality of segments.

21. An apparatus comprising a processor configured to:

receive an entry in a computer readable text format and calculate an alphabet score of the entry for each of a plurality of languages;

calculate an n-gram frequency score of the entry for each of the plurality of languages; and

determine a language associated with the entry based on a composite score comprising a combination of the alphabet score and the n-gram frequency score.

22. An apparatus according to claim 21 , wherein the processor is further configured to compare characters associated with the entry to an alphabet of each of the plurality of languages and generate a score for each of the plurality of languages, the score for each of the plurality of languages being based at least in part on a failure of one or more of the characters to be represented in the corresponding alphabet of a respective one of the plurality of languages being scored.

23. An apparatus according to claim 21 , wherein the processor is further configured to, for each one of the plurality of languages, compare the entry to n-gram statistics for the one of the plurality of languages.

24. An apparatus according to claim 23 , wherein the entry comprises n-characters and wherein the processor is further configured to determine a conditional probability of the nth character of the entry given the preceding n-1 characters.

25. An apparatus according to claim 23 , wherein the processor is further configured to assign a start character and a stop character to a first character and a last character of the entry, respectively, for use in correlation with corresponding start characters and stop characters associated with each n-gram probability of the n-gram statistics.

26. An apparatus according to claim 21 , the processor is further configured to compare the alphabet score for each of the plurality of languages to a threshold.

27. An apparatus according to claim 26 , wherein the processor is further configured to calculate the n-gram frequency score only for ones of the plurality of languages for which the alphabet score meets the threshold.

28. An apparatus according to claim 26 , wherein the processor is further configured to assign the entry a predetermined alphabet score in response to the alphabet score failing to meet the threshold.

29. An apparatus according to claim 21 , further comprising a package ordering element configured to select the language associated with the entry based on a predefined order of languages in response to the combination of the alphabet score and the n-gram frequency score providing an indeterminate result with respect to the language associated with the entry.

30. An apparatus according to claim 21 , wherein the entry is one of a plurality of segments comprising an input text and wherein the processor is further configured to determine the language associated with the input text based on a cumulative combination of alphabet scores and n-gram frequency scores of each one of the plurality of segments.

31. An apparatus according to claim 21 , wherein the apparatus is embodied as a mobile terminal.

32. An apparatus comprising:

means for receiving an entry in a computer readable text format;

means for determining an alphabet score of the entry for each of a plurality of languages;

means for determining an n-gram frequency score of the entry for each of the plurality of languages; and

means for determining a language associated with the entry based on a composite score comprising a combination of the alphabet score and the n-gram frequency score.

33. An apparatus according to claim 32 , further comprising means for selecting the language associated with the entry based on a predefined order of languages in response to the combination of the alphabet score and the n-gram frequency score providing an indeterminate result with respect to the language associated with the entry.