System and method for text normalization in noisy channels

1. A method for text normalization in a plurality of noisy channels, performed on a computing device having a processor, memory, and one or more code sets stored in the memory and executing in the processor, the method comprising:

generating a set of models for a respective set of communication channels;

generating, based on the set of models, a respective set of channel specific error-type adapter frameworks, wherein each of the channel-specific error-type adapter frameworks is optimized for a respective communication channel;

receiving, by the processor, a text entry and channel origin data of the text entry;

determining, by the processor, whether the text entry matches an in-vocabulary (IV) entry or whether the text entry is an out-of-vocabulary (OOV) entry;

if the text entry is determined to have a matching IV entry:

outputting, by the processor, the matching IV entry; and

if the text entry is determined to be an OOV entry:

implementing, by the processor, the relevant channel-specific error-type adapter framework based on the channel origin data;

wherein the channel-specific error-type adapter framework is optimized for the specific channel from which the text entry originated; and

wherein the channel-specific error-type adapter framework further comprises a probabilistic model of one or more error-type priors in the specific channel;

normalizing, by the processor, the text entry using the channel-specific error-type adapter framework;

wherein the channel-specific error-type adapter framework comprises one or more error-type adapters;

wherein the one or more error-type adapters comprise at least one of a spelling error-type adapter, an abbreviation error-type adapter, an acronym error-type adapter, a phonetic shorthand error-type adapter, and a word concatenation error-type adapter; and

wherein each error-type adaptor is configured to model a different type of error to be normalized; and

outputting one or more candidate normalized forms of the text entry.

2. The method of claim 1 , comprising an initial step of building an interpolated language model and one or more static lexicons to be implemented when normalizing the text entry using the channel-specific error-type adapter framework.

3. The method of claim 2 , wherein the interpolated language model comprises a merged combination of a basic language model and one or more channel-specific language models.

4. The method of claim 2 , wherein the one or more static lexicons comprise at least one of a proper name lexicon, an abbreviation lexicon, and an acronym lexicon.

5. The method of claim 1 , wherein the spelling error-type adapter is generated using one or more matrices, each matrix computing a probability of a specific edit operation being performed to identify a spelling error correction.

6. The method of claim 1 , further comprising applying linguistic heuristics to expand the channel-specific error-type adapter framework.

7. A system for text normalization in a plurality of noisy channels, comprising:

a processor;

a memory; and

one or more code sets stored in the memory and executing in the processor, which, when executed, configure the processor to:

generate a set of models for a respective set of communication channels;

generate, based on the set of models, a respective set of channel specific error-type adapter frameworks, wherein each of the channel-specific error-type adapter frameworks is optimized for a respective communication channel;

receive a text entry and channel origin data of the text entry;

determine whether the text entry matches an in-vocabulary (IV) entry or whether the text entry is an out-of-vocabulary (OOV) entry;

if the text entry is determined to have a matching IV entry:

output the matching IV entry; and

if the text entry is determined to be an OOV entry:

implement the relevant channel-specific error-type adapter framework based on the channel origin data;

wherein the channel-specific error-type adapter framework is optimized for the specific channel from which the text entry originated; and

wherein the channel-specific error-type adapter framework further comprises a probabilistic model of one or more error-type priors in the specific channel;

normalize the text entry using the channel-specific error-type adapter framework;

wherein the channel-specific error-type adapter framework comprises one or more error-type adapters;

wherein the one or more error-type adapters comprise at least one of a spelling error-type adapter, an abbreviation error-type adapter, an acronym error-type adapter, a phonetic shorthand error-type adapter, and a word concatenation error-type adapter; and

wherein each error-type adaptor is configured to model a different type of error to be normalized; and

output one or more candidate normalized forms of the text entry.

8. The system of claim 7 , wherein the one or more code sets further configure the processor to build an interpolated language model and one or more static lexicons to be implemented when normalizing the text entry using the channel-specific error-type adapter framework.

9. The system of claim 8 , wherein the interpolated language model comprises a merged combination of a basic language model and one or more channel-specific language models.

10. The system of claim 8 , wherein the one or more static lexicons comprise at least one of a proper name lexicon, an abbreviation lexicon, and an acronym lexicon.

11. The system of claim 7 , wherein the one or more code sets configure the processor to generate the spelling error-type adapter using one or more matrices, each matrix computing a probability of a specific edit operation being performed to identify a spelling error correction.

12. The system of claim 7 , further comprising applying linguistic heuristics to expand the channel-specific error-type adapter framework.

13. A method for text normalization in a plurality of noisy channels, performed on a computing device having a processor, memory, and one or more code sets stored in the memory and executing in the processor, the method comprising:

generating a set of models for a respective set of communication channels;

generating, based on the set of models, a respective set of channel specific error-type adapter frameworks, wherein each of the channel-specific error-type adapter frameworks is optimized for a respective communication channel;

receiving, by the processor, a text entry and channel origin data of the text entry;

determining, by the processor, whether the text entry matches an in-vocabulary (IV) entry or whether the text entry is an out-of-vocabulary (OOV) entry;

wherein the matching IV entry is outputted when the text entry is determined to have a matching IV entry,

wherein a relevant channel-specific error-type adapter framework is implemented based on the channel origin data when the text entry is determined to be an OOV entry,

wherein the channel-specific error-type adapter framework is optimized for the specific channel from which the text entry originated; and

wherein the channel-specific error-type adapter framework further comprises a probabilistic model of one or more error-type priors in the specific channel;

normalizing, by the processor, the text entry using the channel-specific error-type adapter framework;

wherein the channel-specific error-type adapter framework comprises one or more error-type adapters;

wherein the one or more error-type adapters comprise at least one of a spelling error-type adapter, an abbreviation error-type adapter, an acronym error-type adapter, a phonetic shorthand error-type adapter, and a word concatenation error-type adapter; and

wherein each error-type adaptor is configured to model a different type of error to be normalized; and

outputting one or more candidate normalized forms of the text entry.