Domain specific natural language normalization
Embodiments of the present invention provide a method, system and computer program product for the domain specific normalization of a corpus of text. In an embodiment of the invention, a method for domain specific normalization of a corpus of text is provided, including an industrial, organization, demographic or geographic domain. The method includes loading a corpus of text in memory of a computer and determining a domain for the corpus of text. The method also includes retrieving a lexicon of replacement words for the determined domain. Finally, the method includes text simplifying the corpus of text using the retrieved lexicon. In one aspect of the embodiment, the domain is determined through inference based upon words already presence in the corpus of text. In another aspect of the embodiment, the domain is determined based upon meta-data provided with the corpus of text.
1. A method for domain specific normalization of a corpus of text, the method comprising:
loading a corpus of text in memory of a computer;
determining by a processor of the computer a domain for the corpus of text by recognizing a presence of words or phrases in the loaded corpus of text that had been previously correlated to the domain;
retrieving from the memory of the computer by the processor of the computer a lexicon of replacement words for the determined domain, the lexicon comprising a set of source terms, at least one of the source terms being mapped to one of multiple different replacement terms having a complexity value aligned with an average complexity value for the multiple different replacement terms; and,
text simplifying by the processor of the computer the corpus of text using the retrieved lexicon by replacing existing words in the corpus of text with the replacement words.
2. The method of claim 1 , wherein the domain is an industrial domain.
3. The method of claim 1 , wherein the domain is an organizational domain.
4. The method of claim 1 , wherein the domain is a demographic domain.
5. The method of claim 1 , wherein the domain is a geographic domain.
6. The method of claim 1 , wherein the domain is determined through inference based upon words already present in the corpus of text.
7. The method of claim 1 , wherein the domain is determined based upon meta-data provided with the corpus of text.
8. A natural language data processing system configured for domain specific normalization of a corpus of text, the system comprising:
a host computing system comprising at least one computer with memory and at least one processor;
a natural language processor providing logic configured for text simplification executing in the memory of the computer; and,
a domain specific normalization module of the natural language processor comprising program code executing in the host computing system enabled to load a corpus of text, to determine a domain for the corpus of text by recognizing a presence of words or phrases in the loaded corpus of text that had been previously correlated to the domain, to retrieve a lexicon of replacement words for the determined domain, the lexicon comprising a set of source terms, at least one of the source terms being mapped to one of multiple different replacement terms having a complexity value aligned with an average complexity value for the multiple different replacement terms, and to direct the natural language processor to text simplify the corpus of text using the retrieved lexicon by replacing existing words in the corpus of text with the replacement words.
9. The system of claim 8 , wherein the domain is an industrial domain.
10. The system of claim 8 , wherein the domain is an organizational domain.
11. The system of claim 8 , wherein the domain is a demographic domain.
12. The system of claim 8 , wherein the domain is a geographic domain.
13. The system of claim 8 , wherein the program code of the module determines the domain through inference based upon words already present in the corpus of text.
14. A computer program product for domain specific normalization of a corpus of text, the computer program product comprising:
a non-transitory computer readable storage medium comprising a device having computer readable program code embodied therewith, the computer readable program code comprising:
computer readable program code for loading a corpus of text in memory of a computer;
computer readable program code for determining a domain for the corpus of text by recognizing a presence of words or phrases in the loaded corpus of text that had been previously correlated to the domain;
computer readable program code for retrieving a lexicon of replacement words for the determined domain, the lexicon comprising a set of source terms, at least one of the source terms being mapped to one of multiple different replacement terms having a complexity value aligned with an average complexity value for the multiple different replacement terms; and,
computer readable program code for text simplifying the corpus of text using the retrieved lexicon by replacing existing words in the corpus of text with the replacement words.
15. The computer program product of claim 14 , wherein the domain is an industrial domain.
16. The computer program product of claim 14 , wherein the domain is an organizational domain.
17. The computer program product of claim 14 , wherein the domain is a demographic domain.
18. The computer program product of claim 14 , wherein the domain is a geographic domain.
19. The computer program product of claim 14 , wherein the domain is determined through inference based upon words already present in the corpus of text.
20. The computer program product of claim 14 , wherein the domain is determined based upon meta-data provided with the corpus of text.