Method of optimized parsing unstructured and garbled texts lacking whitespaces
A system, method, and computer-readable medium for performing a text parsing operation. The text parsing operation includes: receiving a corpus of text, at least a portion of the corpus of text comprising garbled text; parsing characters within the corpus of text to provide parsed characters from the corpus of text; parsing the parsed characters to generate recognized words from the parsed characters; generating semi-structured text from the recognized words; and, calculating a distribution of recognized words from the semi-structured text.
1. A computer-implementable method for performing a text parsing operation, comprising:
receiving a corpus of text, the corpus of text comprising a text string, at least a portion of the corpus of text comprising garbled text, the garbled text comprising a set of confused or unintelligible words, the set of confused or unintelligible words comprising at least one of two words concatenated into a single text string and a mistakenly capitalized character within the text string;
parsing characters within the corpus of text to provide parsed characters from the corpus of text, the parsing characters within the corpus of text comprising applying a text parsing window to a number of characters to accommodate a string of meaningful text, the parsing characters using the text parsing window to sequentially search characters within the corpus of text;
parsing the parsed characters to generate recognized words from the parsed characters, the parsing the parsed characters generating words from longer strings of text before words from shorter strings of text, the longer strings of text being identified using a longer text parsing window;
generating semi-structured text from the recognized words; and,
calculating a distribution of recognized words from the semi-structured text.
2. The method of claim 1 , further comprising:
normalizing the corpus of text prior to parsing the characters within the corpus of text.
3. The method of claim 2 , wherein:
the normalizing the corpus of text comprises changing a case of at least some of the characters within the corpus of text to conform with a case of other characters within the corpus of text.
4. The method of claim 1 , further comprising:
generating a visual representation of the distribution of recognized words.
5. The method of claim 1 , wherein:
the generating semi-structured text from the recognized words comprises performing a Natural Language Processing operation on the recognized words.
6. The method of claim 1 , wherein:
the calculating a distribution comprises summarizing a frequency distribution of the recognized word.
7. A system comprising:
a processor;
a data bus coupled to the processor; and
a non-transitory, computer-readable storage medium embodying computer program code, the non-transitory, computer-readable storage medium being coupled to the data bus, the computer program code interacting with a plurality of computer operations and comprising instructions executable by the processor and configured for:
receiving a corpus of text, the corpus of text comprising a text string, at least a portion of the corpus of text comprising garbled text, the garbled text comprising a set of confused or unintelligible words, the set of confused or unintelligible words comprising at least one of two words concatenated into a single text string and a mistakenly capitalized character within the text string;
parsing characters within the corpus of text to provide parsed characters from the corpus of text, the parsing characters within the corpus of text comprising applying a text parsing window to a number of characters to accommodate a string of meaningful text, the parsing characters using the text parsing window to sequentially search characters within the corpus of text;
parsing the parsed characters to generate recognized words from the parsed characters, the parsing the parsed characters generating words from longer strings of text before words from shorter strings of text, the longer strings of text being identified using a longer text parsing window;
generating semi-structured text from the recognized words; and,
calculating a distribution of recognized words from the semi-structured text.
8. The system of claim 7 , wherein the instructions executable by the processor are further configured for:
normalizing the corpus of text prior to parsing the characters within the corpus of text.
9. The system of claim 8 , wherein:
the normalizing the corpus of text comprises changing a case of at least some of the characters within the corpus of text to conform with a case of other characters within the corpus of text.
10. The system of claim 7 , wherein the instructions executable by the processor are further configured for:
generating a visual representation of the distribution of recognized words.
11. The system of claim 7 , wherein:
the generating semi-structured text from the recognized words comprises performing a Natural Language Processing operation on the recognized words.
12. The system of claim 7 , wherein:
the calculating a distribution comprises summarizing a frequency distribution of the recognized word.
13. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
receiving a corpus of text, the corpus of text comprising a text string, at least a portion of the corpus of text comprising garbled text, the garbled text comprising a set of confused or unintelligible words, the set of confused or unintelligible words comprising at least one of two words concatenated into a single text string and a mistakenly capitalized character within the text string;
parsing characters within the corpus of text to provide parsed characters from the corpus of text, the parsing characters within the corpus of text comprising applying a text parsing window to a number of characters to accommodate a string of meaningful text, the parsing characters using the text parsing window to sequentially search characters within the corpus of text;
parsing the parsed characters to generate recognized words from the parsed characters, the parsing the parsed characters generating words from longer strings of text before words from shorter strings of text, the longer strings of text being identified using a longer text parsing window;
generating semi-structured text from the recognized words; and,
calculating a distribution of recognized words from the semi-structured text.
14. The non-transitory, computer-readable storage medium of claim 13 , wherein the computer executable instructions are further configured for:
normalizing the corpus of text prior to parsing the characters within the corpus of text.
15. The non-transitory, computer-readable storage medium of claim 14 , wherein:
the normalizing the corpus of text comprises changing a case of at least some of the characters within the corpus of text to conform with a case of other characters within the corpus of text.
16. The non-transitory, computer-readable storage medium of claim 13 , wherein the instructions executable by the processor are further configured for:
generating a visual representation of the distribution of recognized words.
17. The non-transitory, computer-readable storage medium of claim 13 , wherein:
the generating semi-structured text from the recognized words comprises performing a Natural Language Processing operation on the recognized words.
18. The non-transitory, computer-readable storage medium of claim 13 , wherein:
the calculating a distribution comprises summarizing a frequency distribution of the recognized word.
19. The non-transitory, computer-readable storage medium of claim 13 , wherein:
the computer executable instructions are deployable to a client system from a server system at a remote location.
20. The non-transitory, computer-readable storage medium of claim 13 , wherein:
the computer executable instructions are provided by a service provider to a user on an on-demand basis.