Lean parsing: a natural language processing system and method for parsing domain-specific languages
A method and system parses natural language in a unique way, determining important words pertaining to a text corpus of a particular genre, such as tax preparation. Sentences extracted from instructions or forms pertaining to tax preparation, for example are parsed to determine word groups forming various parts of speech, and then are processed to exclude words on an exclusion list and word groups that don't meet predetermined criteria. From the resulting data, synonyms are replaced with a common functional operator and the resulting sentence text is analyzed against predetermined patterns to determine one or more functions to be used in a document preparation system.
1. A method performed by one or more processors of a system, the method comprising:
receiving a set of form data that includes a plurality of sentences;
isolating a number of related sentence segments from the plurality of sentences;
identifying a first set of the sentence segments that includes one or more natural language tokens;
identifying a second set of the sentence segments that do not include natural language tokens;
generating dependency data for each operator of a set of operators associated with the first set of sentence segments;
determining a first predicate structure for each sentence segment of the first set of sentence segments based on the one or more natural language tokens and the generated dependency data;
identifying a first set of tokens included in the second set of sentence segments that match at least one of a number of tokens previously associated with the system;
identifying a second set of tokens included in the second set of sentence segments that do not match at least one of the number of tokens previously associated with the system;
mapping each token of the first and second sets of tokens to at least one operator of the set of operators based at least in part on the generated dependency data;
determining a second predicate structure for each sentence segment of the second set of segments based on the mapping;
associating each predicate structure of the first and second predicate structures with at least one system function; and
executing the at least one system function according to the received set of form data.
2. The method of claim 1 , further comprising:
identifying a number of tokens of the first and second sets of tokens that are a subset of a longer token; and
removing the identified tokens from the first and second sets of tokens.
3. The method of claim 1 , further comprising:
assigning a particular segment type to one or more of the isolated related sentence segments; and
removing the one or more isolated related sentence segments assigned to the particular segment type from the isolated related sentence segments.
4. The method of claim 1 , further comprising:
identifying at least one token included in the number of related sentence segments associated with an exclusion list; and
removing the at least one token from the number of related sentence segments.
5. The method of claim 1 , wherein each of the one or more natural language tokens is associated with at least one operator of a set of operators including at least one of add, subtract, multiply, divide, less than, greater than, and, or, equal to, or not equal to.
6. The method of claim 1 , wherein each sentence segment of the number of related sentence segments has at least one segment type of a number of segment types including at least one of description, amount, instruction, condition, date, or person status.
7. The method of claim 1 , further comprising:
identifying at least one token in the number of related sentence segments that does not pass a token test; and
removing the at least one token from the number of related sentence segments.
8. The method of claim 1 , further comprising:
identifying at least one token in the number of related sentence segments that does not have a particular speech characteristic; and
removing the at least one token from the number of related sentence segments.
9. The method of claim 1 , further comprising:
identifying at least one token in the number of related sentence segments that is a noun; and
removing the at least one token from the number of related sentence segments.
10. The method of claim 1 , further comprising:
identifying at least one token in the number of related sentence segments that is a verb; and
removing the at least one token from the number of related sentence segments.
11. The method of claim 1 , further comprising:
determining, for each token in the number of related sentence segments, a frequency of appearance in a text corpus;
identifying at least one token having a token frequency above a value; and
associating a particular speech characteristic with the at least one token.
12. The method of claim 1 , further comprising:
identifying one or more similar tokens in the number of related sentence segments; and
replacing each of the one or more similar tokens with a single word synonym.
13. The method of claim 1 , wherein the generated dependency data indicates at least one of a first data value for a first data field included in the set of form data, a second data value for a second data field included in a different set of form data, or a constant.
14. The method of claim 1 , wherein the plurality of sentences are associated with at least one of a new tax form or an updated tax form.
15. A system, comprising:
one or more processors; and
at least one memory coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving a set of form data that includes a plurality of sentences;
isolating a number of related sentence segments from the plurality of sentences;
identifying a first set of the sentence segments that includes one or more natural language tokens;
identifying a second set of the sentence segments that do not include natural language tokens;
generating dependency data for each operator of a set of operators associated with the first set of sentence segments;
determining a first predicate structure for each sentence segment of the first set of sentence segments based on the one or more natural language tokens and the generated dependency data;
identifying a first set of tokens included in the second set of sentence segments that match at least one of a number of tokens previously associated with the system;
identifying a second set of tokens included in the second set of sentence segments that do not match at least one of the number of tokens previously associated with the system;
mapping each token of the first and second sets of tokens to at least one operator of the set of operators based at least in part on the generated dependency data;
determining a second predicate structure for each sentence segment of the second set of segments based on the mapping;
associating each predicate structure of the first and second predicate structures with at least one system function; and
executing the at least one system function according to the received set of form data.
16. The system of claim 15 , wherein execution of the instructions causes the system to perform operations further comprising:
identifying a number of tokens of the first and second sets of tokens that are a subset of a longer token; and
removing the identified tokens from the first and second sets of tokens.
17. The system of claim 15 , wherein execution of the instructions causes the system to perform operations further comprising:
assigning a particular segment type to one or more of the isolated related sentence segments; and
removing the one or more isolated related sentence segments assigned to the particular segment type from the isolated related sentence segments.
18. The system of claim 15 , wherein execution of the instructions causes the system to perform operations further comprising:
identifying at least one token included in the number of related sentence segments associated with an exclusion list; and
removing the at least one token from the number of related sentence segments.
19. The system of claim 15 , wherein each of the one or more natural language tokens is associated with at least one operator of a set of operators including at least one of add, subtract, multiply, divide, less than, greater than, and, or, equal to, or not equal to.
20. The system of claim 15 , wherein each sentence segment of the number of related sentence segments has at least one segment type of a number of segment types including at least one of description, amount, instruction, condition, date, or person status.
21. The system of claim 15 , wherein execution of the instructions causes the system to perform operations further comprising:
identifying at least one token in the number of related sentence segments that does not pass a token test; and
removing the at least one token from the number of related sentence segments.
22. The system of claim 15 , wherein execution of the instructions causes the system to perform operations further comprising:
identifying at least one token in the number of related sentence segments that does not have a particular speech characteristic; and
removing the at least one token from the number of related sentence segments.
23. The system of claim 15 , wherein execution of the instructions causes the system to perform operations further comprising:
identifying at least one token in the number of related sentence segments that is a noun; and
removing the at least one token from the number of related sentence segments.
24. The system of claim 15 , wherein execution of the instructions causes the system to perform operations further comprising:
identifying at least one token in the number of related sentence segments that is a verb; and
removing the at least one token from the number of related sentence segments.
25. The system of claim 15 , wherein execution of the instructions causes the system to perform operations further comprising:
determining, for each token in the number of related sentence segments, a frequency of appearance in a text corpus;
identifying at least one token having a token frequency above a value; and
associating a particular speech characteristic with the at least one token.
26. The system of claim 15 , wherein execution of the instructions causes the system to perform operations further comprising:
identifying one or more similar tokens in the number of related sentence segments; and
replacing each of the one or more similar tokens with a single word synonym.
27. The system of claim 15 , wherein the generated dependency data indicates at least one of a first data value for a first data field included in the set of form data, a second data value for a second data field included in a different set of form data, or a constant.
28. The system of claim 15 , wherein the plurality of sentences are associated with at least one of a new tax form or an updated tax form.