Adaptive parser-centric text normalization
Embodiments of the present invention relate to a customizable text normalization framework providing for domain adaptability through modular replacement generators. In one embodiment, a method of and computer program product for text normalization are provided. An input sequence comprising a plurality of tokens is received. A plurality of generators is applied to the input sequence to generate a set of candidate replacements of the tokens of the sequence. A plurality of subsets of the set of candidate replacements is determined such that the candidate replacements of each subset are syntactically consistent. A probability is determined for each of the subsets. A subset of the plurality of subsets having the highest probability is selected. Each candidate replacement of the selected subset is applied to the input sequence to generate an output sequence. The output sequence is outputted.
1. A method comprising:
receiving at a computing node an input sequence comprising a plurality of tokens;
applying by a processor of the computing node a plurality of domain-specific generators to the input sequence to generate a set of candidate replacements of the tokens of the input sequence;
creating in a memory of the computing node a directed graph comprising a plurality of nodes and a plurality of edges, each node having an associated candidate replacement of the set of candidate replacements, and each edge connecting a first node to a second node, the second node being associated with a consistent follower of the candidate replacement associated with the first node, and creating the plurality of edges comprising determining syntactic consistency between each pair of the set of candidate replacements;
determining by the processor a plurality of paths in the directed graph, each of the plurality of paths comprising at least one of the plurality of edges;
determining by the processor a score for each of the paths;
selecting by the processor a path of the plurality of paths having the highest score;
applying by the processor each candidate replacement of the selected path to the input sequence to generate a normalized output sequence; and
evaluating a correctness of the normalized output sequence by parsing the normalized output sequence to obtain a parse result and comparing the parse result with a gold standard that is obtained by parsing a manually normalized sequence.
2. The method of claim 1 , wherein the graph is directed and acyclic.
3. The method of claim 1 , wherein determining the score for each of the paths comprises aggregating a plurality of edge weights along each path.
4. The method of claim 3 , further comprising:
determining edge weights from a training data set, the training data set comprising a plurality of training input sequences, each training input sequence associated with a normalized sequence.
5. The method of claim 4 , wherein determining edge weights comprises performing maximum likelihood estimation.
6. The method of claim 4 , wherein determining edge weights comprises iteratively generating a sequence with a highest score based on the edge weights and updating the edge weights by comparing the sequence with the highest score to the normalized sequence.
7. The method of claim 1 , further comprising:
comparing the normalized output sequence to a training sequence to determine a number of subjects, verbs, and objects overlapping between the training sequence and the normalized output sequence.
8. The method of claim 1 , wherein one of the plurality of generators determines a candidate replacement by determining a minimum edit distance from one of the plurality of input tokens.
9. The method of claim 1 , wherein one of the plurality of generators determines a candidate replacement by changing a case of one of the plurality of input tokens.
10. The method of claim 1 , wherein one of the plurality of generators determines a candidate replacement by correcting the spelling of one of the plurality of input tokens.
11. The method of claim 1 , wherein one of the plurality of generators determines a candidate replacement by expanding a contraction of one of the plurality of input tokens.
12. The method of claim 1 , wherein one of the plurality of generators determines a candidate replacement by looking up one of the plurality of input tokens in a dictionary.
13. The method of claim 1 , wherein the score comprises a probability.
14. A computer program product for text normalization, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable by a processor to:
receive at a computing node an input sequence comprising a plurality of tokens;
apply by a processor of the computing node a plurality of generators to the input sequence to generate a set of candidate replacements of the tokens of the input sequence;
create in a memory of the computing node a directed graph comprising a plurality of nodes and a plurality of edges, each node having an associated candidate replacement of the set of candidate replacements, and each edge connecting a first node to a second node, the second node being associated with a consistent follower of the candidate replacement associated with the first node, and creating the plurality of edges comprising determining syntactic consistency between each pair of the set of candidate replacements;
determine by the processor a plurality of paths in the directed graph, each of the plurality of paths comprising at least one of the plurality of edges;
determine by the processor a score for each of the paths;
select by the processor a path of the plurality of paths having the highest score;
apply by the processor each candidate replacement of the selected path to the input sequence to generate a normalized output sequence; and
evaluate a correctness of the normalized output sequence by parsing the normalized output sequence to obtain a parse result and comparing the parse result with a gold standard that is obtained by parsing a manually normalized sequence.
15. The computer program product of claim 14 , wherein the graph is directed and acyclic.
16. The computer program product of claim 14 , wherein determining the score for each of the paths comprises aggregating a plurality of edge weights along each path.
17. The computer program product of claim 16 , the program code being further executable to:
determining edge weights from a training data set, the training data set comprising a plurality of training input sequences, each training input sequence associated with a normalized sequence.
18. The computer program product of claim 17 , wherein determining edge weights comprises performing maximum likelihood estimation.
19. The computer program product of claim 17 , wherein determining edge weights comprises iteratively generating a sequence with a highest score based on the edge weights and updating the edge weights by comparing the sequence with the highest score to the normalized sequence.
20. The computer program product of claim 17 , the program code being further executable to:
comparing the normalized output sequence to a training sequence to determine a number of subjects, verbs, and objects overlapping between the training sequence and the normalized output sequence.