Post-optical character recognition error correction system and methods of use
In an exemplary embodiment, the invention comprises a principled edit-distance system that performs a method for determining the probability of character errors. In another exemplary embodiment, the invention comprises a post-OCR error correction system that performs a context-sensitive correction method. In another exemplary embodiment, the invention comprises a post-OCR error correction system that performs a comprehensive, unified correction process based on generalized edit distance analysis, wherein the objective is to find a corrected sentence that has the overall smallest edit distance across all levels. In another exemplary embodiment, the invention comprises a post-OCR error correction system that comprises one or more subjective fractional rank-based dictionaries. In another embodiment, the invention comprises a post-OCR error correction system that performs the automatic assignment of rank to words per-document dictionaries.
1. A method for determining a probability of optical character recognition substitution errors in a text to thereby improve the likelihood of properly correcting optical character recognition substitution errors made during a post-optical character recognition process, said method comprising: estimating character substitution probabilities based on pixel edit distances between characters, wherein said estimating comprises: determining a pixel layout of a candidate character; determining a pixel layout of an error character; and determining a number of non-overlapping pixels between the candidate character and the error character, wherein the method for determining the probability of optical character recognition substitution errors further comprises one or more of: determining a word edit distance between a candidate word that contains the candidate character and an error word that contains the error character, which includes: (i) counting a number of spaces between a series of characters that form a candidate word to derive a number of candidate words; (ii) counting a number of spaces between a series of characters that form an error word to derive a number of error words; and (iii) subtracting the number of candidate words from the number of error words; and determining the character edit distance between the candidate character and the error character, which includes: (i) identifying each character contained in the candidate word to derive a set of candidate characters; (ii) identifying each character contained in the error word to derive a set of error characters; and (iii) identifying a number of characters that differ between the set of candidate characters and the set of error characters.
2. The method of claim 1 , wherein the method for determining the probability of optical character recognition substitution errors further comprises: determining the word edit distance between the candidate word that contains the candidate character and the error word that contains the error character; and determining the character edit distance between the candidate character and the error character.
3. The method of claim 2 , wherein the method for determining the probability of optical character recognition substitution errors further comprises taking a sum of the word edit distance, the character edit distance, and the pixel edit distance.
4. The method of claim 2 , wherein the method for determining the probability of optical character recognition substitution errors further comprises assigning a weight value to the candidate word, assigning a weight value to the candidate character, and assigning a weight value to the candidate pixel.
5. The method of claim 4 , wherein the method for determining the probability of optical character recognition substitution errors further comprises: taking a product of the word edit distance and the weight value of the word edit distance to derive a word product; taking a product of the character edit distance and the weight value of the candidate character to derive a character product; taking a product of the pixel edit distance and the weight value of the candidate pixel to derive a pixel product; and summing the word product, the character product, and the pixel product.
6. The method of claim 5 , wherein assigning the weight value to the candidate character and assigning the weight value to the candidate pixel is based on pixel edit distances.
7. The method of claim 5 , wherein the method for determining the probability of optical character recognition substitution errors further comprises assigning a rank value to the candidate word, wherein the rank value is added to the sum of the word edit distance, the character edit distance, and the pixel edit distance.
8. The method of claim 5 , wherein the method further comprises performing a context-sensitive step, said step comprising obtaining contextual information from a language model, and thereby assigning a context-sensitive rank value.
9. The method of claim 8 , wherein the method for determining the probability of optical character recognition substitution errors further comprises adding the context sensitive rank value to the sum of the word edit distance, the character edit distance, and the pixel edit distance.