System and method for identification and extraction of data
A system and method of for describing target data as a sequence of pattern elements and pattern element groups that comprise an overall target pattern is described. Pattern elements may utilize regular expression syntax along with other metadata that describe the behavior of the element. A pattern element group may be a collection of fully defined pattern elements where at least one pattern element from the group must have a match for the overall pattern to match. Patterns contain both pattern elements and pattern element groups. The general process involves first performing optical character recognition (OCR) on the document, which in turn produces a sequence of text tokens representing the lines of text on each page of the document. The search algorithm may then apply each defined pattern to the entire document capturing and/or extracting data that match each pattern's required elements and element groups.
1. A system for identifying and extracting text from an electronic document, the system comprising:
one or more processors;
memory; and
a text identifier and extractor that receives the electronic document, generates a stream of text tokens representing a plurality of lines of text of the electronic document, matches a pattern to a portion of the stream of text tokens, and outputs the text in accordance with the matched pattern,
wherein,
the pattern includes an ordered sequence of a plurality of pattern elements representing the plurality of lines of text,
each pattern element of the plurality of pattern elements describes at least one text token,
the text identifier and extractor matches a text token in the stream of text tokens to a pattern element and continues to consume text tokens from the stream of text tokens until a subsequent text token in the stream of text tokens is matched to a subsequent pattern element having a required attribute,
the pattern element and the subsequent pattern element belong to the same pattern, and
the electronic document is a transcript or certificate.
2. The system of claim 1 , wherein at least one pattern element of the plurality of pattern elements describes the at least one text token according to a regular expression.
3. The system of claim 1 , wherein the plurality of pattern elements further includes attributes that indicate whether a pattern element is one or more of greedy, required, and can be discarded.
4. The system of claim 1 , further comprising at least one pattern element group, wherein the pattern element group includes a plurality of pattern elements.
5. The system of claim 4 , wherein the pattern is further defined according to a sequence of pattern elements and pattern element groups.
6. The system of claim 1 , further including a scanning input device, wherein the scanning input device transforms a physical document into the electronic document, and wherein the electronic document is an electronic representation of the physical document.
7. A method for identifying and extracting text from an electronic document, the method comprising:
receiving, at a text identifier and extractor, the electronic document;
generating a stream of text tokens representing a plurality of lines of text of the electronic document;
matching a pattern to a portion of the stream of text tokens, the pattern including an ordered sequence of a plurality of pattern elements representing the plurality of lines of text, where each pattern element of the plurality of pattern elements describes at least one text token;
matching a text token in the stream of text tokens to a pattern element and continuing to consume text tokens from the stream of text tokens until a subsequent text token in the stream of text tokens is matched to a subsequent pattern element having a required attribute, wherein the pattern element and the subsequent pattern element belong to the same pattern; and
outputting the text in accordance with the matched pattern, wherein the electronic document is a transcript or certificate.
8. The method of claim 7 , wherein at least one pattern element of the plurality of pattern elements describes the at least one text token according to a regular expression.
9. The method of claim 7 , wherein the plurality of pattern elements further includes attributes that indicate whether a pattern element is one or more of greedy, required, and can be discarded.
10. The method of claim 7 , further comprising at least one pattern element group, wherein the pattern element group comprises a plurality of pattern elements.
11. The method of claim 10 , wherein the pattern is further defined according to a sequence of pattern elements and pattern element groups.
12. The method of claim 7 , further comprising transforming a physical document into the electronic document, wherein the electronic document is an electronic representation of the physical document.
13. A non-transitory computer-readable storage medium including processor-executable instructions operable to perform a method comprising:
receiving, at a text identifier and extractor, an electronic document;
generating a stream of text tokens representing a plurality of lines of text of the electronic document;
matching a pattern to a portion of the stream of text tokens, the pattern including an ordered sequence of a plurality of pattern elements representing the plurality of lines of text, where each pattern element of the plurality of pattern elements describes at least one text token;
matching a text token in the stream of text tokens to a pattern element and continuing to consume text tokens from the stream of text tokens until a subsequent text token in the stream of text tokens is matched to a subsequent pattern element having a required attribute, wherein the pattern element and the subsequent pattern element belong to the same pattern; and
outputting text in accordance with the matched pattern, wherein the electronic document is a transcript or certificate.
14. The non-transitory computer-readable storage medium of claim 13 , wherein each of the plurality of pattern elements describes the at least one text token according to a regular expression.
15. The non-transitory computer-readable storage medium of claim 13 , wherein the plurality of pattern elements further includes attributes that indicate whether at least one pattern element of the plurality of pattern elements is one or more of greedy, required, and can be discarded.
16. The non-transitory computer-readable storage medium of claim 13 , further comprising at least one pattern element group, wherein the pattern element group comprises a plurality of pattern elements.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the pattern is further defined according to a sequence of pattern elements and pattern element groups.