Statistical pattern generation for information extraction
An apparatus for extracting selected information from a set of symbols includes said alignment module is configured to retrieve test patterns from a symbol input, and to attempt alignment of test patterns with a canonical pattern. Successful alignment between a particular test pattern and said canonical pattern indicates of existence of information of interest in a particular candidate pattern. Upon detection of a successful alignment, the alignment module passes information concerning the test pattern to a user. Additionally, in response to detecting an unsuccessful attempt to align the first test pattern and the canonical pattern, said alignment module passes, to said user, information concerning the first test pattern.
1. An apparatus for extracting selected information of interest from an input data, comprising:
a non-volatile program memory, for storing software instructions;
a frame memory, for storing a frame having a canonical pattern indicating said selected information, wherein said canonical pattern comprises of a plurality of canonical units in a continuous canonical sequence, and each of the canonical units comprises at least one word, wherein said canonical pattern further comprises an index key, and said index key is a subset of said canonical units in said canonical sequence;
a processor, for executing software instructions to implement following steps:
retrieving an input pattern from said input data, wherein said input pattern comprises a plurality of input units in a continuous input sequence;
determining whether said index key is in said input pattern;
setting match result as unsuccessful and skipping following calculating steps and determining step if said index key is not presented in said input pattern;
proceeding to following calculating steps and determining step if said index key is presented in said input pattern;
calculating an alignment score by comparing said input pattern and said canonical pattern, wherein said alignment score indicates a likelihood of said input pattern matches said canonical pattern;
calculating a first combination score for further adjusting said alignment score if said input units comprises said canonical units in said canonical sequence and an inserted unit, wherein said inserted unit is inserted into said canonical sequence of canonical units, said first combination score corresponds to a first frequency in said frame for said inserted unit and a first adjacent leading unit in said canonical sequence, or to a second frequency in said frame for said inserted unit and a first adjacent trailing unit in said canonical sequence; and
determining whether said selected information of interest is in said input data by comparing said alignment score with a threshold.
2. The apparatus of claim 1 , wherein the processor further executes software instructions for calculating a second combination score for further adjusting said alignment score if said input pattern comprises said canonical pattern in said canonical sequence excluding a deleted unit, wherein said second combination score corresponds to a third frequency in said frame for said deleted unit and a second adjacent leading unit in said canonical sequence, or to a fourth frequency in said frame for said deleted unit and a second adjacent trailing unit in said canonical sequence.
3. The apparatus of claim 1 , wherein at least one of the canonical units is a semantic unit for conveying meaning, wherein said frame further includes a topic slot indicating a subject matter associating with said semantic unit.
4. The apparatus of claim 2 , wherein the processor further executes software instructions for statistical characterizing training data to generate said first frequency, said second frequency, said third frequency, and said fourth frequency, wherein said canonical pattern is extracted from said training data.
5. The apparatus of claim 2 , wherein the canonical pattern further comprises a weight vector with a plurality of weights, each of the weights corresponds to one of said canonical units, respectively; and
wherein said processor further executes software instructions to implement following steps:
producing a match vector of said input pattern, each element of said match vector indicates whether a corresponding canonical unit can be found in said input pattern;
calculating an inner product of said weight vector and said match vector as a baseline score; and
adding said baseline score, said first combination score, and said second combination score as said alignment score.
6. The apparatus of claim 1 , wherein said first combination score is negative if said first frequency or said second frequency is rare.
7. The apparatus of claim 6 , wherein said first combination score is set to a negative magnitude to effectively place said alignment score below the threshold.
8. The apparatus of claim 2 , wherein said first combination score or said second combination score is calculated based on a trigram.