Vector space model for form data extraction
A computer-implemented method for detecting attribute value pairs from corpus data using a computer comprising a processor and a computer readable medium comprising instructions executable by the processor to at least: receive the corpus data comprising at least one pair; detect a layout template of the at least one pair; measure the merit of the layout template by determining at least one of (a) relative magnitudes of content probabilities based on a probability of the contents of an attribute cell and a probability of a corresponding value cell, (b) the validity of a name-value pair, or (c) the pointwise mutual information of a frequency matrix M corresponding to a sparse vector capturing context information of a word; and output detected attribute value pairs.
1 . A computer-implemented vector space model method for detecting attribute value pairs from corpus data using a computer comprising a processor and a computer readable medium comprising instructions executable by the processor to at least:
receive the corpus data comprising at least one attribute value pair;
detect a layout template of the at least one attribute value pair;
measure the merit of the layout template by determining at least one of (a) relative magnitudes of content probabilities based on a probability of the contents of an attribute cell and a probability of a corresponding value cell, (b) validity of a name-value pair, or (c) the pointwise mutual information of a frequency matrix M corresponding to a sparse vector capturing context information of a word; and
output detected attribute value pairs.
2 . The computer-implemented method of claim 1 , wherein the relative magnitudes of content probabilities are determined by a scoring function S( ):
S
(
a
)
=
log
P
(
x
att
)
P
(
x
val
)
where P(x att) is the probability of the contents of the attribute cell and P(x val ) is the probability of the corresponding value cell.
3 . The computer-implemented method of claim 1 , wherein the validity of the name value pair is determined by a step function S( ):
S
(
a
)
=
{
1
if
P
(
x
att
)
>
P
(
x
val
)
0
otherwise
where P(x att) is the probability of the contents of the attribute cell and P(x val ) is the probability of the corresponding value cell.
4 . The computer-implemented method of claim 1 , wherein the pointwise mutual information is determined by a step function S( ):
S
(
a
)
=
{
1
if
PMI
(
x
att
,
x
val
)
>
0
0
otherwise
where x att and x val are two table cells being paired.
5 . The computer-implemented method of claim 4 , wherein the attribute value pair corresponds to at least one of a pair of words, table cells and text phrases.
6 . A computer-implemented method for extracting data from data-bearing tables, using a computer comprising a processor and a computer readable medium comprising instructions executable by the processor to at least:
acquire at least one data-bearing table comprising a plurality of rows and a plurality of columns of data values;
assign at least one table layout template for the at least one data-bearing table;
generate a set of attribute value pairs, A c , for the at least one table layout template;
compute a content score, W content defining a percentage of likely attribute-value pairs for the at least one table layout template;
compute a template score, W template based on attribute-value pairs and word frequencies for the at least one table layout template;
determine whether content score, W content is greater than a threshold associated with a percentage of likely attribute value pairs;
when content score, W content is greater than the threshold, identify a most suitable at least one table layout template for the at least one data-bearing table based on the template score, W template and output a set of attribute value pairs.
7 . The computer-implemented method of claim 6 , wherein the table layout is at least one of a relation layout, an entity layout and an other layout.
8 . The computer-implemented method of claim 7 , wherein the relation layout comprises at least one attribute name found in a first row of the plurality of rows.
9 . The computer-implemented method of claim 7 , wherein the entity layout comprises at least one attribute name found in a first column of the plurality of columns.
10 . The computer-implemented method of claim 7 , wherein the other layout comprises at least one of a matrix and mixed formats.
11 . An apparatus for extracting data from data-bearing tables using a vector space model, the apparatus comprising at least one processor, a computer readable medium comprising instructions executable by the at least one processor to cause the processor, the apparatus comprising:
a dataset acquisition module for acquiring at least one data-bearing table from a data source, the at least one data-bearing table comprising a plurality of rows and a plurality of columns of data values;
a categorizer for assigning at least one table layout template for the at least one data-bearing table;
an attribute generator for generating a set of attribute value pairs, A c , for the at least one table layout template;
a table quality analyzer for computing a content score, W content defining a percentage of likely attribute-value pairs for the at least one table layout template; for computing a template score, W template based on attribute-value pairs and word frequencies for the at least one table layout template;
a classification module for identifying a most suitable at least one table layout template for the at least one data-bearing table based on the template score, W template and a relationship between the content score, W content and a threshold associated with a percentage of likely attribute value pairs, and output a set of attribute value pairs.
12 . The apparatus of claim 11 , wherein the table layout is at least one of a relation layout, an entity layout and an other layout.
13 . The apparatus of claim 11 , wherein the relation layout comprises at least one attribute name found in a first row of the plurality of rows.
14 . The apparatus of claim 11 , wherein the entity layout comprises at least one attribute name found in a first column of the plurality of columns.
15 . The apparatus of claim 11 , wherein the other layout comprises at least one of a matrix and mixed formats.
16 . The apparatus of claim 11 , wherein a merit of the at least one layout template by determining at least one of (a) relative magnitudes of content probabilities based on a probability of the contents of an attribute cell and a probability of a corresponding value cell, (b) validity of a name-value pair, or (c) pointwise mutual information of a frequency matrix M corresponding to a sparse vector capturing context information of a word; and output detected attribute value pairs.
17 . The apparatus of claim 16 , wherein the relative magnitudes of content probabilities are determined by a scoring function S( ):
S
(
a
)
=
log
P
(
x
att
)
P
(
x
val
)
where P (x att) is the probability of the contents of the attribute cell and P(x val ) is the probability of the corresponding value cell.
18 . The apparatus of claim 16 , wherein the validity of the name value pair is determined by a step function S( ):
S
(
a
)
=
{
1
if
P
(
x
att
)
>
P
(
x
val
)
0
otherwise
where P(x att) is the probability of the contents of the attribute cell and P(x val ) is the probability of the corresponding value cell.
19 . The apparatus of claim 16 , wherein the pointwise mutual information is determined by a step function S( ):
S
(
a
)
=
{
1
if
PMI
(
x
att
,
x
val
)
>
0
0
otherwise
where x att and x val are two table cells being paired.
20 . The apparatus of claim 16 , wherein the attribute value pair corresponds to at least one of a pair of words, table cells and text phrases.