Systems and methods for field extraction from unlabeled data
Embodiments described a field extraction system that does not require field-level annotations for training. Specifically, the training process is bootstrapped by mining pseudo-labels from unlabeled forms using simple rules. Then, a transformer-based structure is used to model interactions between text tokens in the input form and predict a field tag for each token accordingly. The pseudo-labels are used to supervise the transformer training. As the pseudo-labels are noisy, a refinement module that contains a sequence of branches is used to refine the pseudo-labels. Each of the refinement branches conducts field tagging and generates refined labels. At each stage, a branch is optimized by the labels ensembled from all previous branches to reduce label noise.
1 . A computer-vision method for field extraction from unlabeled images of forms through a field extraction model, the method comprising:
receiving, via a data interface, an unlabeled image of a form comprising a plurality of fields and a plurality of field values;
detecting, by an optical character recognition (OCR) engine, a first word and a first location of the first word within the unlabeled image of the form;
encoding, by an encoder of the field extraction model implemented on one or more processors, a pair of the first word and the first location into a first feature representation;
generating, by a plurality of pseudo-labels ensemble (PLE) models of the field extraction model placed in parallel and connected to the encoder, a plurality of predicted labels based at least in part on the first feature representation, respectively, wherein the plurality of the PLE models comprise a first PLE model that generates a first field classification score from an input of a second predicted label generated by a second PLE model of the plurality of the PLE models, and then converts the first field classification score to a first predicted label;
computing, at the first PLE model, a corresponding loss component by comparing the first predicted label generated by the first PLE model and a subset of predicted labels generated from a subset of the plurality of PLE models other than the first PLE model as pseudo-labels;
training the plurality of PLE models using a loss objective as a sum of loss components over the plurality of PLE models; and
in response to receiving at least one unlabeled image of the form at a field extraction system:
encoding, by the encoder of the field extraction model, at least a detected word and an associated location on the at least one unlabeled image into a second feature representation;
generating, by the trained plurality of PLE models of the field extraction model, a plurality of field classification scores based at least in part on the second feature representation;
predicting one or more fields and corresponding field values on the at least one unlabeled image of the form based on an average of the plurality of field classification scores generated by the trained plurality of PLE models.
2 . The method of claim 1 , wherein each of the plurality of PLE models comprises a respective classifier that generates a respective predicted label based on the first feature representation.
3 . The method of claim 1 , wherein the first predicted label is generated by:
projecting, via one or more fully connected layers, the first feature representation into a set of field prediction scores; and
generating the first predicted label based on a maximum field prediction score among a set of words.
4 . The method of claim 3 , further comprising:
selecting, for a field from the plurality fields, a word that corresponds to the maximum field prediction score from the set of words when the maximum field prediction score is greater than a pre-defined threshold.
5 . The method of claim 1 , further comprising:
detecting, by a processor, a set of words and a set of locations within the unlabeled image of the form for the set of words;
identifying, from the set of words and the set of locations, a field value for a field based at least in part on a geometric relationship between the set of words;
generating, by a classifier, a field classification distribution from the first representation; and
computing a first loss objective by comparing the field classification distribution with the field value as a pseudo-label.
6 . The method of claim 5 , wherein the first PLE model from the plurality of PLE branches uses the identified field value for the field as a first pseudo-label.
7 . The method of claim 5 , further comprising:
computing a joint loss objective by adding the loss objective with the first loss objective; and
jointly updating the encoder and the plurality of PLE models based on the joint loss objective via backpropagation.
8 . The method of claim 5 , further comprising:
updating the encoder based on the first loss objective via backpropagation.
9 . The method of claim 8 , further comprising:
updating the plurality of PLE model based on the loss objective via backpropagation while fixing parameters of the encoder after updating the encoder.
10 . A system for field extraction from unlabeled images of forms through a field extraction model, the system comprising:
a data interface an unlabeled image of a form comprising a plurality of fields and a plurality of field values;
a memory storing a plurality of processor-executed instructions; and
a processor executing the processor-executed instructions to perform operations comprising:
detecting, by an optical character recognition (OCR) engine, a first word and a first location of the first word within the unlabeled image of the form;
encoding, by an encoder of the field extraction model implemented on one or more processors, a pair of the first word and the first location into a first feature representation;
generating, by a plurality of pseudo-labels ensemble (PLE) models of the field extraction model placed in parallel and connected to the encoder, a plurality of predicted labels based at least in part on the first feature representation, respectively, wherein the plurality of the PLE models comprise a first PLE model that generates a first field classification score from an input of a second predicted label generated by a second PLE model of the plurality of the PLE models, and then converts the first field classification score to a first predicted label;
computing, at the first PLE model, a corresponding loss component by comparing the first predicted label generated by the first PLE model and a subset of predicted labels generated from a subset of the plurality of PLE models other than the first PLE model as pseudo-labels;
training the plurality of PLE models using a loss objective as a sum of loss components over the plurality of PLE models; and
in response to receiving at least one unlabeled image of the form at a field extraction system:
encoding, by the encoder of the field extraction model, at least a detected word and an associated location on the at least one unlabeled image into a second feature representation;
generating, by the trained plurality of PLE models of the field extraction model, a plurality of field classification scores based at least in part on the second feature representation;
predicting one or more fields and corresponding field values on the at least one unlabeled image of the form based on an average of the plurality of field classification scores generated by the trained plurality of PLE models.
11 . The system of claim 10 , wherein each of the plurality of PLE models comprises a respective classifier that generates a respective predicted label based on the first feature representation.
12 . The system of claim 10 , wherein the first predicted label is generated by:
projecting, via one or more fully connected layers, the first feature representation into a set of field prediction scores; and
generating the first predicted label based on a maximum field prediction score among a set of words.
13 . The system of claim 12 , wherein the operations further comprise:
selecting, for a field from the plurality fields, a word that corresponds to the maximum field prediction score from the set of words when the maximum field prediction score is greater than a pre-defined threshold.
14 . The system of claim 10 , wherein the operations further comprise:
detecting, by a processor, a set of words and a set of locations within the unlabeled image of the form for the set of words;
identifying, from the set of words and the set of locations, a field value for a field based at least in part on a geometric relationship between the set of words;
generating, by a classifier, a field classification distribution from the first representation; and
computing a first loss objective by comparing the field classification distribution with the field value as a pseudo-label.
15 . The system of claim 14 , wherein the first PLE model from the plurality of PLE branches uses the identified field value for the field as a first pseudo-label.
16 . The system of claim 14 , wherein the operations further comprise:
computing a joint loss objective by adding the loss objective with the first loss objective; and
jointly updating the encoder and the plurality of PLE models based on the joint loss objective via backpropagation.
17 . The system of claim 16 , wherein the operations further comprise:
updating the encoder based on the first loss objective via backpropagation.
18 . The system of claim 17 , wherein the operations further comprise:
updating the plurality of PLE model based on the loss objective via backpropagation while fixing parameters of the encoder after updating the encoder.
19 . A non-transitory storage processor-readable medium storing processor-executable instructions for field extraction from unlabeled images of forms through a field extraction model, the instructions being executed by a processor to perform operations comprising:
receiving, via a data interface, an unlabeled image of a form comprising a plurality of fields and a plurality of field values;
detecting, by an optical character recognition (OCR) engine, a first word and a first location of the first word within the unlabeled image of the form;
encoding, by an encoder of the field extraction model implemented on one or more processors, a pair of the first word and the first location into a first feature representation;
generating, by a plurality of pseudo-labels ensemble (PLE) models of the field extraction model placed in parallel and connected to the encoder, a plurality of predicted labels based at least in part on the first feature representation, respectively, wherein the plurality of the PLE models comprise a first PLE model that generates a first field classification score from an input of a second predicted label generated by a second PLE model of the plurality of the PLE models, and then converts the first field classification score to a first predicted label;
computing, at the first PLE model, a corresponding loss component by comparing the first predicted label generated by the first PLE model and a subset of predicted labels generated from a subset of the plurality of PLE models other than the first PLE model as pseudo-labels;
training the plurality of PLE models using a loss objective as a sum of loss components over the plurality of PLE models; and
in response to receiving at least one unlabeled image of the form at a field extraction system:
encoding, by the encoder of the field extraction model, at least a detected word and an associated location on the at least one unlabeled image into a second feature representation;
generating, by the trained plurality of PLE models of the field extraction model, a plurality of field classification scores based at least in part on the second feature representation;
predicting one or more fields and corresponding field values on the at least one unlabeled image of the form based on an average of the plurality of field classification scores generated by the trained plurality of PLE models.
20 . The non-transitory storage processor-readable medium of claim 19 , wherein each of the plurality of PLE models comprises a respective classifier that generates a respective predicted label based on the first feature representation, and
wherein the first predicted label is generated by:
projecting, via one or more fully connected layers, the first feature representation into a set of field prediction scores; and
generating the first predicted label based on a maximum field prediction score among a set of words.