Automated methods and systems for locating document subimages in images to facilitate extraction of information from the located document subimages
The present document is directed to methods and subsystems that identify and characterize document-containing subimages in a document-containing image. In one implementation, each type of document is modeled as a set of features that are extracted from a set of images known to contain the document. To locate and characterize a document subimage in an image, the currently described methods and subsystems extract features from the image and then match model features of each model in a set of models to the extracted features to select the model that best corresponds to the extracted features. Additional information contained in the selected model is then used to identify the location of the subimage corresponding to the document and to process the document subimage to correct for a variety of distortions and deficiencies in order to facilitate subsequent data extraction from the corrected document subimage.
1. A document-image-processing system comprising:
a memory; and
a processor operatively coupled to the memory, the processor to:
generate one or more detector-associated sets of features from an image;
apply one or more document-type models to the image, wherein each of the document-type models includes a set of model features to generate a set of final feature pairs, each final feature pair comprising a first feature of the model features and a second matching feature of the one or more detector-associated sets of features; and
in response to determining, based on the final feature pairs, that a document subimage of a document type corresponding to one of the document-type models is present in the image, generate and store locations of one or more points in the document subimage and a document-subimage orientation related to the document subimage.
2. The document-image-processing system of claim 1 further comprising:
for each document subimage determined to be present in the image,
using the locations of one or more points in the document subimage and the document-subimage orientation to extract information from the document subimage.
3. The document-image-processing system of claim 1 , wherein generating the one or more detector-associated sets of features from the image comprises applying one or more document-type models to the image by:
for each detector that detected a model feature included in the model,
matching features generated from the image by the detector to model features detected by the detector to produce an initial set of matching feature pairs,
distributing the matching feature pairs into a transform space, and
adding, to the final set of feature pairs, feature pairs that are grouped together in the transform space.
4. The document-image-processing system of claim 3 wherein matching features generated from the image by the detector to model features detected by the detector further comprises:
selecting, for each model feature, a closest feature generated by the detector, wherein a distance between the model feature and features generated by the detector is determined by a distance metric in a space of feature attribute values, each dimension of the space corresponding to a feature attribute; and
selecting the closest feature as a matching feature when the distance between the model feature and the closest feature is less than a threshold value.
5. The document-image-processing system of claim 3 wherein distributing the matching feature pairs into a transform space further comprises:
computing a difference in two or more feature attributes between the model feature and detector-generated feature of each matching feature pair; and
distributing each matching feature pair into the transform space with dimensions each corresponding to the difference between two attribute values of a feature attribute.
6. The document-image-processing system of claim 1 , further comprising wherein applying one or more tests to the set of final feature pairs to determine whether or not a document subimage of the document type corresponding to the model is present in the image further comprises:
applying one or more of
a test that determines a ratio of a number of the model features in feature pairs of the final set of feature pairs to the number of model features in the model and compares the ratio to a threshold value; and
a test that determines a ratio of a number of detectors that generated one or more features included in the final set of feature pairs to a number of detectors that generated one or more model features and compares the ratio to a threshold value.
7. The document-image-processing system of claim 1 , further comprising applying a transform that transforms model-feature locations to image locations to generate and store the locations of one or more points in the document subimage and the document-subimage orientation, wherein the transform that transforms model-feature locations to image locations is generated by:
applying a parameter-fitting method to generate a matrix model that, when applied to a coordinate vector for a point in a template image for the document used to generate the model, produces a corresponding coordinate vector for a point in the image.
8. The document-image-processing system of claim 7 wherein the parameter-fitting method is a RANSAC-like parameter-fitting method that uses a proximity constraint when selecting initial model points and when adding points to the model.
9. The document-image-processing system of claim 7 , further comprising applying a transform that transforms model-feature locations to image locations to generate and store the locations of one or more points in the document subimage and to generate and store the document-subimage orientation by:
selecting comer points of the document in the template image for the document; using the matrix model to transform coordinate pairs that describe the location of the comer points of the document in the template image to corresponding image coordinate pairs; and
using two image coordinate pairs for document comers to compute a line segment and using the line segment to compute an orientation of the line segment with respect to an image edge.
10. The document-image-processing system of claim 1 , wherein generating the one or more detector-associated sets of features from the image comprises applying one or more feature detectors to the image, wherein the one or more feature detectors each identifies particular locations, geometric features, or other features in the image, such as SIFT features generated by a SIFT feature detector, and wherein the features generated by feature detectors are characterized by attribute values.
11. The document-image-processing system of claim 10 wherein attribute values that characterize features include one or more of:
a magnitude, a scale, an orientation,
a coordinate pair identifying a location, and a descriptor with multiple elements.
12. A method for processing document-images, the method comprising:
generating one or more detector-associated set of features from an image;
applying one or more document-type models to the image, wherein each the document-type models comprises a set of model features to generate a set of final feature pairs, each final feature pair comprising a first feature of the model features and a second matching feature of the one or more detector-associated sets of features; and
in response to determining that a document subimage of a document type corresponding to at least one of the document-type models is present in the image, generating and store locations of one or more points in the document subimage and a document-subimage orientation related to the document subimage.