IP Library › Granted Patent US 12,340,552
Granted Patent B2
US 12,340,552 · App. 17/348,584 · Granted Jun 24, 2025

Iterative recognition-guided thresholding and data extraction

Inventors: Christopher W. Thrasher (Rochester, NY); Alexander Shustorovich (Pittsford, NY); Stephen Michael Thompson (Oceanside, CA); Jan W. Amtrup (Silver Spring, MD); Anthony Macciola (Irvine, CA)
Assignee: Tungsten Automation Corporation
G06V10/28G06T7/11G06T7/136G06T7/187G06V10/25G06V10/457G06T2207/20104
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,340,552
App. No.
17/348,584
Granted
Jun 24, 2025
Kind
B2
Abstract

Techniques for binarization and extraction of information from image data are disclosed. The inventive concepts include independently binarizing portions of the image data on the basis of individual features, e.g. per connected component, and using multiple different binarization thresholds to obtain the best possible binarization result for each portion of the image data. Determining the quality of each binarization result may be based on attempted recognition and/or extraction of information therefrom. Independently binarized portions may be assembled into a contiguous result. In one embodiment, a method includes: identifying a region of interest within a digital image; generating a plurality of binarized images based on the region of interest using different binarization thresholds; and extracting data from some or all of the plurality of binarized images. The extracted data includes connected components that overlap and/or are obscured by unique background. Corresponding systems and computer program products are disclosed.

Claims (27)

1. A computer-implemented method, comprising:

identifying a region of interest of a document depicted within a color digital image;

generating a plurality of binarized images based on the region of interest, wherein the plurality of binarized images are each independently generated using a different one of a plurality of binarization thresholds; and

extracting data from some or all of the plurality of binarized images;

wherein the extracted data comprises one or more connected components represented in the plurality of binarized images;

wherein one or more of the connected components at least partially overlap or are at least partially obscured by a plurality of unique background elements;

wherein the plurality of unique background elements are each independently characterized by different color profiles within the color digital image; and

independently normalizing color within individual regions the color digital image prior to generating the plurality of binarized images, wherein the normalizing comprises stretching a range of intensity values observed in each of a plurality of color channels corresponding to a plurality of pixels of the color digital image such that:

a minimum intensity value observed in each of the plurality of color channels corresponds to a minimum possible intensity value in the color digital image, and

a maximum intensity value observed in each of the plurality of color channels corresponds to a maximum possible intensity value in the color digital image.

2. The computer-implemented method as recited in claim 1 ,

wherein the color digital image is characterized by an insufficient contrast causing boundaries between foreground and background elements of the digital image to be obscured.

3. The computer-implemented method as recited in claim 1 , wherein the color digital image is characterized by an excessive contrast causing single elements of the digital image to be broken into constituent elements.

4. The computer-implemented method as recited in claim 1 , wherein extracting the data is performed on a per-component basis for at least some of the one or more connected components.

5. The computer-implemented method as recited in claim 1 , comprising: estimating an identity of some or all of the one or more connected components, wherein estimating the identity of the one or more connected components utilizes one or more confidence measures, and wherein estimating the identity of some or all of the one or more connected components comprises:

comparing the estimated identity of each respective one of the one or more connected components for which the identity was estimated with an expected identity of the respective one of the one or more connected components; and

comparing an estimated location, within the color digital image, of each respective one of the one or more connected components for which the identity was estimated with an expected location of the respective one of the one or more connected components.

6. The computer-implemented method as recited in claim 1 , wherein at least some of the data is extracted from a trouble region within the color digital image, and wherein the trouble region is characterized by including: one or more shadows, glare, or a combination thereof.

7. The computer-implemented method as recited in claim 1 , comprising inverting some or all of the plurality of binarized images.

8. The computer-implemented method as recited in claim 1 , wherein at least some of the one or more connected components comprise text characters.

9. The computer-implemented method as recited in claim 1 , wherein generating the plurality of binarized images generates a sequence of binarized images, wherein each of member of the sequence of binarized images depicts a same set of connected components according to different ones of the plurality of binarization thresholds, wherein generating the plurality of binarized images generates a plurality of subsequences of binarized images according to different ones of the plurality of binarization thresholds, and wherein each subsequence corresponds to one or more different subsets of the set of connected components.

10. The computer-implemented method as recited in claim 1 , wherein generating the plurality of binarized images generates a sequence of binarized images for each character represented in the region of interest, wherein each sequence of binarized images generated for each character represented in the region of interest is independently characterized by a different one of the plurality of binarization thresholds.

11. The computer-implemented method as recited in claim 1 , wherein the plurality of pixels of the color digital image are each independently characterized by having an intensity value lower than a predefined minimum intensity threshold in at least one of the plurality of color channels.

12. The computer-implemented method as recited in claim 5 , wherein the one or more confidence measures comprise either or both of: an OCR confidence measure and a location confidence measure, wherein the location confidence measure indicates a degree of confidence that the estimated location of the respective one of the one or more connected components matches the expected location of the respective one of the one or more connected components.

13. The computer-implemented method as recited in claim 1 ,

wherein the digital image is characterized by either an insufficient contrast causing boundaries between foreground and background elements of the digital image to be obscured or an excessive contrast causing single elements of the digital image to be broken into constituent elements; and

wherein at least some of the data is extracted from a trouble region within the color digital image.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 15, 2024
From: KOFAX, INC.
To: TUNGSTEN AUTOMATION CORPORATION
Reel/Frame 067428/0392 →
FIRST LIEN INTELLECTUAL PROPERTY SECURITY AGREEMENT Recorded Jul 20, 2022
From: KOFAX, INC.; PSIGEN SOFTWARE, INC.
To: JPMORGAN CHASE BANK, N.A. AS COLLATERAL AGENT
Reel/Frame 060757/0565 →
SECURITY INTEREST Recorded Jul 20, 2022
From: KOFAX, INC.; PSIGEN SOFTWARE, INC.
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
Reel/Frame 060768/0159 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 16, 2021
From: THRASHER, CHRISTOPHER W.; SHUSTOROVICH, ALEXANDER; THOMPSON, STEPHEN MICHAEL; AMTRUP, JAN W.; MACCIOLA, ANTHONY
To: KOFAX, INC.
Reel/Frame 056567/0831 →
Continuity (4)
Continuation 16267205 · Feb 4, 2019
Continuation 15214351 · Jul 19, 2016
Provisional Application 62194783 · Jul 20, 2015
Related Publication 20210383150A1 · Dec 9, 2021
References Cited (136)
US 5572654A · Coelho · 1996 [cited by applicant]
US 6246804B1 · Sato et al. · 2001 [cited by applicant]
US 6611613B1 · Kang et al. · 2003 [cited by applicant]
US 6674900B1 · Ma · 2004 [cited by examiner]
US 6819458B1 · Tanaka et al. · 2004 [cited by applicant]
US 6922485B2 · Hirata · 2005 [cited by applicant]
US 6990233B2 · Park et al. · 2006 [cited by applicant]
US 7035477B2 · Cheatle · 2006 [cited by applicant]
US 7043385B2 · Stokes · 2006 [cited by applicant]
US 7119924B2 · Prabhakar et al. · 2006 [cited by applicant]
US 7133571B2 · Cheatle · 2006 [cited by applicant]
US 7274470B2 · Lemelin et al. · 2007 [cited by applicant]
US 7502521B2 · Sun et al. · 2009 [cited by applicant]
US 7596265B2 · Fan et al. · 2009 [cited by applicant]
US 7945573B1 · Barnes et al. · 2011 [cited by applicant]
US 8027070B2 · Dalrymple · 2011 [cited by applicant]
US 8135261B1 · Black · 2012 [cited by applicant]
US 8275190B2 · Sakai et al. · 2012 [cited by applicant]
US 8531734B2 · Mestha et al. · 2013 [cited by applicant]
US 8549001B1 · Minevski et al. · 2013 [cited by applicant]
US 8644600B2 · Yang et al. · 2014 [cited by applicant]
US 8872979B2 · Bagga et al. · 2014 [cited by applicant]
US 8897565B1 · Palm et al. · 2014 [cited by applicant]
US 9049407B2 · Finlayson · 2015 [cited by applicant]
US 9117146B2 · Li et al. · 2015 [cited by applicant]
US 9152860B2 · Cervin et al. · 2015 [cited by applicant]
US 9177225B1 · Cordova-Diba et al. · 2015 [cited by applicant]
US 9224070B1 · Sundareswara et al. · 2015 [cited by applicant]
US 9299004B2 · Lin et al. · 2016 [cited by applicant]
US 9443298B2 · Ross et al. · 2016 [cited by applicant]
US 9672510B2 · Roach et al. · 2017 [cited by applicant]
US 10331966B1 · Pribble et al. · 2019 [cited by applicant]
US 11062176B2 · Ma et al. · 2021 [cited by applicant]
US 11087407B2 · Macciola et al. · 2021 [cited by applicant]
US 11302109B2 · Shustorovich et al. · 2022 [cited by applicant]
US 11321772B2 · Amtrup et al. · 2022 [cited by applicant]
US 11481878B2 · Ma et al. · 2022 [cited by applicant]
US 11593585B2 · Ma et al. · 2023 [cited by applicant]
US 11620733B2 · Ma et al. · 2023 [cited by applicant]
US 11640721B2 · Ma et al. · 2023 [cited by applicant]
US 11694456B2 · Ma et al. · 2023 [cited by applicant]
US 11818303B2 · Ma et al. · 2023 [cited by applicant]
US 11967164B2 · Ma et al. · 2024 [cited by applicant]
US 11983944B2 · Ma et al. · 2024 [cited by applicant]
US 20030031461A1 · Takayama · 2003 [cited by applicant]
US 20030220875A1 · Lam et al. · 2003 [cited by applicant]
US 20060017959A1 · Downer et al. · 2006 [cited by applicant]
US 20060043264A1 · Sakurai · 2006 [cited by applicant]
US 20070204001A1 · Hoopes et al. · 2007 [cited by applicant]
US 20080163103A1 · Reponen et al. · 2008 [cited by applicant]
US 20080177764A1 · Kise et al. · 2008 [cited by applicant]
US 20080226196A1 · Kojima · 2008 [cited by applicant]
US 20080278599A1 · Shiohara · 2008 [cited by applicant]
US 20090074291A1 · Iinuma · 2009 [cited by examiner]
US 20090262200A1 · Takabatake et al. · 2009 [cited by applicant]
US 20100061629A1 · Ma · 2010 [cited by examiner]
US 20100119113A1 · Kuleschow · 2010 [cited by examiner]
US 20110175984A1 · Tolstaya et al. · 2011 [cited by applicant]
US 20110212717A1 · Rhoads et al. · 2011 [cited by applicant]
US 20110240740A1 · Li et al. · 2011 [cited by applicant]
US 20120011142A1 · Baheti et al. · 2012 [cited by applicant]
US 20120185530A1 · Reza · 2012 [cited by applicant]
US 20120281077A1 · Canero Morales et al. · 2012 [cited by applicant]
US 20130004017A1 · Medasani et al. · 2013 [cited by applicant]
US 20130129248A1 · Doublet et al. · 2013 [cited by applicant]
US 20130155474A1 · Roach et al. · 2013 [cited by applicant]
US 20130243321A1 · Shimazaki · 2013 [cited by examiner]
US 20140086488A1 · Moteki et al. · 2014 [cited by applicant]
US 20140204204A1 · Sumiyoshi et al. · 2014 [cited by applicant]
US 20140270536A1 · Amtrup · 2014 [cited by examiner]
US 20150154742A1 · Bocharov · 2015 [cited by examiner]
US 20150371086A1 · Wang et al. · 2015 [cited by applicant]
US 20160182805A1 · Emmett et al. · 2016 [cited by applicant]
US 20170238011A1 · Pettersson et al. · 2017 [cited by applicant]
US 20170372156A1 · Kalenkov et al. · 2017 [cited by applicant]
US 20180131815A1 · Spivakovsky et al. · 2018 [cited by applicant]
US 20190258895A1 · Sacheti et al. · 2019 [cited by applicant]
US 20190377970A1 · Huber, Jr. et al. · 2019 [cited by applicant]
US 20210383163A1 · Ma et al. · 2021 [cited by applicant]
US 20230206664A1 · Ma et al. · 2023 [cited by applicant]
US 20240048658A1 · Ma et al. · 2024 [cited by applicant]
WO 0165486A1 · 2001 [cited by applicant]
WO 2011113026A2 · 2011 [cited by applicant]
WO 2017173368A1 · 2017 [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/824,518, dated Jun. 1, 2021. [cited by applicant]
Supplemental Notice of Allowance from U.S. Appl. No. 16/206,912, dated May 28, 2021. [cited by applicant]
Ma et al., U.S. Appl. No. 17/348,617, filed Jun. 15, 2021. [cited by applicant]
Examination Report from European Application No. 14 861 942.2, dated Jul. 15, 2022. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/005,171, dated Oct. 21, 2022. [cited by applicant]
Final Office Action from U.S. Appl. No. 15/396,306, dated Oct. 28, 2022. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/005,147, dated Jan. 25, 2023. [cited by applicant]
Corrected Notice of Allowance from U.S. Appl. No. 17/005,147, dated Mar. 2, 2023. [cited by applicant]
Ma et al., U.S. Appl. No. 18/114,876, filed Feb. 27, 2023. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/997,800, dated Sep. 7, 2022. [cited by applicant]
Wagner et al., “Pose Tracking from Natural Features on Mobile Phones,” IEEE International Symposium on Mixed and Augmented Reality, Sep. 2008, pp. 125-134. [cited by applicant]
Dubrofsky, E., “Homography Estimation,” University of British Columbia, Mar. 2009, 32 pages. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/348,617, dated Sep. 21, 2022. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/005,147, dated Sep. 9, 2022. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/006,650, dated Sep. 7, 2022. [cited by applicant]
Supplemental Notice of Allowance from U.S. Appl. No. 16/824,518, dated Jul. 15, 2021. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 15/394,731, dated Aug. 16, 2021. [cited by applicant]
Phillips, P., “Support Vector Machines Applied to Face Recognition,” Advances in Neural Information Processing Systems, 1998, pp. 803-809, retrieved from https://papers.nips.cc/paper/1998/file/a2cc63e065705fe938a4dda490… [cited by applicant]
Kung et al., “Biometric Authentication: A Machine Learning Approach,” Matlab Projects, Chapter 1, 2004, 13 pages. [cited by applicant]
Zhao et al., “Face Recognition: A Literature Survey,” ACM Computing Surveys, vol. 35, No. 4, Dec. 2003, pp. 399-458. [cited by applicant]
Turk et al., “Face Recognition Using Eigenfaces,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1991, pp. 586-591. [cited by applicant]
Joachims, T., “Transductive Inference for Text Classification Using Support Vector Machines,” International Conference on Machine Learning, 1999, 10 pages, retrieved from https://www.cs.cornell.edu/people/tj/publication… [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 15/396,306, dated Feb. 18, 2022. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/997,800, dated Jun. 27, 2022. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/006,636, dated Aug. 4, 2022. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/569,247, dated Dec. 10, 2021. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 15/394,731, dated Dec. 27, 2021. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 16/569,247, dated Aug. 31, 2021. [cited by applicant]
Patent Board Decision on Appeal from U.S. Appl. No. 15/396,306, dated Oct. 29, 2021. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 16/997,800, filed Nov. 9, 2021. [cited by applicant]
Kim et al., “Document Capturing Method with a Camera using Robust Feature Points Detection,” International Conference on Digital Image Computing: Techniques and Applications, IEEE, 2011, pp. 678-682. [cited by applicant]
Bulatov et al., “Smart IDReader: Document Recognition in Video Stream,” 14th IAPR International Conference on Document Analysis and Recognition, 2017, pp. 39-44. [cited by applicant]
Liang et al., “Camera-based analysis of text and documents: a survey,” Digital Object Identifier, International Journal on Document Analysis and Recognition, 2005, pp. 84-104. [cited by applicant]
Supplemental Notice of Allowance from U.S. Appl. No. 15/394,731, dated Mar. 30, 2022. [cited by applicant]
Final Office Action from U.S. Appl. No. 16/997,800, dated Apr. 11, 2022. [cited by applicant]
Corrected Notice of Allowance from U.S. Appl. No. 16/569,247, dated Mar. 16, 2022. [cited by applicant]
Examination Report from European Application No. 10741580.4, dated Feb. 28, 2022. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/006,636, dated Nov. 15, 2022. [cited by applicant]
Supplemental Notice of Allowance from U.S. Appl. No. 17/006,636, dated Nov. 25, 2022. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/348,617, dated Jan. 18, 2023. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/006,650, dated Dec. 23, 2022. [cited by applicant]
Ma et al., U.S. Appl. No. 18/134,473, filed Apr. 13, 2023. [cited by applicant]
Final Office Action from U.S. Appl. No. 17/005,171, dated May 3, 2023. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/348,617, dated Apr. 21, 2023. [cited by applicant]
Supplemental Notice of Allowance from U.S. Appl. No. 17/348,617, dated May 16, 2023. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/005,171, dated Jul. 6, 2023. [cited by applicant]
Ma et al., U.S. Appl. No. 18/377,721, filed Oct. 6, 2023. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 18/114,876, dated Sep. 19, 2023. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 18/134,473, dated Nov. 2, 2023. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 18/114,876, dated Jan. 19, 2024. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 18/134,473, dated Feb. 14, 2024. [cited by applicant]
Supplemental Notice of Allowance from U.S. Appl. No. 18/134,473, dated Mar. 11, 2024. [cited by applicant]