IP Library › Granted Patent US 12,229,805
Granted Patent B2
US 12,229,805 · App. 17/566,135 · Granted Feb 18, 2025

Methods, systems, articles of manufacture, and apparatus for processing an image using visual and textual information

Inventors: Javier Martínez Cebrián (Madrid, ES); Roberto Arroyo (Madrid, ES); David Jiménez (Guadalajara, ES)
Assignee: Nielsen Consumer LLC
G06Q30/0276G06T7/33G06V10/70G06V30/10G06T2207/20081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,229,805
App. No.
17/566,135
Granted
Feb 18, 2025
Kind
B2
Abstract

Methods, apparatus, systems, and articles of manufacture are disclosed for processing an image using visual and textual information. An example apparatus includes at least one memory, instructions in the apparatus, and processor circuitry to execute the instructions to detect regions of interest corresponding to a product promotion of an input digital leaflet, extract textual features from the product promotion by applying an optical character recognition (OCR) algorithm to the product promotion and associating output text data with corresponding ones of the regions of interest, determine a search attribute corresponding to the product promotion, generate a first dataset of candidate products corresponding to the product in the product promotion by comparing the search attribute against a second dataset of products, and select a product from the first dataset of candidate products to associate with the product promotion, the product selected based on a match determination.

Claims (41)

1. An apparatus comprising:

interface circuitry;

machine-readable instructions; and

at least one processor circuit to be programmed by the machine-readable instructions to:

determine a geographic area associated with an input digital leaflet based on metadata extracted from the input digital leaflet;

detect regions of the input digital leaflet corresponding to a product promotion, the regions based on the geographic area;

extract textual features from the regions corresponding to the product promotion by applying an optical character recognition (OCR) algorithm to the product promotion and associating output text data with respective ones of the regions;

determine a search attribute corresponding to a product represented in the product promotion, the search attribute based on the geographic area;

generate a first dataset of candidate products corresponding to the product in the product promotion by comparing the search attribute against a second dataset of products, the second dataset specific to the geographic area; and

select a candidate product from the first dataset of candidate products to associate with the product promotion, the candidate product selected based on a match determination.

2. The apparatus of claim 1 , wherein detecting the regions of the input digital leaflet includes classifying the regions of the input digital leaflet, and one or more of the at least one processor circuit is to detect the regions of the input digital leaflet by applying a trained model based on a region-based convolutional neural network (R-CNN) architecture to the product promotion.

3. The apparatus of claim 1 , wherein the regions of the input digital leaflet include a whole promotion region, a product image region, a description region, and a price promotion region.

4. The apparatus of claim 1 , wherein the search attribute is a categorical value including at least one of product brand, product category, or product class.

5. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to determine a fact attribute corresponding to the product promotion and associate the fact attribute with the product promotion.

6. The apparatus of claim 5 , wherein at least one of the search attribute or target attribute are determined by applying a natural language processing (NLP) based model to the extracted textual features.

7. The apparatus of claim 5 , wherein the fact attribute is a categorical attribute including at least one of promotion type, reduction type, price, quantity, or discount.

8. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to determine a match value or a mismatch value for ones of the first dataset of candidate products, the match value or the mismatch value to include a calculated confidence score.

9. The apparatus of claim 8 , wherein one or more of the at least one processor circuit is to rank the ones of the first dataset of candidate products that received the match value, the ranking based on the respective confidence score.

10. The apparatus of claim 8 , wherein one or more of the at least one processor circuit is to select the candidate product from the first dataset corresponding to the highest ranked candidate product of the ones of the first dataset of candidate products that received the match value.

11. At least one non-transitory computer-readable medium comprising computer-readable instructions to cause at least one processor circuit to at least:

determine a geographic region associated with an input digital leaflet based on metadata obtained from the input digital leaflet;

generate bounding boxes corresponding to a product promotion, the bounding boxes based on the geographic region;

extract textual features from the bounding boxes corresponding to the product promotion by applying an optical character recognition (OCR) algorithm to the product promotion and linking output textual data with respective ones of the bounding boxes;

identify a search attribute corresponding to a product represented in the product promotion, the search attribute based on the geographic region;

evaluate the search attribute against a second dataset of products to generate a first dataset of candidate products corresponding to the product in the product promotion, the second dataset of products based on the geographic region; and

select a candidate product from the first dataset of candidate products to associate with the product promotion, the candidate product selected based on a match assessment.

12. The at least one non-transitory computer-readable medium of claim 11 , wherein identifying the bounding boxes of the input digital leaflet further includes categorizing the bounding boxes of the input digital leaflet, the computer-readable instructions to cause one or more of the at least one processor circuit to identify the bounding boxes of the input digital leaflet by applying a trained region-based convolution neural network (R-CNN)-based model to the product promotion.

13. The at least one non-transitory computer-readable medium of claim 11 , where the computer-readable instructions cause one or more of the at least one processor circuit to determine a fact attribute corresponding to the product promotion and associate the fact attribute with the product promotion.

14. The at least one non-transitory computer-readable medium of claim 13 , wherein the computer-readable instructions cause one or more of the at least one processor circuit to determine at least one of the search attribute or target attribute by applying a natural language processing (NLP) based model to the extracted textual features.

15. The at least one non-transitory computer-readable medium of claim 11 , wherein the computer-readable instructions cause one or more of the at least one processor circuit to determine a match or a mismatch value for ones of the first dataset of candidate products, the match or mismatch value to include a calculated confidence score.

16. The at least one non-transitory computer-readable medium of claim 11 , wherein the computer-readable instructions cause one or more of the at least one processor circuit to rank the ones of the first dataset of candidate products that received the match or mismatch value, the ranking based on the respective confidence scores.

17. The at least one non-transitory computer-readable medium of claim 11 , wherein the computer-readable instructions cause one or more of the at least one processor circuit to select the candidate product from the first dataset corresponding to the highest ranked candidate product of the ones of the first dataset of candidate products that received the match or mismatch value.

18. A method comprising:

determining, by executing an instruction with at least one processor circuit, a country associated with an input digital leaflet based on metadata extracted from the input digital leaflet;

detecting, by executing an instruction with one or more of the at least one processor circuit, regions of the input digital leaflet corresponding to a product promotion, the regions based on the country;

extracting, by executing instructions with one or more of the at least one processor circuit, textual features from the regions corresponding to the product promotion by applying an optical character recognition (OCR) algorithm to the product promotion and associating output text data with respective ones of the regions;

determining, by executing an instruction with one or more of the at least one processor circuit, a search attribute corresponding to a product represented in the product promotion, the search attribute based on the country;

generating, by executing an instruction with one or more of the at least one processor circuit, a first dataset of candidate products corresponding to the product in the product promotion by comparing the search attribute against a second dataset of products, the second dataset specific to the country; and

selecting a candidate product from the first dataset of candidate products to associate with the product promotion, the candidate product selected based on a match determination.

19. The method of claim 18 , wherein one or more of the at least one processor circuit is to determine a match or a mismatch value for ones of the first dataset of candidate products, the match or mismatch value including a confidence score.

20. The method of claim 19 , wherein one or more of the at least one processor circuit is to rank the ones of the first dataset of candidate products that received the match or mismatch value, the ranking based on the respective confidence scores.

Assignments (3)
INTELLECTUAL PROPERTY SECURITY AGREEMENT Recorded Jun 20, 2024
From: NIELSEN CONSUMER LLC
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AND COLLATERAL AGENT
Reel/Frame 067792/0978 →
INTELLECTUAL PROPERTY SECURITY AGREEMENT Recorded Dec 16, 2022
From: NIELSEN CONSUMER LLC
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 062142/0346 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 29, 2022
From: CEBRIÁN, JAVIER MARTÍNEZ; ARROYO, ROBERTO; JIMÉNEZ, DAVID
To: NIELSEN CONSUMER LLC
Reel/Frame 059770/0491 →
Continuity (1)
Related Publication 20230214899A1 · Jul 6, 2023
References Cited (360)
US 3323135A · Miller · 1967 [cited by applicant]
US 5410611A · Huttenlocher et al. · 1995 [cited by applicant]
US 5606690A · Hunter et al. · 1997 [cited by applicant]
US 7454063B1 · Kneisl et al. · 2008 [cited by applicant]
US 7792709B1 · Trandal et al. · 2010 [cited by applicant]
US 8285047B2 · Nagarajan et al. · 2012 [cited by applicant]
US 8494281B2 · Nagarajan · 2013 [cited by applicant]
US 8787695B2 · Wu et al. · 2014 [cited by applicant]
US 8792141B2 · Moore et al. · 2014 [cited by applicant]
US 8983170B2 · Nepomniachtchi et al. · 2015 [cited by applicant]
US 9014432B2 · Fan et al. · 2015 [cited by applicant]
US 9158744B2 · Rao et al. · 2015 [cited by applicant]
US 9239952B2 · Hsu et al. · 2016 [cited by applicant]
US 9262686B1 · Singer · 2016 [cited by applicant]
US 9290022B2 · Makabe · 2016 [cited by applicant]
US 9298685B2 · Barrus · 2016 [cited by applicant]
US 9298979B2 · Nepomniachtchi et al. · 2016 [cited by applicant]
US 9323135B1 · Veloso · 2016 [cited by applicant]
US 9324073B2 · Nepomniachtchi et al. · 2016 [cited by applicant]
US 9384389B1 · Sankaranarayanan et al. · 2016 [cited by applicant]
US 9384839B2 · Avila · 2016 [cited by applicant]
US 9396540B1 · Sampson · 2016 [cited by applicant]
US 9684842B2 · Deng · 2017 [cited by applicant]
US 9710702B2 · Nepomniachtchi et al. · 2017 [cited by applicant]
US 9747504B2 · Ma et al. · 2017 [cited by applicant]
US 9760786B2 · Sahagun · 2017 [cited by applicant]
US 9824270B1 · Mao · 2017 [cited by applicant]
US 9875385B1 · Humphreys · 2018 [cited by applicant]
US 10026061B2 · Cheek · 2018 [cited by applicant]
US 10032072B1 · Tran et al. · 2018 [cited by applicant]
US 10157425B2 · Chelst et al. · 2018 [cited by applicant]
US 10235585B2 · Deng · 2019 [cited by applicant]
US 10242285B2 · Thrasher et al. · 2019 [cited by applicant]
US 10395772B1 · Lucas · 2019 [cited by applicant]
US 10679283B1 · Pesce · 2020 [cited by applicant]
US 11257049B1 · Durazo Almeida · 2022 [cited by applicant]
US 11321956B1 · Geng · 2022 [cited by applicant]
US 11410446B2 · Shanmuganathan · 2022 [cited by applicant]
US 11414053B2 · Tanaami et al. · 2022 [cited by applicant]
US 11468491B2 · Dalal · 2022 [cited by applicant]
US 11476981B2 · Wei et al. · 2022 [cited by applicant]
US 11562557B2 · Miginnis et al. · 2023 [cited by applicant]
US 11587148B2 · Elder · 2023 [cited by applicant]
US 11593552B2 · Sarkar · 2023 [cited by applicant]
US 11609956B2 · Jain · 2023 [cited by applicant]
US 11625930B2 · Rodriguez · 2023 [cited by applicant]
US 11810383B2 · Patel et al. · 2023 [cited by applicant]
US 11842035B2 · Jahjah et al. · 2023 [cited by applicant]
US 20020037097A1 · Hoyos et al. · 2002 [cited by applicant]
US 20030185448A1 · Seeger · 2003 [cited by applicant]
US 20060232619A1 · Otsuka et al. · 2006 [cited by applicant]
US 20070041642A1 · Romanoff et al. · 2007 [cited by applicant]
US 20080205759A1 · Zandifar et al. · 2008 [cited by applicant]
US 20090164422A1 · Pacella · 2009 [cited by applicant]
US 20100306080A1 · Trandal et al. · 2010 [cited by applicant]
US 20110122443A1 · Otsuka et al. · 2011 [cited by applicant]
US 20110243445A1 · Uzelac · 2011 [cited by applicant]
US 20110289395A1 · Breuel et al. · 2011 [cited by applicant]
US 20110311145A1 · Bern et al. · 2011 [cited by applicant]
US 20120183211A1 · Hsu et al. · 2012 [cited by applicant]
US 20120274953A1 · Makabe · 2012 [cited by applicant]
US 20120330971A1 · Thomas et al. · 2012 [cited by applicant]
US 20130058575A1 · Koo et al. · 2013 [cited by applicant]
US 20130170741A9 · Hsu et al. · 2013 [cited by applicant]
US 20140002868A1 · Landa · 2014 [cited by applicant]
US 20140064618A1 · Janssen, Jr. · 2014 [cited by applicant]
US 20140188647A1 · Argue · 2014 [cited by applicant]
US 20140195891A1 · Venkata Radha Krishna Rao et al. · 2014 [cited by applicant]
US 20150039479A1 · Gotanda · 2015 [cited by applicant]
US 20150127428A1 · Gharachorloo · 2015 [cited by applicant]
US 20150169951A1 · Khintsitskiy et al. · 2015 [cited by applicant]
US 20150254778A1 · Kmak · 2015 [cited by applicant]
US 20150317642A1 · Argue · 2015 [cited by applicant]
US 20150363792A1 · Arini · 2015 [cited by applicant]
US 20150363822A1 · Rowe · 2015 [cited by applicant]
US 20160005189A1 · Gray · 2016 [cited by applicant]
US 20160034863A1 · Ross · 2016 [cited by applicant]
US 20160063469A1 · Etzion · 2016 [cited by applicant]
US 20160125383A1 · Chan et al. · 2016 [cited by applicant]
US 20160171585A1 · Singh · 2016 [cited by applicant]
US 20160203625A1 · Khan · 2016 [cited by applicant]
US 20160210507A1 · Abdollahian · 2016 [cited by applicant]
US 20160234431A1 · Kraft · 2016 [cited by applicant]
US 20160307059A1 · Chaudhury · 2016 [cited by applicant]
US 20160342863A1 · Kwon et al. · 2016 [cited by applicant]
US 20170293819A1 · Deng · 2017 [cited by applicant]
US 20180005345A1 · Apodaca · 2018 [cited by applicant]
US 20180053045A1 · Lorenzini · 2018 [cited by applicant]
US 20180060302A1 · Liang et al. · 2018 [cited by applicant]
US 20180317116A1 · Komissarov et al. · 2018 [cited by applicant]
US 20180336509A1 · Guttmann · 2018 [cited by applicant]
US 20190026803A1 · De Guzman · 2019 [cited by examiner]
US 20190050639A1 · Ast · 2019 [cited by applicant]
US 20190080207A1 · Chang · 2019 [cited by applicant]
US 20190171900A1 · Thrasher · 2019 [cited by applicant]
US 20190244020A1 · Yoshino · 2019 [cited by applicant]
US 20190272360A1 · Kursun · 2019 [cited by applicant]
US 20190325211A1 · Ordonez et al. · 2019 [cited by applicant]
US 20190332662A1 · Middendorf et al. · 2019 [cited by applicant]
US 20190354818A1 · Reisswig · 2019 [cited by applicant]
US 20200097718A1 · Schafer · 2020 [cited by applicant]
US 20200142856A1 · Neelamana · 2020 [cited by applicant]
US 20200151444A1 · Price et al. · 2020 [cited by applicant]
US 20200151902A1 · Almazán · 2020 [cited by examiner]
US 20200175267A1 · Schafer et al. · 2020 [cited by applicant]
US 20200249803A1 · Sobel · 2020 [cited by applicant]
US 20200364451A1 · Ammar et al. · 2020 [cited by applicant]
US 20200401798A1 · Foncubierta Rodriguez · 2020 [cited by applicant]
US 20200410231A1 · Chua · 2020 [cited by applicant]
US 20210004880A1 · Benkreira et al. · 2021 [cited by applicant]
US 20210019287A1 · Prasad · 2021 [cited by applicant]
US 20210034856A1 · Torres · 2021 [cited by applicant]
US 20210090694A1 · Colley · 2021 [cited by applicant]
US 20210117665A1 · Simantov · 2021 [cited by applicant]
US 20210117668A1 · Zhong · 2021 [cited by applicant]
US 20210142092A1 · Zhao et al. · 2021 [cited by applicant]
US 20210149926A1 · Komninos · 2021 [cited by applicant]
US 20210158038A1 · Shanmuganathan et al. · 2021 [cited by applicant]
US 20210216765A1 · Xu · 2021 [cited by applicant]
US 20210248420A1 · Zhong · 2021 [cited by applicant]
US 20210295101A1 · Tang · 2021 [cited by applicant]
US 20210319217A1 · Wang · 2021 [cited by applicant]
US 20210334737A1 · Balaji · 2021 [cited by applicant]
US 20210343030A1 · Sagonas · 2021 [cited by applicant]
US 20210357710A1 · Zhang · 2021 [cited by applicant]
US 20210406533A1 · Arroyo et al. · 2021 [cited by applicant]
US 20220004756A1 · Jennings · 2022 [cited by applicant]
US 20220114821A1 · Arroyo et al. · 2022 [cited by applicant]
US 20220180113A1 · Patel · 2022 [cited by applicant]
US 20220189190A1 · Arroyo et al. · 2022 [cited by applicant]
US 20220198185A1 · Prebble · 2022 [cited by applicant]
US 20220350946A1 · Hjerrild · 2022 [cited by applicant]
US 20220383651A1 · Shanmuganathan · 2022 [cited by applicant]
US 20220397809A1 · Talpade · 2022 [cited by applicant]
US 20220414630A1 · Yebes Torres · 2022 [cited by applicant]
US 20230004748A1 · Rodriguez · 2023 [cited by applicant]
US 20230005286A1 · Yebes Torres · 2023 [cited by applicant]
US 20230008198A1 · Gadde · 2023 [cited by applicant]
US 20230057687A1 · Mantri · 2023 [cited by applicant]
US 20230196806A1 · Ramalingam · 2023 [cited by applicant]
US 20230214899A1 · Martínez Cebrián et al. · 2023 [cited by applicant]
US 20230230408A1 · Arroyo et al. · 2023 [cited by applicant]
US 20230394859A1 · Montero et al. · 2023 [cited by applicant]
US 20240013562A1 · Montero · 2024 [cited by applicant]
CA 2957433C · 2020 [cited by examiner]
CN 103123685 · 2013 [cited by applicant]
CN 104866849 · 2015 [cited by applicant]
CN 108229397 · 2018 [cited by applicant]
CN 108829397A · 2018 [cited by applicant]
CN 109389124 · 2019 [cited by applicant]
CN 112446351 · 2021 [cited by applicant]
CN 112560862 · 2021 [cited by applicant]
DE 202013005144 · 2013 [cited by applicant]
GB 108829397 · 2020 [cited by applicant]
GB 2595412A · 2021 [cited by applicant]
JP H0749529 · 1995 [cited by applicant]
JP 2008021850A · 2008 [cited by applicant]
JP 200821850 · 2008 [cited by applicant]
JP 2008210850A · 2008 [cited by applicant]
JP 2008211850A · 2008 [cited by applicant]
JP 2019139737 · 2019 [cited by applicant]
JP 7049529 · 2022 [cited by applicant]
KR 101831204 · 2018 [cited by applicant]
WO 2013044145 · 2013 [cited by applicant]
WO 2018054326A1 · 2018 [cited by applicant]
WO 2018201423A1 · 2018 [cited by applicant]
WO 2020194004 · 2020 [cited by applicant]
WO 2022006295 · 2022 [cited by applicant]
WO 2022123199 · 2022 [cited by applicant]
Arroyo et al., Multi-label classification of promotions in digital leaflets using textual and visual information, arXiv:2010.03331v1 (Year: Oct. 7, 2020). [cited by examiner]
Visich, Bar Codes and Their Applications, RS codes—Proc. of the third (Year: 1999). [cited by examiner]
United Kingdom Patent Office, “Examination Report under section 18(3),” issued in connection with GB Application No. 2112299.9, dated Feb. 17, 2023, 2 pages. [cited by applicant]
United States and Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/883,309, dated Jan. 20, 2023, 14 pages. [cited by applicant]
European Patent Office, “Communication Pursuant to Rule 69 EPC,” issued Jan. 23, 2023 in connection with European Patent Application No. 22184405.3, 2 pages. [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability,” issued in connection with Application No. PCT/US2021/039931, issued Jan. 12, 2023, 6 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 17/075,675, issued Mar. 7, 2023, 11 pages. [cited by applicant]
European Patent Office, “Communication pursuant to Rule 69 EPC,” issued in connection with Application No. 22180113.7, dated Jan. 10, 2023, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/345,940, issued Mar. 16, 2023, 13 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 17/379,280, dated May 5, 2023, 17 pages. [cited by applicant]
Canadian Patent Office, “Office Action,” issued in connection with Application No. 3,124,868, dated Nov. 10, 2022, 4 pages. [cited by applicant]
Huang et al., “Mask R-CNN with Pyramid Attention Network for Scene Text Detection”, arXiv:1811.09058v1, pp. 1-9, https://arxiv.org/abs/1811.09058, Nov. 22, 2018, 9 pages. [cited by applicant]
Feng et al., “Computer vision algorithms and hardware implementations: A survey”, Integration: the VLSI Journal, vol. 69, pp. 309-320, https://www.sciencedirect.com/science/article/pii/S0167926019301762, accepted Jul. 2… [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/075,675, issued Jun. 26, 2023, 8 pages. [cited by applicant]
Google, “Detect Text in Images,” Mar. 29, 2021, 20 pages. Retrieved from http://cloud.google.com/vision/docs/ocr. [cited by applicant]
Qasim et al., “Rethinking Table Recognition using Graph Neural Networks,” In International Conference on Document Analysis and Recognition (ICDAR), Jul. 3, 2019, 6 pages. [cited by applicant]
Nshuti, “Mobile Scanner and OCR (A First Step Towards Receipt to Spreadsheet),” 2015, 3 pages. [cited by applicant]
O'Gorman et al., “Document Image Analysis,” IEEE Computer Society Executive Briefings, 2009, 125 pages. [cited by applicant]
International Searching Authority, “Search Report and Written Opinion,” issued in connection with Application No. PCT/US2021/039931, dated Nov. 4, 2021, 7 pages. [cited by applicant]
Genereux et al., “NLP Challenges in Dealing with OCR-ed Documents of Derogated Quality,” Workshop on Replicability and Reproducibility in Natural Language Processing: adaptive methods, resources and software at IJCAI 20… [cited by applicant]
Govindan et al., “Character Recognition—A Review,” Pattern Recognition, vol. 23, No. 7, pp. 671-683, 1990, 13 pages. [cited by applicant]
Lecun et al., “Deep Learning,” Nature, vol. 521, pp. 436-444, May 28, 2015, 9 pages. [cited by applicant]
Kim et al., “Character-Aware Neural Language Models,” Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI'16), pp. 2741-2749, 2016, 9 pages. [cited by applicant]
Wikipedia, “Precision & Recall,” Dec. 17, 2018 revision, 8 pages. [cited by applicant]
Hui, “mAP (mean Average Precision) for Object Detection,” Mar. 6, 2018, 2 pages. Retrieved from [https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173] on May 11, 2020, 2 pages. [cited by applicant]
Artificial Intelligence & Image Analysis, “Historic Document Conversion,” Industry Paper, accessed on Jan. 30, 2019, 4 pages. [cited by applicant]
Artificial Intelligence & Image Analysis, “Intelligent Automation Eliminates Manual Data Entry From Complex Documents,” White Paper, accessed on Jan. 30, 2019, 3 pages. [cited by applicant]
Vogel et al., “Parallel Implementations of Word Alignment Tool,” Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pp. 49-57, Jun. 2008, 10 pages. [cited by applicant]
International Searching Authority, “International Search Report,” mailed in connection with International Patent Application No. PCT/US2020/061269, on Mar. 11, 2021, 3 pages. [cited by applicant]
International Searching Authority, “Written Opinion,” mailed in connection with International Patent Application No. PCT/US2020/061269, on Mar. 11, 2021, 4 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 16/692,797, issued Mar. 16, 2021, 12 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 16/692,797, issued Oct. 27, 2021, 14 pages. [cited by applicant]
International Searching Authority, “International Search Report, ” mailed in connection with International Patent Application No. PCT/IB2019/000299, on Dec. 23, 2019, 3 pages. [cited by applicant]
International Searching Authority, “Written Opinion,” mailed in connection with International Patent Application No. PCT/IB2019/000299, on Dec. 23, 2019, 4 pages. [cited by applicant]
Bartz et al., “STN-OCT: A Single Neural Network for Text Detection and Text Recognition,” Computer Vision and Pattern Recognition, Jul. 27, 2017, 9 pages. [cited by applicant]
Ozhiganov, “Deep Dive Into OCR for Receipt Recognition,” DZone, Jun. 21, 2017, 18 pages. [cited by applicant]
Akbik et al., “Contextual String Embeddings for Sequence Labeling,” In Proceedings of the 27th International Conference on Computational Linguistics (COLING), 2018, 12 pages. [cited by applicant]
Bojanowski et al., “Enriching Word Vectors with Subword Information,” In Journal Transactions of the Association for Computational Linguistics, 2017, vol. 5, pp. 135-146, 12 pages. [cited by applicant]
Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Jun. 24, 2019,… [cited by applicant]
DeepDive, “Distant Supervision” 2021, 2 pages. [available online on Stanford University website, http://deepdive.stanford.edu/distant_supervision]. [cited by applicant]
Joulin et al., “Bag of Tricks for Efficient Text Classification,” In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Aug. 9, 2016, 5 pages. [cited by applicant]
Krizhevsky et al., “ImageNet Classification with Deep Convolutional Neural Networks,” In International Conference on Neural Information Processing Systems (NIPS), 2012, 9 pages. [cited by applicant]
Konda et al., “ Magellan: Toward Building Entity Matching Management Systems Over Data Science Stacks,” Proceedings of the VLDB Endowment, vol. 9, No. 13, pp. 1581-1584, 2016, 4 pages. [cited by applicant]
Levenshtein, “Binary Codes Capable of Correcting Deletions, Insertions, and Reversals,” Soviet Physics—Doklady, Cybernetics and Control Theory, pp. 707-710, vol. 10, No. 8, Feb. 1966, 4 pages. [cited by applicant]
Mudgal et al., “Deep learning for entity matching: A design space exploration,” In Proceedings of the 2018 International Conference on Management of Data, 2018, Houston, TX, 16 pages. [cited by applicant]
Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection,” In Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 10 pages. [cited by applicant]
Ren et al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” In International Conference on Neural Information Processing Systems (NIPS), pp. 91-99, 2015, 14 pages. [cited by applicant]
Smith et al., “Identification of Common Molecular Subsequences,” Reprinted Journal of Molecular Biology, Academic Press Inc. (London) Ltd., pp. 195-197, 1981, 4 pages. [cited by applicant]
Github, “Tesseract OCR” Tesseract Repository on GitHub, 2020, 4 pages. [available online, https://github.com/tesseract-ocr/]. [cited by applicant]
Vaswani et al., “Attention Is All You Need,” 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, 2017, 11 pages. [cited by applicant]
United States Patent and Trademark Office, “Advisory Action,” issued in connection with U.S. Appl. No. 16/692,797, dated Feb. 16, 2022, 4 pages. [cited by applicant]
Oliveira et al., “dhSegment: A generic deep-learning approach for document segmentation,” In 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2019, 6 pages. [cited by applicant]
Ronneberger et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015, 8 pages. [cited by applicant]
Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” International Journal of Computer Vision (IJCV), Jan. 5, 2004, 28 pages. [cited by applicant]
Marinai, “Introduction to Document Analysis and Recognition,” Machine Learning in Document Analysis and Recognition, 2008, 22 pages. [cited by applicant]
Zhong et al., “PubLayNet: largest dataset ever for document layout analysis,” In International Conference on Document Analysis and Recognition (ICDAR), 2019, 8 pages. [cited by applicant]
Follmann et al., “MVTec D2S: Densely Segmented Supermarket Dataset”. In European Conference on Computer Vision (ECCV), 2018, 17 pages. [cited by applicant]
Osindero et al., “Recursive Recurrent Nets with Attention Modeling for OCR in the Wild,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 10 pages. [cited by applicant]
NielsenIQ Brandbank, “Nielsen Brandbank Product Library,” Online Available. Retrieved on Apr. 1, 2022, 5 pages. [retrieved from: https://www.brandbank.com/us/product-library/]. [cited by applicant]
Ray et al., “U-PC: Unsupervised Planogram Compliance,” in European Conference on Computer Vision (ECCV), 2018, 15 pages. [retrieved from: http://openaccess.thecvf.com/content_ECCV_2018/papers/Archan_Ray_U-PC_Unsupervise… [cited by applicant]
Appalaraju et al., “DocFormer: End-to-End Transformer for Document Understanding,” arXiv (CoRR), 2021, 22 pages. [retrieved from: https://arxiv.org/pdf/2106.11539.pdf]. [cited by applicant]
Hong et al., “BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents,” arXiv (CoRR), 2021, 13 pages. [retrieved from: https://arxiv.org/pdf/2108.04539.pdf]. [cited by applicant]
Hwang et al., “Spatial Dependency Parsing for Semi-Structured Document Information Extraction,” in International Joint Conference on Natural Language Processing (IJCNLP), 2021, 14 pages. [retrieved from: https://arxiv.o… [cited by applicant]
Shen et al., “LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis,” in International Conference on Document Analysis and Recognition (ICDAR), 2021, 16 pages. [retrieved from: https://arxiv.or… [cited by applicant]
Wick et al., “Calamari—A High-Performance Tensorflow-based Deep Learning Package for Optical Character Recognition,” Digital Humanities Quarterly, 2020, 12 pages. [retrieved from: https://arxiv.org/ftp/arxiv/papers/1807… [cited by applicant]
Yu et al., “PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks,” in International Conference on Pattern Recognition (ICPR), 2020, 8 pages. [retrieved from: ht… [cited by applicant]
Zacharias et al., “Image Processing Based Scene-Text Detection and Recognition with Tesseract,” arXiv (CoRR), 2020, 6 pages. [retrieved from: https://arxiv.org/pdf/2004.08079.pdf]. [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability” mailed in connection with International Patent Application No. PCT/IB2019/000299, on Sep. 28, 2021, 5 pages. [cited by applicant]
Github, “Doccano tool,” Github.com, downloaded on Apr. 1, 2022, 12 pages. [retrieved from: https://github.com/doccano/doccano]. [cited by applicant]
Github, “FIAT tool—Fast Image Data Annotation Tool, ” Github.com, downloaded on Apr. 1, 2022, 30 pages. [retrieved from: https://github.com/christopher5106/FastAnnotationTool]. [cited by applicant]
Datasetlist, “Annotation tools for building datasets,” Labeling tools—List of labeling tools, Datasetlist.com, updated Dec. 2021, downloaded on Apr. 1, 2022, 12 pages. [retrieved from: https://www.datasetlist.com/tools/… [cited by applicant]
Xu et al., “LayoutLM: Pre-training of Text and Layout for Document Image Understanding,” in International Conference on Knowledge Discovery & Data Mining (SIGKDD), 2020, 9 pages. [retrieved from: https://arxiv.org/pdf/1… [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 16/692,797, issued Apr. 5, 2022, 10 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 16/692,797, issued Apr. 22, 2022, 3 pages. [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability,” mailed in connection with International Patent Application No. PCT/US2020/061269, on May 17, 2022, 5 pages. [cited by applicant]
European Patent Office, “Communication pursuant to Rules 161(2) and 162 EPC,” issued in connection with Application No. 20891012.5, dated Jun. 29, 2022, 3 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Action” issued in U.S. Appl. No. 17/075,675, on Sep. 22, 2022, 12 pages. [cited by applicant]
Huang et al., “LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking,” 30th ACM International Conference on Multimedia, Jul. 2022, 10 pages. [cited by applicant]
Zhang et al., “Multimodal Pre-training Based on Graph Attention Network for Document Understanding.” IEEE Transactions on Multimedia, vol. 25, Oct. 2022, 13 pages. [cited by applicant]
European Patent Office, “Extended European Search Report,” issued in connection with European Patent Application No. 19921870, completed Oct. 3, 2022, 2 pages. [cited by applicant]
International Searching Authority, “International Search Report,” issued in connection with International Patent Application No. PCT/US2022/034570, mailed on Oct. 20, 2022, 3 pages. [cited by applicant]
International Searching Authority, “Written Opinion,” issued in connection with International Patent Application No. PCT/US2022/034570, mailed on Oct. 20, 2022, 5 pages. [cited by applicant]
Kim et al., “OCR-free Document Understanding Transformer,” European Conference on Computer Vision, Oct. 2022, 29 pages. [cited by applicant]
European Patent Office, “Communication pursuant to Rules 70(2) and 70a(2) EPC,” issued in connection with Application No. 19921870.2, dated Nov. 2, 2022, 1 page. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/364,419, dated Nov. 4, 2022, 10 pages. [cited by applicant]
Canadian Patent Office, “Examiner's Report,” issued in connection with Canadian Patent Application No. 3,124,868, mailed on Nov. 10, 2022, 4 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability”, issued in connection with U.S. Appl. No. 17/364,419, dated Nov. 15, 2022, 2 pages. [cited by applicant]
Zhong et al., “Hierarchical Message-Passing Graph Neural Networks,” Data Mining and Knowledge Discovery, Nov. 17, 2022, 29 pages. [cited by applicant]
European Patent Office, “Extended Search Report,” in connection with European Patent Application No. 22180113.7, completed Nov. 14, 2022, 2 pages. [cited by applicant]
European Patent Office, “European Search Report,” issued in connection with European Patent Application No. 22180113.7, Nov. 22, 2022, 25 pages. [cited by applicant]
Dwivedi et al., “Benchmarking Graph Neural Networks,” Journal of Machine Learning Research, Dec. 2022, 49 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action”, issued in connection with U.S. Appl. No. 17/379,280, dated Dec. 2, 2022, 14 pages. [cited by applicant]
European Patent Office, “Extended European Search Report,” issued in connection with European Patent Application No. 22184405.3, dated Dec. 2, 2022, 4 pages. [cited by applicant]
European Patent Office, “European Search Report,” issued in connection with Application No. 22184405.3, completed Nov. 23, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/364,419, dated Jan. 4, 2023, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability”, issued in connection with U.S. Appl. No. 17/364,419, filed Feb. 15, 2023, 2 pages. [cited by applicant]
United States and Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/883,309, dated May 11, 2023, 9 pages. [cited by applicant]
European Patent Office, “Extended European Search Report,” issued in connection with European Patent Application No. 22214553.4, dated May 17, 2023, 9 pages. [cited by applicant]
United States and Patent and Trademark Office, “Advisory Action,” issued in connection with U.S. Appl. No. 17/075,675, dated May 30, 2023, 3 pages. [cited by applicant]
International Searching Authority, “International Search Report,” issued in connection with International Patent Application No. PCT/US2023/011859, mailed on Jun. 1, 2023, 3 pages. [cited by applicant]
International Searching Authority, Written Opinion, issued in connection with International Patent Application No. PCT/US2023/011859, mailed on Jun. 1, 2023, 4 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 18/191,642, dated Feb. 7, 2024, 18 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/345,940, mailed on Jul. 7, 2023, 8 pages. [cited by applicant]
United Kingdom Intellectual Property Office, “Intention to Grant under Section 18(4),” issued in connection with United Kingdom Patent Application No. 2112299.9, dated Jul. 13, 2023, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Advisory Action,” issued in connection with U.S. Appl. No. 17/379,280, mailed on Jul. 18, 2023, 3 pages. [cited by applicant]
Gopal et al., “What is Intelligent Document Processing?” Nano Net Technologies, URL:[https://nanonets.com/blog/intelligent-document-processing/], Jul. 19, 2023, 21 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/345,940, dated Jul. 20, 2023, 3 pages. [cited by applicant]
Canadian Intellectual Property Office, “Examiner's Report,” issued in connection with Canadian Patent Application No. 3,124,868, dated Aug. 10, 2023, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/883,309, mailed on Aug. 17, 2023, 2 Pages. [cited by applicant]
United Kingdom Intellectual Property Office, “Notification of Grant,” issued in connection with United Kingdom Patent Application No. 2112299.9, dated Aug. 29, 2023, 2 pages. [cited by applicant]
Amazon, “Intelligent Document Processing,” Amazon Web Services, https://aws.amazon.com/machine-learning/ml-use-cases/document-processing/fintech/, retrieved on Sep. 8, 2023, 6 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/075,675, mailed on Oct. 10, 2023, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/710,538, dated Oct. 26, 2023, 6 Pages. [cited by applicant]
European Patent Office, “Extended European Search Report,” issued in connection with European Patent Application No. 20891012.5, dated Nov. 17, 2023, 12 pages. [cited by applicant]
European Patent Office, “Communication pursuant to Rules 70(2) and 70a(2) EPC,” issued in connection with European Patent Application No. 20891012.5, dated Dec. 5, 2023, 1 page. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action” issued in connection with U.S. Appl. No. 17/598,792, mailed on Dec. 29, 2023, 17 pages. [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability,” issued in connection with International Patent Application No. PCT/US2022/034570, issued on Jan. 4, 2024, 7 pages. [cited by applicant]
Leicester et al., “Using Scanner Technology to Collect Expenditure Data,” Fiscal Studies, vol. 30, Issue 3-4, 2009, 29 pages. [cited by applicant]
European Patent Office, “Communication pursuant to Article 94(3) EPC,” issued in connection with European Patent Application No. 19 921 870.2-1207, on Apr. 9, 2024, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 18/476,978, dated Apr. 18, 2024, 20 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/710,538, dated Apr. 19, 2024, 8 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/710,660, on May 28, 2024, 9 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/598,792, dated Jun. 17, 2024, 9 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/191,642, dated Jun. 17, 2024, 9 pages. [cited by applicant]
Canadian Intellectual Property Office, “Office Action,” issued in connection with Canadian Patent Application No. 3,182,471, dated May 28, 2024, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/598,792, dated Jul. 3, 2024, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 18/476,978, dated Aug. 14, 2024, 22 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/710,538, dated Aug. 14, 2024, 8 pages. [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability,” issued in connection with International Application No. PCT/US2023/011859, mailed on Aug. 15, 2024, 6 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/598,792, dated Aug. 27, 2024, 9 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/191,642, dated Aug. 28, 2024, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/710,649, dated Sep. 16, 2024, 12 pages. [cited by applicant]
Poulovassilis et al., “A Nested-Graph Model for the Representation and Manipulation of Complex Objects,” ACM Transactions on Information Systems, vol. 12, Issue 1, Jan. 2, 1994, 34 pages. [cited by applicant]
Hochreiter et al., “Long Short-Term Memory,” Neural Computation, Dec. 1997, 46 pages. [cited by applicant]
Ng et al., “On Spectral Clustering: Analysis and an Algorithm,” NIPS'01: Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic, Jan. 2001, 8 pages. [cited by applicant]
Oliveira et al., “A New Method for Text-Line Segmentation for Warped Documents,” International Conference Image Analysis and Recognition, Jun. 21, 2010, 11 pages. [cited by applicant]
Chung et al., “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.” NIPS 2014 Deep Learning and Representation Workshop, Dec. 2014, 9 pages. [cited by applicant]
Kipf et al., “Semi-Supervised Classification with Graph Convolutional Networks,” 5th International Conference on Learning Representations, Feb. 22, 2017, 14 Pages. [cited by applicant]
Velickcvic et al., “Graph Attention Networks,” 2018 International Conference on Learning Representations, Feb. 4, 2018 12 pages. [cited by applicant]
Li et al., “Extracting Figures and Captions from Scientific Publications,” Short Paper, Conference on Information and Knowledge Management 2018 (CIKM18), CIKM'18, Oct. 22-26, 2018, Torino, Italy, 4 pages. [cited by applicant]
Elfwing et al. “Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning,” Neural Networks: Journal of the International Neural Network Society, vol. 107, Nov. 2017, 18 pages. [cited by applicant]
Loshchilov et al., “Decoupled Weight Decay Regularization,” 2019 International Conference on Learning Representations, Published as a conference paper at ICLR 2019, Jan. 4, 2019, 19 pages. [cited by applicant]
Nathancy, “How do Imake masks to set all of image background, except the text, to white?”, stakoverflow.com, https://stackoverflow.com/questions/56465359/how-do-i-make-masks-to-set-all-of-image-background-except-the-tex… [cited by applicant]
Hu et al., “Hierarchical Graph Convolutional Networks for Semi-supervised Node Classification,” 2019 International Joint Conference on Artificial Intelligence, Jun. 2019, 8 pages. [cited by applicant]
Zhong et al., “PubLayNet: largest dataset ever for document layout analysis,” In International Conference on Document Analysis and Recognition (ICDAR), dated Aug. 16, 2019, 8 pages. [cited by applicant]
Jaume et al., “FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents,” International Conference on Document Analysis and Recognition (ICDAR), Oct. 2019, 6 pages. [cited by applicant]
Yadati et al., “HyperGCN: Hypergraph Convolutional Networks for Semi-Supervised Classification,” Proceedings of the 33rd International Conference on Neural Information Processing Systems, Jan. 2019, 18 pages. [cited by applicant]
Carbonell et al., “Named Entity Recognition and Relation Extraction with Graph Neural Networks in Semi Structured Documents,” 2020 International Conference on Pattern Recognition (ICPR), Jan. 10, 2021, 6 pages. [cited by applicant]
Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” International Conference on Learning Representations, Jul. 2019, 13 pages. [cited by applicant]
Dong et al., “HNHN: Hypergraph Networks with Hyperedge Neurons.” ArXiv abs/2006.12278, Jun. 22, 2020, 11 pages. [cited by applicant]
Chen et al., “HGMF: Heterogeneous Graph-Based Fusion for Multimodal Data with Incompleteness,” Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, Aug. 23-27, 2020,… [cited by applicant]
Wang et al., “DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding,” 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), Nov. 16-20, 2020, 11 pag… [cited by applicant]
Zhu et al., “Heterogeneous Mini-Graph Neural Network and Its Application to Fraud Invitation Detection,” 2020 IEEE International Conference on Data Mining (ICDM), Nov. 17, 2020, 9 pages. [cited by applicant]
Bandyopadhyay et al., “Hypergraph Attention Isomorphism Network by Learning Line Graph Expansion,” 2020 IEEE International Conference on Big Data, Dec. 10, 2020, 10 pages. [cited by applicant]
Arroyo et al., “Multi-label classification of promotions in digital leaflets using textual and visual information,” Proceedings of the Workshop on Natural Language Processing in E-Commerce (EComNLP), pp. 11-20, Barcelon… [cited by applicant]
Nguyen Dang et al., “End-to-End Hierarchical Relation Extraction for Generic Form Understanding,” International Conference on Pattern Recognition (ICPR), Jun. 2021, 8 pages. [cited by applicant]
Xu et al., “LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding,” arXiv, Sep. 2021, 10 pages. [cited by applicant]
Ma et al., “Graph Attention Networks with Positional Embeddings,” Pacific-Asia Conference on Knowledge Discovery and Data Mining, Oct. 2021, 13 pages. [cited by applicant]
Chen et al., “TextPolar: irregular scene text detection using polar representation,” International Journal on Document Analysis and Recognition (IJDAR), May 7, 2021, 9 pages. [cited by applicant]
Li et al. “SelfDoc: Self-Supervised Document Representation Learning.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2021, 10 pages. [cited by applicant]
Xu et al., “LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding.” ACL, Jan. 2022 13 pages. [cited by applicant]
Li et al., “StructuralLM: Structural Pre-training for Form Understanding.” 59th Annual Meeting of the Association for Computational Linguistics, May 2021, 10 pages. [cited by applicant]
Huang et al., “UniGNN: a Unified Framework for Graph and Hypergraph Neural Networks.” 30th International Joint Conference on Artificial Intelligence (IJCAI), May 2021, 9 pages. [cited by applicant]
Tang et al., “MatchVIE: Exploiting Match Relevancy between Entities for Visual Information Extraction”, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21), pp. 1039-1045, 2… [cited by applicant]
Qian et al., “A Region-Based Hypergraph Network for Joint Entity-Relation Extraction,” Knowledge-Based Systems. vol. 228, Sep. 2021, 8 pages. [cited by applicant]
Davis et al., “Visual FUDGE: Form Understanding via Dynamic Graph Editing,” International Conference on Document Analysis and Recognition (ICDAR), Jul. 2021, 16 pages. [cited by applicant]
Powalski et al., “Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer,” International Conference on Document Analysis and Recognition, Mar. 2021, 17 pages. [cited by applicant]
Garncarek et al., “LAMBERT: Layout-Aware Language Modeling for Information Extraction,” International Conference on Document Analysis and Recognition (ICDAR), May 2021, 16 pages. [cited by applicant]
Prabhu et al., “MTL-FoUn: A Multi-Task Learning Approach to Form Understanding,” 2021 International Conference on Document Analysis and Recognition (ICDAR), Sep. 5, 2021, 5 pages. [cited by applicant]
Li et al., “StrucTexT: Structured Text Understanding with Multi-Modal Transformers,” ACM International Conference on Multimedia (ACM Multimedia), Nov. 2021, 9 pages. [cited by applicant]
European Patent Office, “Communication pursuant to Rules 161(2) and 162 EPC,” in connection with European Patent Application No. 19921870.2, issued Nov. 5, 2021, 3 pages. [cited by applicant]
Zhang et al., “Entity Relation Extraction as Dependency Parsing in Visually Rich Documents,” Empirical Methods in Natural Language Processing (EMNLP), Nov. 7-11, 2021, 10 pages. [cited by applicant]
Hwang et al., “Cost-Effective End-to-end Information Extraction for Semi-structured Document Images,” Empirical Methods in Natural Language Processing (EMNLP), Aug. 2021, 9 pages. [cited by applicant]
Gu et al., “UniDoc: Unified Pretraining Framework for Document Understanding,” 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Dec. 6, 2021, 12 pages. [cited by applicant]
Park et al. “CORD: A Consolidated Receipt Dataset for Post-OCR Parsing. In Workshop on Document Intelligence,” 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, 4 pages. [cited by applicant]
Wang et al., “LiLT: A Simple Yet Effective Language-Independent Layout Transformer for Structured Document Understanding,” Annual Meeting of the Association for Computational Linguistics (ACL), Feb. 2022, 11 pages. [cited by applicant]
Gu et al, “XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understanding”, in Conference on Computer Vision and Pattern Recognition ( CVPR), 2022, 10 pages. [cited by applicant]
Villota et al., “Text Classification Models for Form Entity Linking,” International Symposium on Distributed Computing and Artificial Intelligence, Dec. 2021 10 pages. [cited by applicant]
Datasetlist, “A tool using OpenCV to annotate images for image classification, optical character reading, . . . ,” Datasetlist.com, dated Jul. 13, 2022, 30 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/345,940, dated Aug. 18, 2022, 8 pages. [cited by applicant]
Crandall et al., “Extraction of special effects caption text events from digital video,” IJDAR, Department of Computer Science and Engineering, The Pennsylvania State University, 202 Pond Laboratory, Universirty Park, P… [cited by applicant]
United States Patent and Trademark Office, Supplemental Notice of Allowability, issued in connection with U.S. Appl. No. 17/710,538, dated Sep. 11, 2024, 3 pages. [cited by applicant]
United States Patent and Trademark Office, “Second Notice of Allowability,” issued in connection with U.S. Appl. No. 18/191,642, dated Sep. 16, 2024, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/710,660, dated Sep. 25, 2024, 9 pages. [cited by applicant]
United States Patent and Trademark Office, “Advisory Action,” issued in connection with U.S. Appl. No. 18/476,978, dated Oct. 7, 2024, 3 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/598,792, dated Oct. 10, 2024, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/710,660, dated Oct. 9, 2024, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/598,792, dated Dec. 5, 2024, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/476,978, dated Dec. 13, 2024, 11 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/822,664, dated Dec. 31, 2024, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/148,947, dated Jan. 8, 2025, 12 pages. [cited by applicant]
Cited By (1)
US 12,646,347