IP Library › Granted Patent US 12,380,270
Granted Patent B2
US 12,380,270 · App. 18/241,782 · Granted Aug 5, 2025

System, method, and computer program for testing the accuracy of software that identifies user interface elements on a webpage

Inventors: Dennis Hou (Melbourne, AU); Bill Vasiliadis (Melbourne, AU); Travis Coulter (Vancouver, CA); Jae Jung (Melbourne, AU); Adam Roth (Sunshine Coast, AU)
Assignee: Rakuten Group, Inc.
G06F40/174G06F16/951G06F16/957G06F16/958G06F40/117G06F40/169G06F11/321
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,380,270
App. No.
18/241,782
Granted
Aug 5, 2025
Kind
B2
Abstract

The present disclosure relates to a system, method, and computer program for testing the accuracy of software that identifies user interface (UI) elements on a webpage (“the auto-identifier software”). The system enables a user to tag UI elements on a webpage with labels. The system creates a normalized specification for the webpage, where the specification includes a mapping of UI elements to normalized labels. The system uses the auto-identifier software to identify UI elements on the webpage. The system evaluates the performance of the auto-identifier software with respect to the webpage using the specification. The system displays diagnostics related to the performance of the auto-identifier software. In certain embodiments, the method is used for testing the accuracy of autofill software on a webform. In certain embodiments, the method is used for testing the accuracy of cart-scraper software on a checkout page.

Claims (32)

1. A method, performed by a computer system, for testing the accuracy of software that identifies UI elements on a webpage (“the auto-identifier software”), the method comprising:

enabling a user to tag UI elements on a webpage with labels, wherein the UI elements are checkout cart fields and wherein the auto-identifier software is designed to automatically scrape checkout cart data from a webpage;

creating and storing a normalized specification for the webpage, wherein the specification is created by a browser extension automatically tagging checkout cart fields with a label and a user correcting or adding one or more labels to their corresponding checkout cart field(s) to create and store a representation of all possible UI elements on the webpage plus relevant metadata in the specification, wherein the relevant metadata does not include metadata related to styling or behavior, and wherein the specification includes a mapping of UI elements to normalized labels;

using the auto-identifier software to identify UI elements on the webpage;

evaluating the performance of the auto-identifier software with respect to the webpage using the stored specification, wherein evaluating the performance of the auto-identifier software comprises determining whether the auto-identifier software scraped the cart data it is tasked to scrape and ignored the cart data it is tasked to ignore; and

displaying diagnostics related to the performance of the auto-identifier software, wherein displaying diagnostics comprises displaying statistics related to the performance of the software for a select UI element across a plurality of webpages.

2. The method of claim 1 , wherein the auto-identifier software is also designed to automatically fill in fields in a webform.

3. The method of claim 2 , wherein the webform is a checkout page.

4. The method of claim 2 , wherein the specification for the webform includes a mapping of fields on the webform to normalized field labels.

5. The method of claim 2 , wherein evaluating the performance of the auto-identifier software comprises determining whether the auto-identifier software correctly identified and correctly filled the fields on the webform that the auto-identifier software is tasked with auto-filling and correctly ignored the fields on the webform that the auto-identifier software is not tasked with auto-filling.

6. The method of claim 2 , wherein displaying diagnostics comprises displaying statistics regarding the performance of the autofill software on the webform.

7. The method of claim 1 , wherein the specification includes a mapping of cart data on the webpage to normalized labels.

8. The method of claim 1 , wherein enabling a user to tag UI elements on a webpage comprising providing a browser extension for a web browser that enables a user to tag UI elements on a webpage.

9. The method of claim 8 , wherein the browser extension includes code that first automatically tags select UI elements with labels and then enables the user to edit the labels and tag additional UI elements.

10. A non-transitory computer-readable medium comprising a computer program, that, when executed by a computer system, enables the computer system to perform the following steps for testing the accuracy of software that identifies UI elements on a webpage (“the auto-identifier software”), the steps comprising:

enabling a user to tag UI elements on a webpage with labels, wherein the UI elements are checkout cart fields and wherein the auto-identifier software is designed to automatically scrape checkout cart data from a webpage;

creating and storing a normalized specification for the webpage, wherein the specification is created by a browser extension automatically tagging checkout cart fields with a label and a user correcting or adding one or more labels to their corresponding checkout cart field(s) to create and store a representation of all possible UI elements on the webpage plus relevant metadata in the specification, wherein the relevant metadata does not include metadata related to styling or behavior, and wherein the specification includes a mapping of UI elements to normalized labels;

using the auto-identifier software to identify UI elements on the webpage;

evaluating the performance of the auto-identifier software with respect to the webpage using the stored specification, wherein evaluating the performance of the auto-identifier software comprises determining whether the auto-identifier software scraped the cart data it is tasked to scrape and ignored the cart data it is tasked to ignore; and

displaying diagnostics related to the performance of the auto-identifier software, wherein displaying diagnostics comprises displaying statistics related to the performance of the software for a select UI element across a plurality of webpages.

11. The non-transitory computer-readable medium of claim 10 , wherein the auto-identifier software is also designed to automatically fill in fields in a webform.

12. The non-transitory computer-readable medium of claim 11 , wherein displaying diagnostics comprises displaying statistics regarding the performance of the autofill software on the webform.

13. A computer system for testing the accuracy of software that identifies UI elements on a webpage (“the auto-identifier software”), the system comprising

one or more processors;

one or more memory units coupled to the one or more processors, wherein the one or more memory units store instructions that, when executed by the one or more processors, cause the system to perform the operations of:

enabling a user to tag UI elements on a webpage with labels, wherein the UI elements are checkout cart fields and wherein the auto-identifier software is designed to automatically scrape checkout cart data from a webpage;

creating and storing a normalized specification for the webpage, wherein the specification is created by a browser extension automatically tagging checkout cart fields with a label and a user correcting or adding one or more labels to their corresponding checkout cart field(s) to create and store a representation of all possible UI elements on the webpage plus relevant metadata in the specification, wherein the relevant metadata does not include metadata related to styling or behavior, and wherein the specification includes a mapping of UI elements to normalized labels;

using the auto-identifier software to identify UI elements on the webpage;

evaluating the performance of the auto-identifier software with respect to the webpage using the stored specification, wherein evaluating the performance of the auto-identifier software comprises determining whether the auto-identifier software scraped the cart data it is tasked to scrape and ignored the cart data it is tasked to ignore; and

displaying diagnostics related to the performance of the auto-identifier software, wherein displaying diagnostics comprises displaying statistics related to the performance of the software for a select UI element across a plurality of webpages.

14. The computer system of claim 13 , wherein the auto-identifier software is also designed to automatically fill in fields in a webform.

15. The computer system of claim 14 , wherein displaying diagnostics comprises displaying statistics regarding the performance of the autofill software on the webform.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 12, 2023
From: HOU, DENNIS; VASILIADIS, BILL; COULTER, TRAVIS; JUNG, JAE; ROTH, ADAM
To: RAKUTEN GROUP, INC.
Reel/Frame 064878/0880 →
Continuity (2)
Continuation 17566233 · Dec 30, 2021
Related Publication 20230419021A1 · Dec 28, 2023
References Cited (92)
US 8112317B1 · Ballaro et al. · 2012 [cited by applicant]
US 10229436B1 · Fan et al. · 2019 [cited by applicant]
US 10311499B1 · Kim et al. · 2019 [cited by applicant]
US 10614381B2 · Hoffman et al. · 2020 [cited by applicant]
US 10698967B2 · Shen et al. · 2020 [cited by applicant]
US 11068921B1 · Engle et al. · 2021 [cited by applicant]
US 11323570B1 · Lamba et al. · 2022 [cited by applicant]
US 20060248442A1 · Rosenstein et al. · 2006 [cited by applicant]
US 20090177959A1 · Chakrabarti et al. · 2009 [cited by applicant]
US 20100191594A1 · White et al. · 2010 [cited by applicant]
US 20110106600A1 · Malik et al. · 2011 [cited by applicant]
US 20110137717A1 · Reuthe et al. · 2011 [cited by applicant]
US 20110153401A1 · Jellema et al. · 2011 [cited by applicant]
US 20110173527A1 · Zunger · 2011 [cited by applicant]
US 20120016862A1 · Rajan · 2012 [cited by applicant]
US 20130014002A1 · Cai et al. · 2013 [cited by applicant]
US 20140032267A1 · Smith et al. · 2014 [cited by applicant]
US 20140074584A1 · Fisher · 2014 [cited by applicant]
US 20140207872A1 · Roy et al. · 2014 [cited by applicant]
US 20140278880A1 · Lemphers et al. · 2014 [cited by applicant]
US 20150106181A1 · Kluth · 2015 [cited by applicant]
US 20160217484A9 · Hudson et al. · 2016 [cited by applicant]
US 20170185894A1 · Volkovs et al. · 2017 [cited by applicant]
US 20170345075A1 · Khandelwal et al. · 2017 [cited by applicant]
US 20180293499A1 · He et al. · 2018 [cited by applicant]
US 20180315069A1 · Kamireddy et al. · 2018 [cited by applicant]
US 20180336279A1 · Ahern et al. · 2018 [cited by applicant]
US 20190385185A1 · Shiffert et al. · 2019 [cited by applicant]
US 20200084023A1 · Bessonov et al. · 2020 [cited by applicant]
US 20200134698A1 · Dabral et al. · 2020 [cited by applicant]
US 20200159647A1 · Puszkiewicz et al. · 2020 [cited by applicant]
US 20200160050A1 · Bhotika et al. · 2020 [cited by applicant]
US 20200210511A1 · Korobov et al. · 2020 [cited by applicant]
US 20200320163A1 · Raiskin · 2020 [cited by applicant]
US 20200357012A1 · Kendall et al. · 2020 [cited by applicant]
US 20210044870A1 · Li et al. · 2021 [cited by applicant]
US 20210073846A1 · Hudson · 2021 [cited by applicant]
US 20210089710A1 · Chernov et al. · 2021 [cited by applicant]
US 20210103975A1 · Walters · 2021 [cited by examiner]
US 20210125262A1 · Corrieri et al. · 2021 [cited by applicant]
US 20210182892A1 · Henderson et al. · 2021 [cited by applicant]
US 20210191990A1 · Shi et al. · 2021 [cited by applicant]
US 20210312519A1 · Sharshevsky et al. · 2021 [cited by applicant]
US 20220222735A1 · Senthil · 2022 [cited by applicant]
US 20220318791A1 · Ballenger et al. · 2022 [cited by applicant]
US 20220335031A1 · Eberlein · 2022 [cited by examiner]
US 20230018387A1 · Kuksta · 2023 [cited by examiner]
CN 102646095 · 2012 [cited by applicant]
JP 2007140982A · 2007 [cited by applicant]
JP 2021018601 · 2021 [cited by applicant]
WO 2016177646 · 2016 [cited by applicant]
WO 2020055615 · 2020 [cited by applicant]
WO 2020061064 · 2020 [cited by applicant]
Shota, Tamamura et al., “An automated testing method for input forms of Web applications”, 2021 Information Processing Society of Japan, 8 pages. [cited by applicant]
Concise Explanation of Relevance of Cited Document 1: Shota, Tamamura “An automated testing method for input forms of Web applications”, IPSJ Research Report, DCC, 2021-DCC-027, Information Processing Society of Japan, … [cited by applicant]
Anonymous: “Automatic Extraction and Provision of Coupon Codes for Online Shopping”, IP.com, Jan. 16, 2019, pp. 1-6. [cited by applicant]
Bronstein et al., “Data Fusion through Cross-modality Metric Learning using Similarity-Sensitive Hashing”, 2010, pp. 1-8. [cited by applicant]
Cao et al., “Collective Deep Quantization for Efficient Cross-Modal Retrieval”, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017, pp. 3974-3980. [cited by applicant]
Cao et al., “Correlation Hashing Network for Efficient Cross-Modal Retrieval”, 2016, pp. 1-12. [cited by applicant]
Cheng et al., “Wide & Deep Learning for Recommender Systems”, DLRS '16, Sep. 15, 2016, pp. 1-4. [cited by applicant]
Deng et al., “DeepCF: A Unified Framework of Representation Learning and Matching Function Learning in Recommender System”, 2019, pp. 1-9. [cited by applicant]
Ding et al., “Collective Matrix Factorization Hashing for Multimodal Data”, 2014, pp. 4321-4328. [cited by applicant]
Dong et al., “A Hybrid Collaborative Filtering Model with Deep Structure for Recommender Systems”, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017, pp. 1309-1315. [cited by applicant]
Gong et al., “Learning Binary Codes for High-Dimensional Data Using Bilinear Projections”, 2013, pp. 484-491. [cited by applicant]
Guo et al., “DeepFM: A Factorization-Machine based Neural Network for CTR Prediction”, Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017, pp. 1725-1731. [cited by applicant]
He, et al., “Neural Collaborative Filtering” Proceedings of the 26th International Conference on World Wide Web (WWW'17) [online] , Apr. 3, 2017, pp. 173-182. [cited by applicant]
He et al., “Neural Factorization Machines for Sparse Predictive Analytics”, SIGIR '17, Aug. 7-11, 2017, pp. 355-364. [cited by applicant]
He et al., “Outer Product-based Neural Collaborative Filtering”, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence 2018, pp. 2227-2233. [cited by applicant]
Javkar et al., “Best Offer Recommendation Service”, 2016 International Conference on Advances in Computing, Communications and Informatics, pp. 2430-2436. [cited by applicant]
Jiang et al., “Deep Cross-Modal Hashing”, 2017, pp. 3232-3240. [cited by applicant]
Koren, “Factorization Meets the Neighborhood: a Multifaceted Collaborative Filtering Model”, KDD 2008, Aug. 24-27, 2008, pp. 426-434. [cited by applicant]
Kumar et al., “Learning Hash Functions for Cross-View Similarity Search”, 2011, pp. 1-6. [cited by applicant]
Li et al., “Deep Binary Reconstruction for Cross-modal Hashing”, MM '17, Oct. 23-27, 2017, pp. 1-8. [cited by applicant]
Li et al., “Deep Collaborative Filtering via Marginalized Denoising Auto-encoder”, CIKM '15, Oct. 19-23, 2015, pp. 811-820. [cited by applicant]
Li et al., “Coupled Cycle-GAN: Unsupervised Hashing Network for Cross-Modal Retrieval”, Thirty-Third AAAI Conference on Artificial Intelligence, 2019, pp. 176-183. [cited by applicant]
Li et al., “Deep Heterogeneous Autoencoders for Collaborative Filtering”, 2018, pp. 1-6. [cited by applicant]
Li et al., “Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrieval”, 2018, pp. 4242-4251. [cited by applicant]
Lian et al., “xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems”, KDD 2018, Aug. 19-23, 2018, pp. 1-10. [cited by applicant]
Liu et al., “Recommender Systems with Heterogeneous Side Information”, WWW '19, May 13-17, 2019, pp. 1-7. [cited by applicant]
Long et al., “Composite Correlation Quantization for Efficient Multimodal Retrieval”, SIGIR '16, Jul. 17-21, 2016, pp. 1-11. [cited by applicant]
Ricci et al., “Recommender Systems Handbook”, 2011, 845 pages. [cited by applicant]
Sedhain et al., “AutoRec: Autoencoders Meet Collaborative Filtering”, WWWWW 2015 Companion, May 18-22, 2015, pp. 1-2. [cited by applicant]
Strub et al., “Hybrid Recommender System based on Autoencoders”, Workshop on Deep Learning for Recommender Systems, Sep. 2016, pp. 1-7. [cited by applicant]
Su et al., “Deep Joint-Semantics Reconstructing Hashing for Large-Scale Unsupervised Cross-Modal Retrieval”, 2019, pp. 3027-3035. [cited by applicant]
Ujwal et al., “Classification-based Adaptive Web Scraper”, 16th IEEE International Conference on Machine Learning and Applications, 2017, pp. 125-132. [cited by applicant]
Wang et al., “Collaborative Deep Learning for Recommender Systems”, KDD '15, Aug. 10-13, 2015, pp. 1235-1244. [cited by applicant]
Watfa et al., “Method and System for Providing an Enhanced Online Shopping Experience to Users Through a Web Browser Extension/Mobile App”, Mar. 30, 2021, pp. 1-5. [cited by applicant]
Wang et al., “Effective Multi-Modal Retrieval based on Stacked Auto-Encoders”, Proceedings of the VLDB Endowment, Sep. 1-5, 2014, pp. 649-660. [cited by applicant]
Wu et al., “Quantized Correlation Hashing for Fast Cross-Modal Search”, Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015, pp. 3946-3952. [cited by applicant]
Wu et al., “Unsupervised Deep Hashing via Binary Latent Factor Models for Large-scale Cross-modal Retrieval”, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018, p. 2854-28… [cited by applicant]
Zhang et al., “Collaborative Quantization for Cross-Modal Similarity Search”, 2019, pp. 1-10. [cited by applicant]
Zhang et al., “Large-Scale Multimodal Hashing with Semantic Correlation Maximization”, Association for the Advancement of Artificial Intelligence, 2014, pp. 1-7. [cited by applicant]