IP Library › Granted Patent US 12,293,157
Granted Patent B2
US 12,293,157 · App. 18/068,642 · Granted May 6, 2025

Semantic matching between a source screen or source data and a target screen using semantic artificial intelligence

Inventors: Mircea Neagovici (Bucharest, RO); Cosmin Voicu (Bucharest, RO)
Assignee: UiPath, Inc.
G06F40/30G06F3/0481G06F40/289G06N20/00G06V30/153G06F17/10G06F17/18G06F30/20G06V30/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,293,157
App. No.
18/068,642
Filed
Dec 20, 2022
Granted
May 6, 2025
Kind
B2
Art Unit
2189
USPC
704/9
Abstract

Semantic matching between a source screen or source data and a target screen using semantic artificial intelligence (AI) for robotic process automation (RPA) workflows is disclosed. The source data or source screen and the target screen are selected on a matching interface, semantic matching is performed between the source data/screen and the target screen using an artificial intelligence/machine learning (AI/ML) model, and matching graphical elements and unmatched graphical elements are highlighted, allowing the developer to see which graphical elements match and which do not. The matching interface may also provide a confidence score of the individual matches, provide an overall mapping score, and allow the developer to hide/unhide the matched/unmatched graphical elements. Activities of an RPA workflow may be automatically created based on the semantic mapping that can be executed to perform the automation.

Claims (86)

1. A non-transitory computer-readable medium storing a computer program, wherein the computer program is configured to cause at least one processor to:

extract data from a source application or a data source after receiving user input via a data extraction interface;

predict a type of the source application or the data source using a classification algorithm;

receive an indication of a target application to copy the extracted data into, via a semantic copy and paste interface;

call one or more artificial intelligence/machine learning (AI/ML) models that have been trained to perform semantic matching between labels in a user interface of the source application and labels in a user interface of the target application, between data elements in the data source and the labels in the user interface of the target application, or both; and

based on the predicted type of the source application or the data source and the semantic matching, automatically copy the extracted data from the source application or data source into the target application, via the semantic copy and paste interface, wherein

at least one field in the user interface of the target application differs from one or more fields of the user interface of the source application and/or one or more data elements from the source data.

2. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:

wait to receive confirmation of the prediction of the source type or a change to the predicted source type; and

after receiving the confirmation or the source type change, prompting a user to provide the indication of the target application.

3. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:

predict a type of the target application using the classification algorithm.

4. The non-transitory computer-readable medium of claim 3 , wherein the computer program is further configured to cause the at least one processor to:

wait to receive confirmation of the prediction of the target type or a change to the predicted target type; and

proceed with the automatic copying of the extracted data from the source application or data source into the target application after receiving the confirmation or the target type change.

5. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:

during the copying, prompt a user before each data entry;

copy the data entry when the user approves; and

skip copying of the data entry when the user declines.

6. The non-transitory computer-readable medium of claim 1 , wherein the semantic copy and paste interface is configured to provide data extraction functionality, data input functionality, copy and paste functionality, and extracted data display functionality.

7. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:

prompt a user to indicate the source application or data source; and

facilitate the indication of the source application or data source via indicate on screen functionality.

8. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:

prompt a user to indicate the target application; and

facilitate the indication of the target application via indicate on screen functionality.

9. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:

display a summary of the extracted data.

10. The non-transitory computer-readable medium of claim 1 , wherein

the semantic matching is performed based on indications of graphical elements associated with semantically matched labels in the target application and respective confidence scores from the one or more AI/ML models.

11. The non-transitory computer-readable medium of claim 1 , wherein the computer program is a robotic process automation (RPA) robot.

12. A computer-implemented method, comprising:

extracting data from a source application or a data source, by a robotic process automation (RPA) robot executing on a computing system, after receiving user input via a data extraction interface;

predicting a type of the source application or the data source using a classification algorithm, by the RPA robot;

receiving an indication of a target application to copy the extracted data from a semantic copy and paste interface, by the RPA robot;

predicting a type of the target application using the classification algorithm, by the RPA robot;

calling, by the RPA robot, one or more artificial intelligence/machine learning (AI/ML) models that have been trained to perform semantic matching between labels in a user interface of the source application and labels in a user interface of the target application, between data elements in the data source and the labels in the user interface of the target application, or both; and

based on the predicted type of the source application or the data source and the semantic matching, automatically copying the extracted data from the source application or data source into the target application, by the RPA robot, wherein

at least one field in the user interface of the target application differs from one or more fields of the user interface of the source application and/or one or more data elements from the source data.

13. The computer-implemented method of claim 12 , further comprising:

waiting to receive confirmation of the prediction of the source type or a change to the predicted source type, by the RPA robot; and

after receiving the confirmation or the source type change, prompting a user to provide the indication of the target application, by the RPA robot.

14. The computer-implemented method of claim 12 , further comprising:

waiting to receive confirmation of the prediction of the target type or a change to the predicted target type, by the RPA robot; and

proceeding with the automatic copying of the extracted data from the source application or data source into the target application after receiving the confirmation or the target type change, by the RPA robot.

15. The computer-implemented method of claim 12 , further comprising:

during the copying, prompting a user before each data entry, by the RPA robot;

copying the data entry when the user approves, by the RPA robot; and

skipping copying of the data entry when the user declines, by the RPA robot.

16. The computer-implemented method of claim 12 , wherein the semantic copy and paste interface is configured to provide data extraction functionality, data input functionality, copy and paste functionality, and extracted data display functionality.

17. The computer-implemented method of claim 12 , further comprising:

prompting a user to indicate the source application or data source, by the RPA robot; and

facilitating the indication of the source application or data source via indicate on screen functionality, by the RPA robot.

18. The computer-implemented method of claim 12 , further comprising:

prompting a user to indicate the target application, by the RPA robot; and

facilitating the indication of the target application via indicate on screen functionality, by the RPA robot.

19. The computer-implemented method of claim 12 , wherein the semantic matching is performed

based on indications of graphical elements associated with semantically matched labels in the target application and respective confidence scores from the one or more AI/ML models.

20. A computing system, comprising:

memory storing computer program instructions; and

at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause at least one processor to:

extract data from a source application or a data source after receiving user input via a data extraction interface;

predict a type of the source application or the data source using a classification algorithm;

receive an indication of a target application to copy the extracted data into, via a semantic copy and paste interface;

predict a type of the target application using the classification algorithm;

call one or more artificial intelligence/machine learning (AI/ML) models that have been trained to perform semantic matching between labels in a user interface of the source application and labels in a user interface of the target application, between data elements in the data source and the labels in the user interface of the target application, or both; and

based on indications of graphical elements associated with semantically matched labels in the user interface of the target application and respective confidence scores from the one or more AI/ML models, automatically copy the extracted data from fields in the user interface of the source application having labels that the one or more AI/ML models identified as semantically matching fields in the user interface of the target application, automatically copy the data elements from the source data into the fields of the user interface of the target application having labels that the one or more AI/ML models identified as semantically matching the data elements from the data source, or both, wherein

at least one field in the user interface of the target application differs from one or more fields of the user interface of the source application and/or one or more data elements from the source data.

21. The computing system of claim 20 , wherein the computer program instructions are further configured to cause the at least one processor to:

wait to receive confirmation of the prediction of the source type or a change to the predicted source type; and

after receiving the confirmation or the source type change, prompting a user to provide the indication of the target application.

22. The computing system of claim 20 , wherein the computer program instructions are further configured to cause the at least one processor to:

wait to receive confirmation of the prediction of the target type or a change to the predicted target type; and

proceed with the automatic copying of the extracted data from the source application or data source into the target application after receiving the confirmation or the target type change.

23. The computing system of claim 20 , wherein the computer program instructions are further configured to cause the at least one processor to:

during the copying, prompt a user before each data entry;

copy the data entry when the user approves; and

skip copying of the data entry when the user declines.

24. The computing system of claim 20 , wherein the semantic copy and paste interface is configured to provide data extraction functionality, data input functionality, copy and paste functionality, and extracted data display functionality.

25. The computing system of claim 20 , wherein the computer program instructions are further configured to cause the at least one processor to:

prompt a user to indicate the source application or data source; and

facilitate the indication of the source application or data source via indicate on screen functionality.

26. The computing system of claim 20 , wherein the computer program instructions are further configured to cause the at least one processor to:

prompt a user to indicate the target application; and

facilitate the indication of the target application via indicate on screen functionality.

27. The computing system of claim 20 , wherein the computer program instructions are for a robotic process automation (RPA) robot.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 20, 2022
From: NEAGOVICI, MIRCEA; VOICU, COSMIN
To: UIPATH, INC.
Reel/Frame 062156/0066 →
Continuity (2)
Continuation 17494744 · Oct 5, 2021
Related Publication 20230125223A1 · Apr 27, 2023
References Cited (57)
US 7624353B2 · Beumer · 2009 [cited by applicant]
US 10860905B1 · Gligan et al. · 2020 [cited by applicant]
US 10936351B2 · Voicu · 2021 [cited by applicant]
US 11200073B1 · Voicu · 2021 [cited by applicant]
US 20080027930A1 · Bohannon et al. · 2008 [cited by applicant]
US 20180032626A1 · Ben-Aharon et al. · 2018 [cited by applicant]
US 20190213039A1 · Lecue et al. · 2019 [cited by applicant]
US 20190213252A1 · Simard · 2019 [cited by examiner]
US 20190318020A1 · Chauhan et al. · 2019 [cited by applicant]
US 20200249964A1 · Fernandes et al. · 2020 [cited by applicant]
US 20200366566A1 · Avasarala et al. · 2020 [cited by applicant]
US 20210019157A1 · Voicu · 2021 [cited by applicant]
US 20210019574A1 · Voicu · 2021 [cited by applicant]
US 20210089332A1 · Zohar et al. · 2021 [cited by applicant]
US 20210097274A1 · Gligan et al. · 2021 [cited by applicant]
US 20210109834A1 · Singh et al. · 2021 [cited by applicant]
US 20210294945A1 · Müller · 2021 [cited by examiner]
US 20210334470A1 · Molin et al. · 2021 [cited by applicant]
US 20210397157A1 · Cote et al. · 2021 [cited by applicant]
US 20220011732A1 · Hall · 2022 [cited by applicant]
US 20220012024A1 · Grigore · 2022 [cited by applicant]
US 20220197674A1 · Ginoya · 2022 [cited by examiner]
US 20220317978A1 · Barik et al. · 2022 [cited by applicant]
US 20230182296A1 · Sermanet et al. · 2023 [cited by applicant]
CN 112231033A · 2021 [cited by applicant]
CN 112967132A · 2021 [cited by applicant]
CN 113435160A · 2021 [cited by applicant]
WO 2020061700A1 · 2020 [cited by applicant]
WO 2021055102A1 · 2021 [cited by applicant]
Reddy KN, Harichandana U, Alekhya T, Rajesh SM. A study of robotic process automation among artificial intelligence. International Journal of Scientific and Research Publications. Feb. 2019;9(2):392-7. (Year: 2019). [cited by examiner]
Taulli T, Taulli T. Robotic Process Automation (RPA) An Easier Path to AI. Artificial Intelligence Basics: A Non-Technical Introduction. 2019:91-102. (Year: 2019). [cited by examiner]
Extended European Search Report, issued Feb. 8, 2024, European Patent Application No. EP21820901.3. [cited by applicant]
Chuan Guo et al., “On Calibration of Modern Neural Networks,” Proceedings of the 34th International Conference on Machine Learning, PMLR 70:1321-1330 (2017). [cited by applicant]
International Search Report, issued Jun. 22, 2022, PCT Application No. PCT/US21/57169. [cited by applicant]
Jitterbit Mapping Mode Page available at https://success.jitterbit.com/display/CS/Mapping+Mode (last updated Oct. 20, 2021). [cited by applicant]
Jonathan Grandperrin, “How to use confidence scores in machine learning models,” available at https://towardsdatascience.com/how-to-use-confidence-scores-in-machine-learning-models-abe9773306fa (Jan. 19, 2021). [cited by applicant]
Mappings in Informatica Page available at https://www.guru99.com/mappings-informatica.html (Last updated Oct. 7, 2021). [cited by applicant]
Nils Reimers et al., “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” available at https://arxiv.org/pdf/1908.10084.pdf (Aug. 27, 2019). [cited by applicant]
Pratik Shukla et al., “Main Types of Neural Networks and Their Applications—Tutorial,” available at https://pub.towardsai.net/main-types-of-neural-networks-and-its-applications-tutorial-734480d7ec8e (last updated Aug. 1… [cited by applicant]
SBERT.net SentenceTransformers Documentation available at https://www.sbert.net/ (last accessed Oct. 4, 2021). [cited by applicant]
Tony Yiu, “Understanding Neural Networks,” available at https://towardsdatascience.com/understanding-neural-networks-19020b758230 (Jun. 2, 2019). [cited by applicant]
JiPath Object Repository Page available at https://docs.uipath.com/studio/docs/about-object-repository (last accessed Oct. 4, 2021). [cited by applicant]
European Search Report issued in European Application No. 22199898.2 on Feb. 8, 2023. [cited by applicant]
Giunchiglia et al., “Semantic Matching,” The Knowledge Engineering Review, Cambridge University Press, vol. 18, No. 3, pp. 265-280 (Sep. 1, 2003). [cited by applicant]
Lasker et al., “Semi-Supervised Semantic Matching,” Pattern Recognition :5ht Asian Conference, ACPR, Auckland, New Zealand, Nov. 26-29, 2019, LNCS 11131, pp. 444-455 (Jan. 23, 2019). [cited by applicant]
Chakrabarti, et al., “TableQnA: Answering List Intent Queries with Web Tables”, arXiv preprint arXiv:2001.04828. Jan. 10, 2020. (Year: 2020). [cited by applicant]
Chuen-Meei Gan, “Notice of Allowance”, issued Aug. 21, 2024, U.S. Appl. No. 17/494,744. [cited by applicant]
Chuen-Meei Gan, “Notice of Allowance”, issued Aug. 26, 2024, U.S. Appl. No. 18/052,378. [cited by applicant]
K. P. Naveen Reddy et al., “A Study of Robotic Process Automation Among Artificial Intelligence,” International Journal of Scientific and Research Publications, vol. 9, Issue 2 (Feb. 2019) (cited by Examiner in email on… [cited by applicant]
Kaushik Chakrabarti et al., “TableQnA: Answering ListIntent Queries With Web Tables,” available at https://arxiv.org/pdf/2001.04828 (Jan. 10, 2020) (cited by Examiner in email on Jul. 16, 2024). [cited by applicant]
Nga B Nguyen, “Non-Final Office Action”, issued Aug. 27, 2024, U.S. Appl. No. 17/746,855. [cited by applicant]
Nga B Nguyen, “Non-Final Office Action”, issued Oct. 24, 2024, U.S. Appl. No. 18/056,018. [cited by applicant]
Reddy, et al. “A Study of Robotic Process Automation Among Artificial Intelligence”, International Journal of Scientific and Research Publications. Feb. 2019;9(2):392-7. (Year: 2019). [cited by applicant]
Taulli, Tom, “Robotic Process Automation (RPA) An Easier Path to AI. Artificial Intelligence Basics: A Non-Technical Introduction”, 2019:91-102. (Year: 2019). [cited by applicant]
Tom Taulli, “Robotic Process Automation (RPA) An Easier Path to AI,” chapter 5, available at https://link.springer.com/chapter/10.1007/978-1-4842-5028-0_5 (2019) (cited by Examiner in email on Jul. 16, 2024). [cited by applicant]
Nga B Nguyen, “Notice of Allowance”, issued Jan. 2, 2025, U.S. Appl. No. 17/746,855. [cited by applicant]
Nga B Nguyen, “Notice of Allowance”, issued Mar. 26, 2025, U.S. Appl. No. 18/056,018. [cited by applicant]
Cited By (1)
US 12,737,461