Automatic data transfer between a source and a target using semantic artificial intelligence for robotic process automation
Automatic data transfer between a source and a target using semantic artificial intelligence (AI) for robotic process automation (RPA) is disclosed. A user may be provided with the option of selecting a source and a target and indicating through an intuitive user interface that he or she would like to copy data from the source to the destination, regardless of format. This may be done at design time or at run time. For instance, the source and/or target may be a web page, a graphical user interface (GUI) of an application, an image, a file explorer, a spreadsheet, a relational database, a flat file source, any other suitable format, or any combination thereof. The source and the target may have different formats. The source, target, or both may not necessarily be visible to the user.
1. A non-transitory computer-readable medium storing a computer program, the computer program configured to cause at least one processor to:
receive a selection of a source from a user via a graphical user interface (GUI) and/or an input device, the source comprising a plurality of screens, a plurality of screen versions, or both;
receive a selection of a target from the user via the GUI and/or the input device;
store metadata from a clipboard of an operating system, previous screen entries from the plurality of screens and/or the plurality of screen versions of the source, or both, in a persistent data structure;
call one or more artificial intelligence/machine learning (AI/ML) models that have been trained to perform semantic matching between labels in the source and labels in the target, between values in the source and the labels in the target, or both, the one or more AI/ML models configured to use the stored metadata in the persistent data structure and to suggest matches between the labels in the source and the labels in the target, between the values in the source and the labels in the target, or both, and respective confidence scores;
receive the suggested matches and respective confidence scores;
automatically copy values from the source to the target based on the semantic matching between the labels in the source and the labels in the target, between the values in the source and the labels in the target, or both, responsive to the matches exceeding a threshold confidence score; and
display the automatically copied values in the GUI.
2. The non-transitory computer-readable medium of claim 1 , wherein the persistent data structure comprises associations of labels to values, associations of label/value pairs to screens of the source, or both.
3. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
generate one or more activities in a robotic process automation (RPA) workflow that copy data from the values in the source having labels that the one or more AI/ML models identified as semantically matching the labels in the target into respective fields or locations in the target.
4. The non-transitory computer-readable medium of claim 3 , wherein the computer program is further configured to cause the at least one processor to:
generate an automation implementing the one or more generated activities in the RPA workflow; and
deploy the generated automation and an RPA robot configured to execute the automation in a runtime environment.
5. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
determine a start of a copy-and-paste task;
determine an end of the copy-and-paste task; and
generate associated activities for the start and the end of the copy-and-paste task.
6. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
receive indications of fields or locations in the source associated with semantically matched labels in the target and respective confidence scores from the one or more AI/ML models; and
display the graphical elements or locations associated with the semantically matched labels on the target in a matching interface, display the respective confidence scores for the potential matching fields or locations identified by the one or more AI/ML models in the matching interface, or both.
7. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
receive a correction to a label, field, or location in the target identified by the one or more AI/ML models as having an associated semantically matching label in the source, receive an indication of a new label, field, or location in the target that was not semantically matched to a label in the source by the one or more AI/ML models, or both;
collect information pertaining to the corrected and/or newly labeled label, field, or location in the target and the associated label in the source; and
directly or indirectly store the collected information for retraining of the one or more AI/ML models in the persistent data structure or another persistent data structure, the persistent data structure or the other persistent data structure comprising label and field associations.
8. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
generate a composite confidence score from the confidence scores for the fields or locations associated with semantically matched labels in the target; and
display the composite confidence score in the matching interface.
9. The non-transitory computer-readable medium of claim 1 , wherein the one or more AI/ML models are trained by providing words and phrases with semantic associations between the words and phrases such that similar words and phrases for a given word or phrase can be identified, and providing contextual labels pertaining to a screen in which the words and phrases appear.
10. The non-transitory computer-readable medium of claim 1 , wherein the computer program is or comprises a robotic process automation (RPA) designer application.
11. The non-transitory computer-readable medium of claim 1 , wherein
the one or more AI/ML models comprise a computer vision (CV) model, an optical character recognition (OCR) model, a label matching model, and an input data matching model,
the label matching model matches labels detected by the OCR model with fields or locations detected by the CV model, and
the input data model receives the matching labels from the label matching model and semantically matches the labels and values from the source with the labels and fields or locations associated with the semantically matched labels on the target.
12. The non-transitory computer-readable medium of claim 1 , wherein the computer program is or comprises a robotic process automation (RPA) robot.
13. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
display a type selection interface providing options for supported types for the source, the target, or both.
14. The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
select at least one additional source; and
provide information from the at least one additional source to the one or more AI/ML models in addition to information from the source to perform the semantic matching.
15. A computer-implemented method, comprising:
receiving, by a computing system, a selection of a source from a user via a graphical user interface (GUI) and/or an input device;
receiving, by the computing system, a selection of a target from the user via the GUI and/or the input device;
calling, by the computing system, one or more artificial intelligence/machine learning (AI/ML) models that have been trained to perform semantic matching between labels in a source and labels in a target, between values in the source and the labels in the target, or both, the one or more AI/ML models configured to use stored metadata in a persistent data structure and to suggest matches between the labels in the source and the labels in the target, between the values in the source and the labels in the target, or both, and respective confidence scores, the stored metadata comprising data from a clipboard of an operating system, previous screen entries from the plurality of screens and/or the plurality of screen versions of the source, or both;
receiving, by the computing system, the suggested matches and respective confidence scores;
automatically copying values from the source to the target, by the computing system, based on the semantic matching between the labels in the source and the labels in the target, between the values in the source and the labels in the target, or both; and
displaying, by the computing system, the automatically copied values in the GUI.
16. The computer-implemented method of claim 15 , wherein the persistent data structure comprises associations of labels to values, associations of label/value pairs to screens of the source, or both.
17. The computer-implemented method of claim 15 , further comprising:
generating, by the computing system, one or more activities in a robotic process automation (RPA) workflow that copy data from the values in the source having labels that the one or more AI/ML models identified as semantically matching the labels in the target into respective fields or locations in the target.
18. The computer-implemented method of claim 17 , further comprising:
generating an automation implementing the one or more generated activities in the RPA workflow, by the computing system; and
deploying the generated automation and an RPA robot configured to execute the automation in a runtime environment, by the computing system.
19. The computer-implemented method of claim 15 , further comprising:
determining, by the computing system, a start of a copy-and-paste task;
determining, by the computing system, an end of the copy-and-paste task; and
generating, by the computing system, associated activities for the start and the end of the copy-and-paste task.
20. The computer-implemented method of claim 15 , further comprising:
receiving indications of fields or locations in the source associated with semantically matched labels in the target and respective confidence scores from the one or more AI/ML models, by the computing system; and
displaying, by the computing system, the graphical elements or locations associated with the semantically matched labels on the target in a matching interface, displaying the respective confidence scores for the potential matching fields or locations identified by the one or more AI/ML models in the matching interface, or both.
21. The computer-implemented method of claim 15 , further comprising:
receiving, by the computing system, a correction to a label, field, or location in the target identified by the one or more AI/ML models as having an associated semantically matching label in the source, receiving an indication of a new label, field, or location in the target that was not semantically matched to a label in the source by the one or more AI/ML models, or both;
collecting information pertaining to the corrected and/or newly labeled label, field, or location in the target and the associated label in the source, by the computing system; and
directly or indirectly storing the collected information for retraining of the one or more AI/ML models in the persistent data structure or another persistent data structure, by the computing system, the persistent data structure or the other persistent data structure comprising label and field associations.
22. The computer-implemented method of claim 15 , further comprising:
generating a composite confidence score from the confidence scores for the fields or locations associated with semantically matched labels in the target, by the computing system; and
displaying the composite confidence score in the matching interface, by the computing system.
23. The computer-implemented method of claim 15 , wherein the one or more AI/ML models are trained by providing words and phrases with semantic associations between the words and phrases such that similar words and phrases for a given word or phrase can be identified, and providing contextual labels pertaining to a screen in which the words and phrases appear.
24. The computer-implemented method of claim 15 , wherein
the one or more AI/ML models comprise a computer vision (CV) model, an optical character recognition (OCR) model, a label matching model, and an input data matching model,
the label matching model matches labels detected by the OCR model with fields or locations detected by the CV model, and
the input data model receives the matching labels from the label matching model and semantically matches the labels and values from the source with the labels and fields or locations associated with the semantically matched labels on the target.
25. The computer-implemented method of claim 15 , further comprising:
displaying, by the computing system, a type selection interface providing options for supported types for the source, the target, or both.
26. The computer-implemented method of claim 15 , wherein the one or more AI/ML models use information from a plurality of sources to perform the semantic matching between labels in the plurality of sources and the labels in the target, between the values in the plurality of sources and the labels in the target, or both.
27. A computing system, comprising:
memory storing computer program instructions; and
at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to:
receive a selection of a source from a user via a graphical user interface (GUI) and/or an input device;
receive a selection of a target from the user via the GUI and/or the input device;
call one or more artificial intelligence/machine learning (AI/ML) models that have been trained to perform semantic matching between labels in the source and labels in the target, between values in the source and the labels in the target, or both, the one or more AI/ML models configured to use stored metadata in a persistent data structure and to suggest matches between the labels in the source and the labels in the target, between the values in the source and the labels in the target, or both, and respective confidence scores, the stored metadata comprising data from a clipboard of an operating system, previous screen entries from the plurality of screens and/or the plurality of screen versions of the source, or both;
automatically copy values from the source to the target based on the semantic matching between the labels in the source and the labels in the target, between the values in the source and the labels in the target, or both, responsive to the matches exceeding a threshold confidence score; and
display the automatically copied values in the GUI, wherein
the computer program instructions are or comprise a robotic process automation (RPA) designer application or an RPA robot.
28. The computing system of claim 27 , wherein the persistent data structure comprises associations of labels to values, associations of label/value pairs to screens of the source, or both.
29. The computing system of claim 27 , wherein the computer program instructions are further configured to cause the at least one processor to:
generate one or more activities in an RPA workflow that copy data from the values in the source having labels that the one or more AI/ML models identified as semantically matching the labels in the target into respective fields or locations in the target.
30. The computing system of claim 29 , wherein the computer program instructions are further configured to cause the at least one processor to:
generate an automation implementing the one or more generated activities in the RPA workflow; and
deploy the generated automation and an RPA robot configured to execute the automation in a runtime environment.
31. The computing system of claim 27 , wherein the computer program instructions are further configured to cause the at least one processor to:
determine a start of a copy-and-paste task;
determine an end of the copy-and-paste task; and
generate associated activities for the start and the end of the copy-and-paste task.
32. The computing system of claim 27 , wherein the computer program instructions are further configured to cause the at least one processor to:
receive indications of fields or locations in the source associated with semantically matched labels in the target and respective confidence scores from the one or more AI/ML models; and
display the graphical elements or locations associated with the semantically matched labels on the target in a matching interface, display the respective confidence scores for the potential matching fields or locations identified by the one or more AI/ML models in the matching interface, or both.
33. The computing system of claim 27 , wherein the computer program instructions are further configured to cause the at least one processor to:
receive a correction to a label, field, or location in the target identified by the one or more AI/ML models as having an associated semantically matching label in the source, receive an indication of a new label, field, or location in the target that was not semantically matched to a label in the source by the one or more AI/ML models, or both;
collect information pertaining to the corrected and/or newly labeled label, field, or location in the target and the associated label in the source; and
directly or indirectly store the collected information for retraining of the one or more AI/ML models in the persistent data structure or another persistent data structure, the persistent data structure or the other persistent data structure comprising label and field associations.
34. The computing system of claim 27 , wherein the computer program instructions are further configured to cause the at least one processor to:
generate a composite confidence score from the confidence scores for the fields or locations associated with semantically matched labels in the target; and
display the composite confidence score in the matching interface.
35. The computing system of claim 27 , wherein the one or more AI/ML models are trained by providing words and phrases with semantic associations between the words and phrases such that similar words and phrases for a given word or phrase can be identified, and providing contextual labels pertaining to a screen in which the words and phrases appear.
36. The computing system of claim 27 , wherein
the one or more AI/ML models comprise a computer vision (CV) model, an optical character recognition (OCR) model, a label matching model, and an input data matching model,
the label matching model matches labels detected by the OCR model with fields or locations detected by the CV model, and
the input data model receives the matching labels from the label matching model and semantically matches the labels and values from the source with the labels and fields or locations associated with the semantically matched labels on the target.
37. The computing system of claim 27 , wherein the computer program instructions are further configured to cause the at least one processor to:
display a type selection interface providing options for supported types for the source, the target, or both.
38. The computing system of claim 27 , wherein the one or more AI/ML models use information from a plurality of sources to perform the semantic matching between labels in the plurality of sources and the labels in the target, between the values in the plurality of sources and the labels in the target, or both.