Intent-based automation
Intent-based automation that discovers automatable tasks and/or determines task variants in data is disclosed. Task capture data may be utilized to determine task variants in task mining data. Semantic understanding of user actions by artificial intelligence (AI)/machine learning (ML) model(s), for example, may be applied to determine the intent of the user rather than only focusing on what actions the user is performing on the computing system. Application logs and semantic understanding may be used to facilitate a more accurate determination of what the user actually intends to do. Task capture for individual user flows may be performed. Once these are captured, task capture algorithms and AI/ML models are used to determine which parts of the flows are similar and/or match and which parts are unique. The path through these flows can then be followed to build a process graph that includes decision points representing the unique flows.
1 . A non-transitory computer-readable medium storing a computer program for performing intent-based automation, the computer program configured to cause at least one processor to:
run a clustering artificial intelligence (AI)/machine learning (ML) model on vector representations of a sequence of screens pertaining to a captured task flow to produce a trace comprising a sequence of clusters; and
run a classifier AI/ML model on the sequence of clusters in the trace from the clustering AI/ML model; and
when the classifier AI/ML model identifies a task type from the sequence of clusters in the trace:
map the classified task type to a robotic process automation (RPA) workflow that accomplishes an intent of the captured task flow.
2 . The non-transitory computer-readable medium of claim 1 , wherein the classifier AI/ML model is configured to compare the sequence of clusters in the trace from the clustering AI/ML model to other sequences of clusters representing traces from previously identified task types to determine the task type.
3 . The non-transitory computer-readable medium of claim 1 , wherein the classifier AI/ML model is configured to represent individual screens or the sequence of screens as vectors and compute a vector distance from vectors for previously identified task types.
4 . The non-transitory computer-readable medium of claim 1 , wherein the clustering AI/ML model receives and uses system information, semantic information from a natural language processing (NLP) model, or both, to provide context to the classified task type.
5 . The non-transitory computer-readable medium of claim 4 , wherein the system information comprises user interaction events, application programming interface (API) events, operating system (OS)/document object model (DOM) events, user interaction/input types, location data, an indication of which mouse button was pressed, an application or process name, a UI descriptor, or any combination thereof.
6 . The non-transitory computer-readable medium of claim 1 , wherein the computer program is further configured to cause the at least one processor to:
automatically generate or open the RPA workflow that accomplishes the intent of the captured task flow in an RPA designer application.
7 . The non-transitory computer-readable medium of claim 1 , wherein when the classifier AI/ML model does not identify a task type from the sequence of clusters in the trace, the computer program is further configured to cause the at least one processor to:
automatically generate activities for actions in the task flow.
8 . The non-transitory computer-readable medium of claim 7 , wherein the automatic generation of the activities comprises creating user interface (UI) descriptors for the generated activities using the sequence of screens, system information, semantic information from a natural language processing (NLP) model, or any combination thereof.
9 . The non-transitory computer-readable medium of claim 7 , wherein the computer program is further configured to cause the at least one processor to:
generate an RPA workflow comprising the generated activities.
10 . The non-transitory computer-readable medium of claim 9 , wherein the computer program is further configured to cause the at least one processor to:
send the trace of the sequence of clusters in the trace to a repository for future mappings.
11 . A computer-implemented method, comprising:
running a clustering artificial intelligence (AI)/machine learning (ML) model on vector representations of a sequence of screens pertaining to a captured task flow to produce a trace comprising a sequence of clusters; and
running a classifier AI/ML model on the sequence of clusters in the trace from the clustering AI/ML model
when the classifier AI/ML model identifies a task type from the sequence of clusters in the trace:
mapping the classified task type to a robotic process automation (RPA) workflow that accomplishes an intent of the captured task flow, and
when the classifier AI/ML model does not identify a task type from the sequence of clusters in the trace:
automatically generating activities for actions in the task flow.
12 . The computer-implemented method of claim 11 , wherein the classifier AI/ML model is configured to compare the sequence of clusters in the trace from the clustering AI/ML model to other sequences of clusters representing traces from previously identified task types to determine the task type.
13 . The computer-implemented method of claim 11 , wherein the classifier AI/ML model is configured to represent individual screens or the sequence of screens as vectors and compute a vector distance from vectors for previously identified task types.
14 . The computer-implemented method of claim 11 , wherein the clustering AI/ML model receives and uses system information, semantic information from a natural language processing (NLP) model, or both, to provide context to the classified task type.
15 . The computer-implemented method of claim 14 , wherein the system information comprises user interaction events, application programming interface (API) events, operating system (OS)/document object model (DOM) events, user interaction/input types, location data, an indication of which mouse button was pressed, an application or process name, a UI descriptor, or any combination thereof.
16 . The computer-implemented method of claim 11 , further comprising:
automatically generating or opening the RPA workflow that accomplishes the intent of the captured task flow in an RPA designer application.
17 . The computer-implemented method of claim 11 , wherein the automatic generation of the activities comprises creating user interface (UI) descriptors for the generated activities using the sequence of screens, system information, semantic information from a natural language processing (NLP) model, or any combination thereof.
18 . The computer-implemented method of claim 11 , further comprising:
sending the trace of the sequence of clusters in the trace to a repository for future mappings.
19 . A computing system, comprising:
memory storing computer program instructions; and
at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause the at least one processor to:
run a clustering artificial intelligence (AI)/machine learning (ML) model on vector representations of a sequence of screens pertaining to a captured task flow to produce a trace comprising a sequence of clusters; and
run a classifier AI/ML model on the sequence of clusters in the trace from the clustering AI/ML model; and
when the classifier AI/ML model identifies a task type from the sequence of clusters in the trace:
map the classified task type to a robotic process automation (RPA) workflow that accomplishes an intent of the captured task flow, wherein
the classifier AI/ML model is configured to compare the sequence of clusters in the trace from the clustering AI/ML model to other sequences of clusters representing traces from previously identified task types to determine the task type, or
the classifier AI/ML model is configured to represent individual screens or the sequence of screens as vectors and compute a vector distance from vectors for previously identified task types.
20 . The computing system of claim 19 , wherein the clustering AI/ML model receives and uses system information, semantic information from a natural language processing (NLP) model, or both, to provide context to the classified task type.
21 . The computing system of claim 20 , wherein the system information comprises user interaction events, application programming interface (API) events, operating system (OS)/document object model (DOM) events, user interaction/input types, location data, an indication of which mouse button was pressed, an application or process name, a UI descriptor, or any combination thereof.
22 . The computing system of claim 19 , wherein the computer program instructions are further configured to cause the at least one processor to:
automatically generate or open the RPA workflow that accomplishes the intent of the captured task flow in an RPA designer application.
23 . The computing system of claim 19 , wherein when the classifier AI/ML model does not identify a task type from the sequence of clusters in the trace, the computer program instructions are further configured to cause the at least one processor to:
automatically generate activities for actions in the task flow.
24 . The computing system of claim 23 , wherein the automatic generation of the activities comprises creating user interface (UI) descriptors for the generated activities using the sequence of screens, system information, semantic information from a natural language processing (NLP) model, or any combination thereof.
25 . The computing system of claim 23 , wherein the computer program instructions are further configured to cause the at least one processor to:
generate an RPA workflow comprising the generated activities.
26 . The computing system of claim 25 , wherein the computer program instructions are further configured to cause the at least one processor to:
send the trace of the sequence of clusters in the trace to a repository for future mappings.