Systems and methods for discovering automatable tasks
According to at least one aspect, a system for analyzing computer usage data of a user to identify an underlying task being performed by the user is provided. The system includes a hardware processor and a non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the hardware processor, cause the hardware processor to perform: receiving a plurality of events each indicative of an action performed by a user and contextual information associated with the action performed by the user, clustering the plurality of events to generate a plurality of clustered events, identifying a plurality of sub-tasks in the plurality of clustered events that each comprise a sequence of clustered events, identifying a task in the plurality of clustered events being performed by the user that comprises at least one sub-task, and generating a score for the task indicative of a difficulty of automating the task.
1 . A system, comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one computer hardware processor to perform:
receiving, by a computing device, an indication of an action performed by a user of the computing device via a graphical user interface (GUI) of an application executing on the computing device;
gathering, by the computing device, contextual information associated with the action performed by the user via the GUI of the application executing on the computing device;
storing information indicative of the action and the contextual information in a volatile memory of the computing device as an event;
determining whether the event stored in the volatile memory of the computing device includes personal information of the user;
after determining that the event stored in the volatile memory of the computing device does not include the personal information, writing the event to an event log in a non-volatile memory, the event log including a plurality of events;
identifying a task, in furtherance of which the user performed the action, at least in part by clustering events in the plurality of events; and
generating a score for the task indicative of a difficulty of automating the task, wherein the score is generated using at least one value selected from the group consisting of: information identifying applications used to perform the task, a number of applications used to perform the task, a number of user interface elements used in the task, an amount of natural language input provided to the applications during the task, a number of keystrokes performed in the task, a number of clicks performed in the task, a ratio between keystrokes and clicks performed in the task, a percentage of time spent performing typing actions in the task, a percentage of time spent performing clicking actions in the task, and a frequency of copy and paste actions performed in the task.
2 . The system of claim 1 , wherein the action includes at least one of a keystroke and a click.
3 . The system of claim 1 , wherein gathering the contextual information includes gathering information regarding a state of the user interface when the user performed the action.
4 . The system of claim 3 , wherein gathering the information regarding the state of the user interface includes gathering information selected from the group consisting of: a name of an active application, a title in a title bar of the active application, a value in at least one field of the active application, a size of a window of the active application, and a location of the window of the active application within the user interface.
5 . The system of claim 1 , wherein storing the information indicative of the action and the contextual information in the volatile memory as the event comprises storing the information indicative of the action and the contextual information in the volatile memory without at least a portion of the personal information.
6 . The system of claim 1 ,
wherein the at least one computer hardware processor includes a first computer hardware processor part of the computing device and a second computer hardware processor part of an external system to the computing device, and
wherein acts of receiving, gathering, storing, determining, and writing, are performed by the first computer hardware processor; and
acts of identifying the task and generating the score are performed by the second computer hardware processor.
7 . The system of claim 6 , wherein the processor-executable instructions further cause the at least one hardware processor to perform:
transmitting the event log to the external system.
8 . A method, comprising:
using at least one computer hardware processor to perform:
receiving an indication of an action performed by a user of a computing device via a graphical user interface (GUI) of an application executing on the computing device;
gathering contextual information associated with the action performed by the user via the GUI of the application executing on the computing device;
storing information indicative of the action and the contextual information in a volatile memory of the computing device as an event;
determining whether the event stored in the volatile memory of the computing device includes personal information of the user;
after determining that the event stored in the volatile memory of the computing device does not include the personal information, writing the event to an event log in a non-volatile memory, the event log including a plurality of events;
identifying a task, in furtherance of which the user performed the action, at least in part by clustering events in the plurality of events; and
generating a score for the task indicative of a difficulty of automating the task, wherein the score is generated using at least one value selected from the group consisting of: information identifying applications used to perform the task, a number of applications used to perform the task, a number of user interface elements used in the task, an amount of natural language input provided to the applications during the task, a number of keystrokes performed in the task, a number of clicks performed in the task, a ratio between keystrokes and clicks performed in the task, a percentage of time spent performing typing actions in the task, a percentage of time spent performing clicking actions in the task, and a frequency of copy and paste actions performed in the task.
9 . The method of claim 8 , wherein the action includes at least one of a keystroke and a click.
10 . The method of claim 8 , wherein gathering the contextual information includes gathering information regarding a state of the user interface when the user performed the action.
11 . The method of claim 10 , wherein gathering the information regarding the state of the user interface includes gathering information selected from the group consisting of: a name of an active application, a title in a title bar of the active application, a value in at least one field of the active application, a size of a window of the active application, and a location of the window of the active application within the user interface.
12 . The method of claim 8 , wherein storing the information indicative of the action and the contextual information in the volatile memory as the event comprises storing the information indicative of the action and the contextual information in the volatile memory without at least a portion of the personal information.
13 . The method of claim 8 ,
wherein the at least one computer hardware processor includes a first computer hardware processor part of the computing device and a second computer hardware processor part of an external system to the computing device, and
wherein acts of receiving, gathering, storing, determining, and writing, are performed by the first computer hardware processor; and
acts of identifying the task and generating the score are performed by the second computer hardware processor.
14 . The method of claim 13 , further comprising:
transmitting the event log to the external system.
15 . At least one non-transitory computer readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform:
receiving an indication of an action performed by a user of a computing device via a graphical user interface (GUI) of an application executing on the computing device;
gathering contextual information associated with the action performed by the user via the GUI of the application executing on the computing device;
storing information indicative of the action and the contextual information in a volatile memory of the computing device as an event;
determining whether the event stored in the volatile memory of the computing device includes personal information of the user;
after determining that the event stored in the volatile memory of the computing device does not include the personal information, writing the event to an event log in a non-volatile memory, the event log including a plurality of events;
identifying a task, in furtherance of which the user performed the action, at least in part by clustering events in the plurality of events; and
generating a score for the task indicative of a difficulty of automating the task, wherein the score is generated using at least one value selected from the group consisting of: information identifying applications used to perform the task, a number of applications used to perform the task, a number of user interface elements used in the task, an amount of natural language input provided to the applications during the task, a number of keystrokes performed in the task, a number of clicks performed in the task, a ratio between keystrokes and clicks performed in the task, a percentage of time spent performing typing actions in the task, a percentage of time spent performing clicking actions in the task, and a frequency of copy and paste actions performed in the task.
16 . The at least one non-transitory computer readable storage medium of claim 15 , wherein the action includes at least one of a keystroke and a click.
17 . The at least one non-transitory computer readable storage medium of claim 15 , wherein gathering the contextual information includes gathering information regarding a state of the user interface when the user performed the action.
18 . The at least one non-transitory computer readable storage medium of claim 17 , wherein gathering the information regarding the state of the user interface includes gathering information selected from the group consisting of: a name of an active application, a title in a title bar of the active application, a value in at least one field of the active application, a size of a window of the active application, and a location of the window of the active application within the user interface.
19 . The at least one non-transitory computer readable storage medium of claim 15 , wherein storing the information indicative of the action and the contextual information in the volatile memory as the event comprises storing the information indicative of the action and the contextual information in the volatile memory without at least a portion of the personal information.
20 . The at least one non-transitory computer readable storage medium of claim 15 ,
wherein the at least one computer hardware processor includes a first computer hardware processor part of the computing device and a second computer hardware processor part of an external system to the computing device, and
wherein acts of receiving, gathering, storing, determining, and writing, are performed by the first computer hardware processor; and
acts of identifying the task and generating the score are performed by the second computer hardware processor.