UI workflow optimization based on expected next UI interaction
In several approaches, a processor feeds an interaction by a user into a recurrent neural network (RNN) model. A processor identifies, using the RNN model trained to identify a specific workflow based on the interaction, an in-process workflow of the user. A processor predicts, using the RNN model, a next interaction of the user based on a next expected step in the in-process workflow. A processor performs at least one optimization function related to the next interaction of the user including at least one of autofocusing on a next user interface (UI) element associated with the next expected step in the in-process workflow, snapping a text cursor to the next UI element associated with the next expected step in the in-process workflow, and providing a UI prompt with a link to the next UI element associated with the next expected step in the in-process workflow.
1. A method comprising:
responsive to a recurrent neural network (RNN) model being enabled for use, inputting into the RNN model, by one or more processors, a user interaction with a user interface (UI) element of a user interface;
identifying, by the one or more processors, using the RNN model trained to identify a specific workflow based on the user interaction with the UI element of the user interface, an in-process workflow of the user;
predicting, by the one or more processors, using the RNN model, a next user interaction with the user interface based on a next expected step in the in-process workflow; and
performing, by the one or more processors, at least one optimization function related to the next user interaction with the user interface, wherein the at least one optimization function is selected from the group consisting of: autofocusing on a next UI element associated with the next expected step in the in-process workflow, snapping a text cursor to the next UI element associated with the next expected step in the in-process workflow, and providing a UI prompt with a link to the next UI element associated with the next expected step in the in-process workflow.
2. The method of claim 1 , wherein the user interaction with the UI element of the user interface is identified using task mining.
3. The method of claim 1 , further comprising:
capturing, by the one or more processors, user interactions with UI elements of the user interface using task mining;
building, by the one or more processors, one or more workflows from the captured user interactions;
training, by the one or more processors, a RNN on the one or more workflows; and
generating, by the one or more processors, the RNN model from the trained RNN for identifying a workflow the user has initiated.
4. The method of claim 3 , further comprising:
iterating, by the one or more processors, through the capturing step, the building step, the training step, and the generating step until a pre-defined confidence threshold for the RNN model is reached; and
responsive to the pre-defined confidence threshold being reached, prompting, by the one or more processors, the user that the RNN model is ready for use.
5. The method of claim 1 , further comprising:
responsive to identifying a manual override by the user to a different UI element than the next UI element associated with the next expected step of the in-process workflow, re-training, by the one or more processors, the RNN model based on the different UI element to update a set of steps of the workflow.
6. The method of claim 1 , further comprising:
enabling, by the one or more processors, the user to set up a shortcut key for triggering the at least one optimization function.
7. The method of claim 3 , wherein capturing user interactions with the UI elements of the user interface using the task mining comprises:
capturing, by the one or more processors, user clicks, text cursor movement of the user, user inputs and associated contextual data, and UI element interactions by the user; and
capturing, by the one or more processors, information associated with applications used by the user, sequence of actions performed by the user, and time taken for each action by the user.
8. A computer program product comprising:
one or more computer readable storage media and program instructions collectively stored on the one or more computer readable storage media, the stored program instructions comprising:
program instructions to, responsive to a recurrent neural network (RNN) model being enabled for use, input into the RNN a user interaction with a user interface (UI) element of a user interface;
program instructions to identify, using the RNN model trained to identify a specific workflow based on the user interaction with the UI element of the user interface, an in-process workflow of the user;
program instructions to predict, using the RNN model, a next user interaction with the user interface based on a next expected step in the in-process workflow; and
program instructions to perform at least one optimization function related to the next user interaction with the user interface, wherein the at least one optimization function is selected from the group consisting of: autofocusing on a next UI element associated with the next expected step in the in-process workflow, snapping a text cursor to the next UI element associated with the next expected step in the in-process workflow, and providing a UI prompt with a link to the next UI element associated with the next expected step in the in-process workflow.
9. The computer program product of claim 8 , wherein the user interaction with the UI element of the user interface is identified using task mining.
10. The computer program product of claim 8 , further comprising:
program instructions to capture user interactions with UI elements of the user interface using task mining;
program instructions to build one or more workflows from the captured user interactions;
program instructions to train a RNN on the one or more workflows; and
program instructions to generate the RNN model from the trained RNN for identifying a workflow the user has initiated.
11. The computer program product of claim 10 , further comprising:
program instructions to iterate through the capturing step, the building step, the training step, and the generating step until a pre-defined confidence threshold for the RNN model is reached; and
program instructions to, responsive to the pre-defined confidence threshold being reached, prompt the user that the RNN model is ready for use.
12. The computer program product of claim 8 , further comprising:
program instructions to, responsive to identifying a manual override by the user to a different UI element than the next UI element associated with the next expected step of the in-process workflow, re-train the RNN model based on the different UI element to update a set of steps of the workflow.
13. The computer program product of claim 8 , further comprising:
program instructions to enable the user to set up a shortcut key for triggering the at least one optimization function.
14. The computer program product of claim 10 , wherein the program instructions to capture user interactions with the UI elements of the user interface using the task mining comprise:
program instructions to capture user clicks, text cursor movement of the user, user inputs and associated contextual data, and UI element interactions by the user; and
program instructions to capture information associated with applications used by the user, sequence of actions performed by the user, and time taken for each action by the user.
15. A computer system comprising:
one or more computer processors;
one or more computer readable storage media;
program instructions collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the stored program instructions comprising:
program instructions to, responsive to a recurrent neural network (RNN) model being enabled for use, feed input into the RNN model a user interaction with a user interface (UI) element of a user interface;
program instructions to identify, using the RNN model trained to identify a specific workflow based on the user interaction with the UI element of the user interface, an in-process workflow of the user;
program instructions to predict, using the RNN model, a next user interaction with the user interface based on a next expected step in the in-process workflow; and
program instructions to perform at least one optimization function related to the next user interaction with the user interface, wherein the at least one optimization function is selected from the group consisting of: autofocusing on a next UI element associated with the next expected step in the in-process workflow, snapping a text cursor to the next UI element associated with the next expected step in the in-process workflow, and providing a UI prompt with a link to the next UI element associated with the next expected step in the in-process workflow.
16. The computer system of claim 15 , further comprising:
program instructions to capture user interactions with UI elements of the user interface using task mining;
program instructions to build one or more workflows from the captured user interactions;
program instructions to train a RNN on the one or more workflows; and
program instructions to generate the RNN model from the trained RNN for identifying a workflow the user has initiated.
17. The computer system of claim 16 , further comprising:
program instructions to iterate through the capturing step, the building step, the training step, and the generating step until a pre-defined confidence threshold for the RNN model is reached; and
program instructions to, responsive to the pre-defined confidence threshold being reached, prompt the user that the RNN model is ready for use.
18. The computer system of claim 15 , further comprising:
program instructions to, responsive to identifying a manual override by the user to a different UI element than the next UI element associated with the next expected step of the in-process workflow, re-train the RNN model based on the different UI element to update a set of steps of the workflow.
19. The computer system of claim 15 , further comprising:
program instructions to enable the user to set up a shortcut key for triggering the at least one optimization function.
20. The computer system of claim 16 , wherein the program instructions to capture user interactions with the UI elements of the user interface using the task mining comprise:
program instructions to capture user clicks, text cursor movement of the user, user inputs and associated contextual data, and UI element interactions by the user; and
program instructions to capture information associated with applications used by the user, sequence of actions performed by the user, and time taken for each action by the user.