AI-BASED MODELS FOR ASSISTANCE WITH NEAR-REPETITVE COMPUTER-BASED TASKS
In one aspect, a device includes a processor system and storage accessible to the processor system. The storage includes instructions executable by the processor system to track user inputs as a user interacts with a first graphical user interface (GUI) presented on a display. The instructions are also executable to execute a prediction model to identify a near-repetitive action from the user inputs. Based on the identification of the near-repetitive action, the instructions are executable to present a foreshadow action on the display. The foreshadow action is selectable to command the processor system to autonomously perform a real action corresponding to the foreshadowed action.
1 . A device, comprising:
a processor system; and
storage accessible to the processor system and comprising instructions executable by the processor system to:
track user inputs as a user interacts with a first graphical user interface (GUI) presented on a display;
execute a prediction model to identify a near-repetitive action from the user inputs; and
based on the identification of the near-repetitive action, present a foreshadow action on the display, the foreshadow action being selectable by the user to command the processor system to autonomously perform a real action corresponding to the foreshadowed action.
2 . The device of claim 1 , wherein the near-repetitive action is established by a same type of user input directed to different locations on the first GUI.
3 . The device of claim 1 , wherein the near-repetitive action is not a same repeated action at a same location on the first GUI.
4 . The device of claim 1 , wherein the instructions are executable to:
track the user inputs as the user sources data from the first GUI and inputs the data to a second GUI different from the first GUI;
execute the prediction model to identify the near-repetitive action from the user inputs sourcing the data from the first GUI and inputting the data to the second GUI; and
based on the identification of the near-repetitive action, present the foreshadow action on the display.
5 . The device of claim 4 , wherein the first GUI is associated with a first spreadsheet, and wherein the second GUI is associated with a second spreadsheet different from the first spreadsheet.
6 . The device of claim 4 , wherein the first GUI is associated with a first word processing document, and wherein the second GUI is associated with a second word processing document different from the first word processing document.
7 . The device of claim 1 , wherein the first GUI is a web-based fillable form, and wherein the instructions are executable to:
track the user inputs as the user interacts with the web-based fillable form;
execute the prediction model to identify the near-repetitive action from the user inputs, the near-repetitive action established by filling in different input fields of the web-based fillable form; and
based on the identification of the near-repetitive action, present the foreshadow action on the display.
8 . The device of claim 1 , where the user inputs are established by one or more of: keyboard inputs, mouse button selections, mouse-based cursor directional movement, trackpad inputs.
9 . The device of claim 1 , wherein the prediction model is established by at least one artificial neural network that is trained to make inferences using pattern recognition.
10 . The device of claim 1 , wherein the foreshadow action is selectable via one or more of: voice input, selection of a predetermined key on a keyboard, a predetermined mouse manipulation, selection of a selector presented on the display.
11 . The device of claim 1 , wherein the instructions are executable to:
responsive to identifying, from the user inputs, a threshold number of correct foreshadow actions, present an option that is selectable to command the processor system to auto-complete plural additional near-repetitive actions of the same type as the identified near-repetitive action but without the user having to select additional respective foreshadow actions, the threshold number being greater than one.
12 . The device of claim 11 , wherein respective foreshadow actions are classified as correct foreshadow actions based on user selection of the respective foreshadow actions.
13 . The device of claim 1 , comprising the display.
14 . A method, comprising:
tracking user inputs as a user interacts with a first graphical user interface (GUI) presented on a display;
executing a prediction model to identify, from the user inputs, a same action being directed to different locations of the first GUI; and
based on the identification of the same action being directed to different locations of the first GUI, presenting a predicted action on the display, the predicted action being selectable by the user to provide a command to a device to autonomously perform a real action corresponding to the predicted action.
15 . The method of claim 14 , wherein the same action directed to different locations of the first GUI is established by a same type of user input directed to different locations on the first GUI.
16 . The method of claim 15 , wherein the same type of user input comprises one or more of: a same type of cursor movement, a same type of mouse click, a same type of keyboard command.
17 . The method of claim 14 , wherein the same action is established by a sequence of first input to the first GUI and second input to a second GUI different from the first GUI, the first GUI associated with a first application (“app”) and the second GUI associated with a second app different from the first app.
18 . At least one computer readable storage medium (CRSM) that is not a transitory signal, the at least one CRSM comprising instructions executable by a processor system to:
track user inputs as a user interacts with a first graphical user interface (GUI) presented on a display;
execute a prediction model to infer, from the user inputs, a same action being directed to different locations of the first GUI; and
based on the inference of the same action being directed to different locations of the first GUI, present a predicted action on the display, the predicted action being selectable by the user to command the processor system to autonomously perform a real action corresponding to the predicted action.
19 . The CRSM of claim 18 , wherein the same action directed to different locations of the first GUI is established by a same type of user input directed to different spreadsheet cells on the first GUI.
20 . The CRSM of claim 18 , wherein the same action directed to different locations of the first GUI is established by a same type of user input directed to different input fields of a web-based fillable form that establishes the first GUI.