Granted Patent B1

US 12,650,865 · App. 19/283,678 · Granted Jun 9, 2026

Inventors: Anshuman Sahu (Los Gatos, CA); Diego Felix de Almeida (San Jose, CA); Sharat Jagannath (San Jose, CA); Clifford Green (San Diego, CA); Kelsey Rae Conophy (Gallatin Gateway, MT); Katelyn Elizabeth Swift-Spong (San Francisco, CA); Sonia Sharma (Fremont, CA); Anmol Joshi (San Diego, CA); Pavel Rakhlin (San Francisco, CA); Shivakumara Narayanaswamy (Mountain View, CA); Trever Thomas Cullen (San Diego, CA); Ziv Heimlich Shtacher (Hoboken, NJ)

Assignee: Intuit Inc.

G06F9/453G06F3/0484G06F16/3326G06F40/174

Abstract

Claims (60)

1 . A method for assisting a user of an online resource, the method performed by one or more processors associated with the online resource and comprising:

receiving, from a user over a communications network coupled to the online resource, a request for an automated assistant associated with the online resource to provide step-by-step instructions for completing a task by the user, wherein the task includes a sequence of steps corresponding to a plurality of user interface (UI) elements presented on a display screen associated with the user;

establishing, over the communications network, a user interaction between the user and an automated assistant based on the request;

presenting, on the display screen via the communications network, an assistant interface associated with the user interaction;

generating instructions for the user to perform each step of the sequence of steps of the task based at least in part on transcripts of one or more previous user interactions with the automated assistant;

providing, to the user over the communications network, the instructions in the assistant interface presented on the display screen;

for each step of the task:

determining that the user has navigated to and selected the UI element corresponding to the step based at least in part on the user's cursor position on the display screen;

determining that input data obtained from the user via the selected UI element results in completion of the step; and

generating, on the display screen via the communications network, animation indicating a next UI element corresponding to the next step of the task to be selected by the user, wherein an orientation of the animation is based on the user's cursor position relative to a location of the next UI element presented on the display screen;

determining whether or not user actions associated with the plurality of UI elements presented on the display screen result in completion of the sequence of steps of the task; and

indicating, to the user via the assistant interface presented on the display screen, completion of the task in response to determining that the user actions resulted in the completion of the sequence of steps.

2 . The method of claim 1 , wherein the instructions are further based on a mapping between the sequence of steps and the plurality of UI elements presented on the display screen.

3 . The method of claim 2 , wherein the mapping is based on a graph of nodes interconnected by branches, each node associated with instructions for a corresponding step of the sequence of steps, and each branch originating at a respective node is associated with a different user action associated with the UI element corresponding to the respective node.

4 . The method of claim 3 , wherein generating the instructions includes:

determining which of the plurality of UI elements presented on the display screen is currently selected by the user;

identifying a sequence of nodes in the graph of nodes based on the step corresponding to the UI element currently selected by the user; and

sequentially providing, on the assistant interface presented on the display screen, the instructions associated with the identified sequence of nodes.

5 . The method of claim 1 , further comprising:

determining that a user action associated with a respective UI element does not result in completion of the step corresponding to the respective UI element;

updating the instructions for the corresponding step based at least in part on the user action associated with the respective UI element; and

providing, to the user over the communications network, the updated instructions in the assistant interface presented on the display screen.

6 . The method of claim 5 , further comprising:

determining that the corresponding step cannot be completed by the user based on the updated instructions;

selecting an expert based on a type of user input data associated with the corresponding step; and

initiating a conversation between the user and the selected expert over the communications network, the conversation based at least in part on the user interaction between the user and the automated assistant.

7 . The method of claim 6 , further comprising:

generating a transcript of the conversation between the user and the selected expert in response to determining that the user completed the corresponding step during the conversation; and

combining the transcript of the conversation with portions of transcripts of previous user interactions associated with the corresponding step.

8 . A computing system associated with an online resource, the computing system comprising:

one or more processors; and

a memory communicatively coupled with the one or more processors and storing program code that, when executed by the one or more processors, causes the computing system to:

receive, from a user over a communications network coupled to the online resource, a request for an automated assistant associated with the online resource to provide step-by-step instructions for completing a task by the user, wherein the task includes a sequence of steps corresponding to a plurality of user interface (UI) elements presented on a display screen associated with the user;

establish, over the communications network, a user interaction between the user and an automated assistant based on the request;

present, on the display screen via the communications network, an assistant interface associated with the automated assistant;

generate instructions for the user to perform each step of the sequence of steps of the task based at least in part on transcripts of one or more previous user interactions with the automated assistant;

provide, to the user over the communications network, the instructions in the assistant interface presented on the display screen;

for each step of the task:

determining that the user has navigated to and selected the UI element corresponding to the step based at least in part on the user's cursor position on the display screen;

determining that input data obtained from the user via the selected UI element results in completion of the step; and

generating, on the display screen via the communications network, animation indicating a next UI element corresponding to the next step of the task to be selected by the user, wherein an orientation of the animation is based on the user's cursor position relative to a location of the next UI element presented on the display screen;

determine whether or not user actions associated with the plurality of UI elements presented on the display screen result in completion of the sequence of steps of the task; and

indicate, to the user via the assistant interface presented on the display screen, completion of the task in response to determining that the user actions resulted in the completion of the sequence of steps.

9 . The computing system of claim 8 , wherein the instructions are further based on a mapping between the sequence of steps and the plurality of UI elements presented on the display screen.

10 . The computing system of claim 9 , wherein the mapping is based on a graph of nodes interconnected by branches, each node associated with instructions for a corresponding step of the sequence of steps, and each branch originating at a respective node is associated with a different user action associated with the UI element corresponding to the respective node.

11 . The computing system of claim 10 , wherein execution of the program code to generate the instructions causes the computing system to:

determine which of the plurality of UI elements presented on the display screen is currently selected by the user;

identify a sequence of nodes in the graph of nodes based on the step corresponding to the UI element currently selected by the user; and

sequentially provide, on the assistant interface presented on the display screen, the instructions associated with the identified sequence of nodes.

12 . The computing system of claim 8 , wherein execution of the program code further causes the computing system to:

determine that a user action associated with a respective UI element does not result in completion of the step corresponding to the respective UI element;

update the instructions for the corresponding step based at least in part on the user action associated with the respective UI element; and

provide, to the user over the communications network, the updated instructions in the assistant interface presented on the display screen.

13 . The computing system of claim 12 , wherein execution of the program code further causes the computing system to:

determine that the corresponding step cannot be completed by the user based on the updated instructions;

select an expert based on a type of user input data associated with the corresponding step; and

initiate a conversation between the user and the selected expert over the communications network, the conversation based at least in part on the user interaction between the user and the automated assistant.

14 . The computing system of claim 13 , wherein execution of the program code further causes the computing system to:

generate a transcript of the conversation between the user and the selected expert in response to determining that the user completed the corresponding step during the conversation; and

combine the transcript of the conversation with portions of transcripts of previous user interactions associated with the corresponding step.

Assignments (1)

References Cited (104)