Granted Patent B2

US 12,499,679 · App. 18/236,290 · Granted Dec 16, 2025

Inventors: Ibrahim Badr (Zurich, CH); Nils Grimsmo (Adliswil, CH); Gökhan Bakir (Zurich, CH)

Assignee: GOOGLE LLC

G06V20/20G06F3/167G06F16/5854G06F16/90332H04L51/02G06V20/68

Abstract

Claims (92)

1 . A method implemented by one or more processors, comprising:

receiving a natural language input that is provided by a user via an automated assistant interface of a client device;

determining, based on processing the natural language input, that the natural language input indicates a request, by the user, that is related to an object in an environment with the client device;

in response to determining that the natural language input indicates the request that is related to the object in the environment:

causing the client device to capture at least one image of the environment;

determining, based on processing of both the natural language input and the at least one image, whether the request is resolvable using the natural language input and the at least one image;

in response to determining that the request is resolvable using the natural language input and the at least one image:

resolving the request without any prompting of the user to provide additional information related to the object;

in response to determining that the request is not resolvable using the natural language input and the at least one image:

providing, for presentation via the client device or an additional client device, a prompt that is tailored to the request;

receiving, in response to the prompt, one or both of:

an additional image captured by the client device, and

voice input; and

resolving the request using the natural language input and the at least one image, and further based on one or both of:

the additional image received in response to the prompt, and

the voice input received in response to the prompt; and

causing output, that reflects the resolution of the request, to be rendered at the client device or the additional client device.

2 . The method of claim 1 , wherein the additional image is received in response to the prompt, and wherein resolving the request is based on the additional image.

3 . The method of claim 1 , wherein the voice input is received in response to the prompt, and wherein resolving the request is based on the voice input.

4 . The method of claim 1 , wherein the additional image and the voice input are received in response to the prompt, and wherein resolving the request is based on the additional image and is based on the voice input.

5 . The method of claim 1 , wherein resolving the request comprises:

resolving attributes of the object;

issuing a query based on the attributes;

receiving one or more results that are responsive to the issued query; and

generating the output based on at least one result, of the received one or more results.

6 . The method of claim 5 , wherein issuing the query based on the attributes comprises:

transmitting the query to an agent over one or more networks; and

receiving the one or more results that are responsive to the query from the agent in response to transmitting the query to the agent.

7 . The method of claim 6 , further comprising:

selecting the agent from a plurality of available agents;

wherein transmitting the query to the agent is based on selecting the agent from the plurality of available agents.

8 . A system, comprising:

memory storing instructions;

one or more processors operable to execute the instructions to:

receive a natural language input provided by a user via an automated assistant interface of a client device;

determine, based on processing the natural language input, that the natural language input indicates a request, by the user, that is related to an object in an environment with the client device;

in response to determining that the natural language input indicates the request that is related to the object in the environment:

cause the client device to capture at least one image of the environment;

determine, based on processing of both the natural language input and the at least one image, whether the request is resolvable using the natural language input and the at least one image;

in response to determining that the request is resolvable using the natural language input and the at least one image:

resolve the request without any prompting of the user to provide additional information related to the object;

in response to determining that the request is not resolvable using the natural language input and the at least one image:

provide, for presentation via the client device or an additional client device, a prompt that is tailored to the request;

receive, in response to the prompt, one or both of:

 an additional image captured by the client device, and

 voice input; and

resolve the request using the natural language input and the at least one image, and further based on one or both of:

 the additional image received in response to the prompt, and

 the voice input received in response to the prompt; and

cause output, that reflects the resolution of the request, to be rendered at the client device or the additional client device.

9 . The system of claim 8 , wherein the additional image is received in response to the prompt, and wherein resolving the request is based on the additional image.

10 . The system of claim 8 , wherein the voice input is received in response to the prompt, and wherein resolving the request is based on the voice input.

11 . The system of claim 8 , wherein the additional image and the voice input are received in response to the prompt, and wherein resolving the request is based on the additional image and is based on the voice input.

12 . The system of claim 8 , wherein in resolving the request one or more of the processors are to:

resolve attributes of the object;

issue a query based on the attributes;

receive one or more results that are responsive to the issued query; and

generate the output based on at least one result, of the received one or more results.

13 . The system of claim 12 , wherein in issuing the query based on the attributes one or more of the processors are to:

transmit the query to an agent over one or more networks; and

receive the one or more results that are responsive to the query from the agent in response to transmitting the query to the agent.

14 . The system of claim 13 , wherein one or more of the processors are further operable to execute the instructions to:

select the agent from a plurality of available agents;

wherein transmitting the query to the agent is based on selecting the agent from the plurality of available agents.

15 . One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause performance of a method comprising:

receiving a natural language input that is provided by a user via an automated assistant interface of a client device;

determining, based on processing the natural language input, that the natural language input indicates a request, by the user, that is related to an object in an environment with the client device;

in response to determining that the natural language input indicates the request that is related to the object in the environment:

causing the client device to capture at least one image of the environment;

determining, based on processing of both the natural language input and the at least one image, whether the request is resolvable using the natural language input and the at least one image;

in response to determining that the request is resolvable using the natural language input and the at least one image:

resolving the request without any prompting of the user to provide additional information related to the object;

in response to determining that the request is not resolvable using the natural language input and the at least one image:

providing, for presentation via the client device or an additional client device, a prompt that is tailored to the request;

receiving, in response to the prompt, one or both of:

an additional image captured by the client device, and

voice input; and

resolving the request using the natural language input and the at least one image, and further based on one or both of:

the additional image received in response to the prompt, and

the voice input received in response to the prompt; and

causing output, that reflects the resolution of the request, to be rendered at the client device or the additional client device.

16 . The non-transitory computer readable media of claim 15 , wherein the additional image is received in response to the prompt, and wherein resolving the request is based on the additional image.

17 . The non-transitory computer readable media of claim 15 , wherein the voice input is received in response to the prompt, and wherein resolving the request is based on the voice input.

18 . The non-transitory computer readable media of claim 15 , wherein the additional image and the voice input are received in response to the prompt, and wherein resolving the request is based on the additional image and is based on the voice input.

19 . The non-transitory computer readable media of claim 15 , wherein resolving the request comprises:

resolving attributes of the object;

issuing a query based on the attributes;

receiving one or more results that are responsive to the issued query; and

generating the output based on at least one result, of the received one or more results.

20 . The non-transitory computer readable media of claim 19 , wherein issuing the query based on the attributes comprises:

transmitting the query to an agent over one or more networks; and

receiving the one or more results that are responsive to the query from the agent in response to transmitting the query to the agent.

Assignments (2)

Continuity (5)

References Cited (71)