Multimodal dialogs using large language model(s) and visual language model(s)

Inventors: Tuan Nguyen (San Jose, CA); Sergei Volnov (London, GB); William A. Truong (San Jose, CA); Yunfan Ye (Sunnyvale, CA); Sana Mithani (Plantation, FL); Neel Joshi (Newcastle, WA); Alexey Galata (San Jose, CA); Tzu-Chan Chuang (San Francisco, CA); Liang-yu Chen (Sunnyvale, CA); Qiong Huang (San Jose, CA); Krunal Shah (Mountain View, CA); Sai Aditya Chitturu (Sunnyvale, CA)

Assignee: GOOGLE LLC

G06F40/40 G06F40/30 G06F40/35 G06V10/82

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12682181
App. No.: 18/217,313
Granted: Jul 14, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Implementations relate to leveraging large language model(s) (LLMs) and vision language model(s) (VLMs) to facilitate human-to-computer dialogs. In various implementations, one or more digital images may be processed using one or more VLMs to generate VLM output indicative of a state of an environment. An LLM prompt may be assembled based on the VLM output and a natural language input. The LLM prompt may be processed using one or more LLMs to generate content that is responsive to the natural language input. The content that is responsive to the natural language input may subsequently be rendered at one or more output devices.

Claims (36)

1 . A method implemented using one or more processors and comprising:

receiving one or more digital images of an environment;

processing an ambiguous natural language input received from a user using one or more large language models (LLMs) to generate a synthetic follow-up query, wherein the synthetic follow-up query is derived from the ambiguous natural language input and seeks information about the environment of the user that is usable to resolve the ambiguous natural language input;

assembling a vision language model (VLM) prompt to include the one or more digital images and the synthetic follow-up query;

processing the VLM prompt using one or more VLMs to generate VLM output indicative of a state of the environment;

assembling an LLM prompt based on the VLM output and the ambiguous natural language input, wherein the assembling includes assembling one or more terms from the VLM output into the ambiguous natural language input to provide the LLM prompt;

processing the LLM prompt using one or more of the LLMs to generate content that is responsive to the ambiguous natural language input; and

causing the content that is responsive to the ambiguous natural language input to be rendered at one or more output devices.

2 . The method of claim 1 , wherein the synthetic follow-up query seeks information about one or more items that are present in the environment.

3 . The method of claim 2 , wherein the information comprises one or more physical attributes of one or more of the items that are present in the environment.

4 . The method of claim 2 , wherein the information comprises one or more locations of one or more of the items within the environment.

5 . The method of claim 1 , further comprising detecting, based on one or more sensor signals, that an individual is present in the environment, wherein the one or more digital images are processed, using one or more of the VLMs, in response to the detecting.

6 . The method of claim 1 , wherein the LLM prompt comprises embeddings generated from the VLM output indicative of the state of the environment and the ambiguous natural language input.

7 . The method of claim 1 , wherein the LLM prompt comprises a sequence of natural language tokens assembled from the VLM output indicative of the state of the environment and the ambiguous natural language input.

8 . A method implemented using one or more processors and comprising:

receiving an ambiguous natural language input and one or more digital images of an environment;

processing the ambiguous natural language input using one or more large language models (LLMs) to generate a synthetic follow-up query, wherein the synthetic follow-up query is derived from the ambiguous natural language input and seeks information that is usable to resolve the ambiguous natural language input;

assembling a vision language model (VLM) prompt based on the synthetic follow-up query and the one or more digital images of the environment;

processing the VLM prompt using one or more VLMs to generate VLM output indicative of a state of the environment;

assembling an LLM prompt based on the VLM output and the ambiguous natural language input, wherein the assembling includes assembling one or more aspects of the VLM output into the ambiguous natural language input to provide the LLM prompt;

processing the LLM prompt using one or more of the LLMs to generate content that is responsive to the ambiguous natural language input; and

causing the content that is responsive to the ambiguous natural language input to be rendered at one or more output devices.

9 . The method of claim 8 , wherein the synthetic follow-up query seeks information about one or more items that are present in the environment.

10 . The method of claim 9 , wherein the information comprises one or more physical attributes of one or more of the items that are present in the environment.

11 . The method of claim 9 , wherein the information comprises one or more locations of one or more of the items within the environment.

12 . A system comprising one or more processors and memory storing instructions that, in response to execution, cause the one or more processors to:

receive one or more digital images of an environment and process an ambiguous natural language input received from a user using one or more large language models (LLMs) to generate a synthetic follow-up query, wherein the synthetic follow-up query is derived from the ambiguous natural language input and seeks information about the environment of the user that is usable to resolve the ambiguous natural language input;

assemble a vision language model (VLM) prompt to include the one or more digital images of the environment of the user and the synthetic follow-up query;

process the VLM prompt using one or more VLMs to generate VLM output indicative of a state of the environment;

assemble an LLM prompt based on the VLM output and the ambiguous natural language input, wherein the instructions to assemble include instructions to assemble one or more terms from the VLM output into the ambiguous natural language input to provide the LLM prompt;

process the LLM prompt using one or more of the LLMs to generate content that is responsive to the ambiguous natural language input; and

cause the content that is responsive to the ambiguous natural language input to be rendered at one or more output devices.

13 . The system of claim 12 , wherein the synthetic follow-up query seeks information about one or more items that are present in the environment.

14 . The system of claim 13 , wherein the information comprises one or more physical attributes of one or more of the items that are present in the environment.

15 . The system of claim 13 , wherein the information comprises one or more locations of one or more of the items within the environment.

16 . The system of claim 12 , further comprising instructions to detect, based on one or more sensor signals, that an individual is present in the environment, wherein the one or more digital images are processed, using one or more of the VLMs, in response to the detecting.