IP Library › Granted Patent US 12,576,523

Granted Patent B2

US 12,576,523 · App. 19/008,421 · Granted Mar 17, 2026

Controlling robots using multi-modal language models

Inventors: Peter Raymond Florence (San Francisco, CA); Danny Michael Driess (Berlin, DE); Igor Mordatch (Oakland, CA); Andy Zeng (Stanford, CA); Seyed Mohammad Mehdi Sajjadi (Berlin, DE); Klaus Greff (Berlin, DE)

Assignee: Google LLC

B25J9/1658

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,576,523
App. No.: 19/008,421
Granted: Mar 17, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling an agent interacting with an environment. In one aspect, a method comprises: receiving one or more observations of an environment; receiving an input text sequence that describes a task to be performed by a robot in the environment; generating an encoded representation of the input text sequence in an embedding space; generating a corresponding encoded representation of each of the one or more observations in the embedding space; generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation; processing the sequence of input tokens using a language model neural network to generate an output text sequence that comprises high-level natural language instructions; and determining, from the high-level natural language instructions, one or more actions to be performed by the robot.

Claims (62)

1 . A computer-implemented method for controlling a robot to perform a task, the method comprising:

receiving one or more observations that characterize a state of an environment at a time step, wherein the environment that comprises one or more objects;

receiving an input text sequence that describes the task to be performed by the robot in the environment and that references the one or more objects in the environment;

generating an encoded representation of the input text sequence in an embedding space;

generating a corresponding encoded representation of each of the one or more observations in the embedding space;

generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation;

processing the sequence of input tokens using a language model neural network to autoregressively generate an output text sequence that specifies a high-level plan for the robot at the time step, wherein the high-level plan at the time step defines a plurality of operations to be performed by the robot and an order of performance of the plurality of operations by the robot in response to the one or more observations that characterize the state of the environment at the time step;

processing at least the output text sequence using a policy neural network to map the high-level plan to one or more actions to be performed by robot hardware of the robot in response to the one or more observations that characterize the state of the environment at the time step; and

controlling the robot to perform the task based on the one or more actions.

2 . The method of claim 1 , wherein receiving the one or more observations of the environment comprises:

receiving a first state vector that represents one or more of a pose, size, or color of an object; and

receiving a second state vector that represents a pose of the robot.

3 . The method of claim 1 , wherein receiving the one or more observations of the environment comprises:

receiving an image that characterizes the environment.

4 . The method of claim 1 , wherein generating the corresponding encoded representation of each of the one or more observations comprises:

processing each of the one or more observations using an encoder neural network, a projector neural network, or both.

5 . The method of claim 4 , wherein the encoder neural network is configured as an image encoder neural network, wherein the projector neural network is configured as a multilayer perceptron (MLP), and wherein processing each of the one or more observations comprises:

processing an input comprising an image using the image encoder neural network to generate an image encoder embedding of the image; and

processing the image encoder embedding of the image using the MLP to generate an encoded representation of the image.

6 . The method of claim 5 , wherein the image encoder neural network is configured as a convolutional neural network, a Transformer neural network, or a vision Transformer (ViT) neural network.

7 . The method of claim 5 , wherein the input comprises the image and an object instance mask that identifies different portions of the image as each containing a corresponding object.

8 . The method of claim 6 , wherein the image encoder neural network comprises a slot mixer decoder that computes an output by applying an attention mechanism over multiple slots that each represent a respective portion of the image using one or more query rays having predetermined directions.

9 . The method of claim 1 , wherein the corresponding encoded representation of each observation is interleaved in between the encoded representation of the input text sequence in the sequence of input tokens.

10 . The method of claim 1 , wherein generating the sequence of input tokens comprises:

assigning an index to each of the one or more objects included in the environment;

generating an encoded representation of a prefix that defines a mapping between the one or more objects and the assigned indices; and

generating a sequence of input tokens that comprises the prefix.

11 . The method of claim 1 , wherein generating the encoded representation of the input text sequence comprises:

determining a respective product of each one-het encoded vector in an ene het encoded representation of the input text sequence and an embedding matrix having values that have been learned jointly with the language model neural network.

12 . The method of claim 1 , wherein the robot hardware of the robot comprises an actuator, a motor, a driver, or a gripper.

13 . The method of claim 1 , wherein processing at least the output text sequence using the policy neural network to map the high-level plan to the one or more actions to be performed by the robot hardware of the robot comprises:

processing at least the high-level plan natural language instructions using the policy neural network to generate one or more policy network outputs; and

selecting, in accordance with the one or more policy network outputs, the one or more actions to be performed by the robot hardware of the robot from a space of possible actions.

14 . A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for controlling a robot to perform a task, the operations comprising:

receiving one or more observations that characterize a state of an environment at a time step, wherein the environment that-comprises one or more objects;

receiving an input text sequence that describes the task to be performed by the robot in the environment and that references the one or more objects in the environment;

generating an encoded representation of the input text sequence in an embedding space;

generating a corresponding encoded representation of each of the one or more observations in the embedding space;

generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation;

controlling the robot to perform the task based on the one or more actions.

15 . The system of claim 14 , wherein receiving the one or more observations of the environment comprises:

receiving a first state vector that represents one or more of a pose, size, or color of an object; and

receiving a second state vector that represents a pose of the robot.

16 . The system of claim 14 , wherein receiving the one or more observations of the environment comprises:

receiving an image that characterizes the environment.

17 . The system of claim 14 , wherein generating the corresponding encoded representation of each of the one or more observations comprises:

processing each of the one or more observations using an encoder neural network, a projector neural network, or both.

18 . The system of claim 17 , wherein the encoder neural network is configured as an image encoder neural network, wherein the projector neural network is configured as a multilayer perceptron (MLP), and wherein processing each of the one or more observations comprises:

processing an input comprising an image using the image encoder neural network to generate an image encoder embedding of the image; and

processing the image encoder embedding of the image using the MLP to generate an encoded representation of the image.

19 . The system of claim 18 , wherein the image encoder neural network is configured as a convolutional neural network, a Transformer neural network, or a vision Transformer (ViT) neural network.

20 . A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for controlling a robot to perform a task, the operations comprising:

receiving one or more observations that characterize a state of an environment at a time step, wherein the environment that comprises one or more objects;

receiving an input text sequence that describes the task to be performed by the robot in the environment and that references the one or more objects in the environment;

generating an encoded representation of the input text sequence in an embedding space;

generating a corresponding encoded representation of each of the one or more observations in the embedding space;

generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation;

controlling the robot to perform the task based on the one or more actions.

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 27, 2025

From: FLORENCE, PETER RAYMOND; DRIESS, DANNY MICHAEL; MORDATCH, IGOR; ZENG, ANDY; SAJJADI, SEYED MOHAMMAD MEHDI; GREFF, KLAUS

To: GOOGLE LLC

Reel/Frame 071551/0048 →

Continuity (3)

Continuation PCTUS2024014283 · Feb 2, 2024

Provisional Application 63443002 · Feb 2, 2023

Related Publication 20250144795A1 · May 8, 2025

References Cited (71)

US 11354509B2 · Hermann et al. · 2022 [cited by applicant]

US 20230178076A1 · Abramson · 2023 [cited by examiner]

Li, Shuang, et al. “Pre-Trained Language Models for Interactive Decision-Making.” (2022) (Year: 2022). [cited by examiner]

Jiang, Yunfan, et al. “Vima: General robot manipulation with multimodal prompts.” (2022) (Year: 2022). [cited by examiner]

Guhur, Pierre-Louis, et al. “Instruction-driven history-aware policies for robotic manipulations.” (2022) (Year: 2022). [cited by examiner]

Sajjadi, Mehdi SM, et al. “Object scene representation transformer.” (2022) (Year: 2022). [cited by examiner]

Tsimpoukelli, Maria, et al. “Multimodal few-shot learning with frozen language models.” (2021) (Year: 2021). [cited by examiner]

Jeet. “One Hot encoding of text data in Natural Language Processing.” Medium, Aug. 14, 2020, medium.com/analytics-vidhya/one-hot-encoding-of-text-data-in-natural-language-processing-2242fefb2148 (Year: 2020). [cited by examiner]

Ahn et al., “Do as I can, not as I say: Grounding language in robotic affordances,” CoRR, Submitted on Aug. 16, 2022, arXiv:2204.01691v2, pp. 1-34. [cited by applicant]

Alayrac et al., “Flamingo: a visual language model for few-shot learning,” CoRR, Submitted on Apr. 29, 2022, arXiv:2204.14198v1, pp. 1-66. [cited by applicant]

Alayrac et al., “Flamingo: a visual language model for few-shot learning,” CoRR, Submitted on Nov. 15, 2022, arXiv:2204.14198v2, pp. 1-54. [cited by applicant]

Bommasani et al., “On the Opportunities and Risks of Foundation Models,” CoRR, Submitted on Aug. 18, 2021, arXiv:2108.07258v2, pp. 1-212. [cited by applicant]

Brohan et al., “RT-1: Robotics Transformer for Real-World Control at Scale”, CoRR, Submitted on Dec. 13, 2022, arXiv:2212.06817v1, pp. 1-31. [cited by applicant]

Brown et al., “Language models are few-shot learners,” Paper, Presented at the 34th Conference on Neural Information Processing Systems, Virtual Event, Dec. 6-12, 2020; Advances in Neural Information Processing Systems … [cited by applicant]

Changpinyo et al., “All You May Need for VQA are Image Captions,” CoRR, Submitted on May 4, 2022, arXiv:2205.01883v1, 17 pages. [cited by applicant]

Chen et al., “Evaluating large language models trained on code,” CoRR, Submitted on Jul. 7, 2021, arXiv:2107.03374v1, 35 pages. [cited by applicant]

Chen et al., “Microsoft COCO Captions: Data Collection and Evaluation Server,” CoRR, Submitted on Apr. 3, 2015, arXiv:1504.00325v2, pp. 1-7. [cited by applicant]

Chen et al., “PaLI: A jointly-scaled multilingual language-image model,” CoRR, Submitted on Sep. 14, 2022, arXiv:2209.06794v1, pp. 1-30. [cited by applicant]

Chen et al., “PaLI: A jointly-scaled multilingual language-image model,” CoRR, Submitted on Sep. 16, 2022, arXiv:2209.06794v2, pp. 1-30. [cited by applicant]

Chen et al., “Pix2seq: A language modeling framework for object detection,” CoRR, Submitted on Sep. 22, 2021, arXiv:2109.10852v1, pp. 1-16. [cited by applicant]

Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways,” CoRR, Submitted on Oct. 5, 2020, arXiv:2204.02311v5, pp. 1-87. [cited by applicant]

Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways,” CoRR, Submitted on Sep. 29, 2020, arXiv:2204.02311v4, pp. 1-87. [cited by applicant]

Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” CoRR, Submitted on Oct. 11, 2018, arXiv:1810.04805v1, 14 pages. [cited by applicant]

Dosovitskiy et al., “An image is worth 16×16 words: Transformers for image recognition at scale,” CoRR, Submitted on Oct. 22, 2020, arXiv:2010.11929v1, pp. 1-21. [cited by applicant]

Driess et al., “Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image,” Paper, Presented at the Robotics: Science and Systems, Virtual Event, Jul. 12-16, 20… [cited by applicant]

Glaese et al., “Improving alignment of dialogue agents via targeted human judgements,” CoRR, Submitted on Sep. 28, 2022, arXiv:2209.14375v1, pp. 1-77. [cited by applicant]

Goyal et al., “Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering,” Paper, Presented at the Conference on Computer Vision and Pattern Recognition, Honolulu, HI, Jul. 21-26… [cited by applicant]

Guhur et al., “Instruction-driven history-aware policies for robotic manipulations,” CoRR, Submitted on Dec. 17, 2022, arXiv:2209.04899v3, pp. 1-19. [cited by applicant]

Guhur et al., “Instruction-driven history-aware policies for robotic manipulations,” CoRR, Submitted on Sep. 22, 2022, arXiv:2209.04899v2, pp. 1-25. [cited by applicant]

Hu et al., “Scaling up vision-language pre-training for image captioning,” Paper, Presented at the Conference on Computer Vision and Pattern Recognition, New Orleans, Jun. 18-24, 2022; Proceedings of the Conference on C… [cited by applicant]

Huang et al., “Inner Monologue: Embodied Reasoning through Planning with Language Models,” CoRR, Submitted on Jul. 12, 2022, arXiv:2207.05608v1, pp. 1-25. [cited by applicant]

Huang et al., “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” CoRR, Submitted on Jan. 18, 2022, arXiv:2201.07207v1, pp. 1-32. [cited by applicant]

International Search Report and Written Opinion in International Appln. No. PCT/US2024/014283, mailed on May 6, 2024, 14 pages. [cited by applicant]

Jang et al., “BC-Z: Zero-shot task generalization with robotic imitation learning,” Paper, Presented at the Conference on Robot Learning, London, UK, Nov. 8-11, 2021; Proceedings of the 5th Conference on Robot Learning,… [cited by applicant]

Jiang et al., “VIMA: General Robot Manipulation with Multimodal Prompts,” CoRR, Submitted on Oct. 6, 2022, arXiv:2210.03094v1, pp. 1-40. [cited by applicant]

Kalashnikov et al., “Scalable deep reinforcement learning for vision-based robotic manipulation,” Paper, Presented at the Conference on Robot Learning, Zurich, Switzerland, Oct. 29-31, 2018; Proceedings of the 2nd Confe… [cited by applicant]

Kojima et al., “Large language models are zero-shot reasoners,” CoRR, Submitted on Jun. 9, 2022, arXiv:2205.11916v2, pp. 1-40. [cited by applicant]

Kojima et al., “Large language models are zero-shot reasoners,” CoRR, Submitted on May 24, 2022, arXiv:2205.11916v1, pp. 1-36. [cited by applicant]

Lester et al., “The Power of Scale for Parameter-Efficient Prompt Tuning,” CoRR, Submitted on Sep. 2, 2021, arXiv:2104.08691v2, 15 pages. [cited by applicant]

Lewkowycz et al., “Solving quantitative reasoning problems with language models,” CoRR, Submitted on Jul. 1, 2022, arXiv:2206.14858v2, pp. 1-54. [cited by applicant]

Li et al., “Pre-trained language models for interactive decision-making,” CoRR, Submitted on Oct. 29, 2022, arXiv:2202.01771v4, pp. 1-23. [cited by applicant]

Li et al., “TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models,” CoRR, Submitted on Sep. 25, 2021, arXiv:2109.10282v3, 10 pages. [cited by applicant]

Liu et al., “Instruction-Following Agents with Jointly Pre-Trained Vision-Language Models,” CoRR, Submitted on Oct. 24, 2022, arXiv:2210.13431v1, pp. 1-15. [cited by applicant]

Lynch et al., “Grounding Language in Play,” CoRR, Submitted on May 15, 2020, arXiv:2005.07648v1, pp. 1-23. [cited by applicant]

Lynch et al., “Interactive Language: Talking to Robots in Real Time,” CoRR, Submitted on Oct. 12, 2022, arXiv:2210.06407v1, 11 pages. [cited by applicant]

Lynch et al., “Language Conditioned Imitation Learning over Unstructured Data,” CoRR, Submitted on Jul. 7, 2021, arXiv:2005.07648v2, 18 pages. [cited by applicant]

Marino et al., “OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge,” Paper, Presented at the Conference on Computer Vision and Pattern Recognition, Long Beach, CA, Jun. 15-20, 2019; Proceedings o… [cited by applicant]

Nair et al., “Learning language-conditioned robot behavior from offline data and crowd-sourced annotation,” Paper, Presented at the Conference on Robot Learning, London, UK, Nov. 8-11, 2021; Proceedings of the 5th Confe… [cited by applicant]

Piergiovanni et al., “Pre-training image-language transformers for open-vocabulary tasks,” CoRR, Submitted on Sep. 9, 2022, arXiv:2209.04372v1, 5 pages. [cited by applicant]

Polu et al., “Formal Mathematics Statement Curriculum Learning,” CoRR, Submitted on Feb. 3, 2022, arXiv:2202.01344v1, 26 pages. [cited by applicant]

Reed et al., “A Generalist Agent,” CoRR, Submitted on May 19, 2022, arXiv:2205.06175v2, pp. 1-40. [cited by applicant]

Reed et al., “A Generalist Agent,” CoRR, Submitted on Nov. 11, 2022, arXiv:2205.06175v3, pp. 1-42. [cited by applicant]

Ryoo et al., “TokenLearner: What can 8 learned tokens do for images and videos?,” CoRR, Submitted on Dec. 7, 2021, arXiv:2106.11297v3, pp. 1-21. [cited by applicant]

Ryoo et al., “TokenLearner: What can 8 learned tokens do for images and videos?,” CoRR, Submitted on Jun. 21, 2021, arXiv:2106.11297v1, pp. 1-15. [cited by applicant]

Ryoo et al., “TokenLearner: What can 8 learned tokens do for images and videos?,” CoRR, Submitted on Oct. 5, 2021, arXiv:2106.11297v2, pp. 1-21. [cited by applicant]

Sajjadi et al., “Object Scene Representation Transformer,” Paper, Presented at the 36th Conference on Neural Information Processing Systems, New Orleans, LA, Nov. 28-Dec. 9, 2022; Advances in Neural Information Processi… [cited by applicant]

Sharma et al., “Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning,” Paper, Presented at the Annual Meeting of the Association for Computational Linguistics, Melbourne, Aus… [cited by applicant]

Sharma et al., “Skill Induction and Planning with Latent Language,” CoRR, Submitted on Oct. 4, 2021, arXiv:2110.01517v1, 13 pages. [cited by applicant]

Shridhar et al., “CLIPORT: What and Where Pathways for Robotic Manipulation,” Paper, Presented at the Conference on Robot Learning, London, UK, Nov. 8-11, 2021; Proceedings of the 5th Conference on Robot Learning, 2022,… [cited by applicant]

Shridhar et al., “Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation,” CoRR, Submitted on Nov. 11, 2022, arXiv:2209.05451v2, pp. 1-28. [cited by applicant]

Silva et al., “LanCon-Learn: Learning With Language to Enable Generalization in Multi-Task Manipulation,” IEEE Robotics and Automation Letters, Apr. 2022, 7(2):1-8. [cited by applicant]

Singh et al., “ProgPrompt: Generating situated robot task plans using large language models,” Paper, Presented at the 36th Conference on Neural Information Processing Systems, New Orleans, LA, Nov. 28-Dec. 9, 2022; Auto… [cited by applicant]

Tellex et al., “Robots that use language,” Annual Review of Control, Robotics, and Autonomous Systems, May 2020, 3:25-55. [cited by applicant]

Thoppilan et al., “LaMDA: Language Models for Dialog Applications,” CoRR, Submitted on Feb. 10, 2022, arXiv:2201.08239v3, pp. 1-47. [cited by applicant]

Tsimpoukelli et al., “Multimodal few-shot learning with frozen language models,” Paper, Presented at the 35th Conference on Neural Information Processing Systems, Virtual Event, Dec. 6-14, 2021; Advances in Neural Infor… [cited by applicant]

Wei et al., “Chain of thought prompting elicits reasoning in large language models,” CoRR, Submitted on Oct. 10, 2022, arXiv:2201.11903v5, pp. 1-43. [cited by applicant]

Xiao et al., “Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models,” CoRR, Submitted on Nov. 22, 2022, arXiv:2211.11736, pp. 1-20. [cited by applicant]

Zeng et al., “Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language,” CoRR, Submitted on May 27, 2022, arXiv:2204.00598v2, pp. 1-30. [cited by applicant]

Zhang et al., “Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring,” CoRR, Submitted on Jun. 7, 2021, arXiv:2106.03427v1, 12 pages. [cited by applicant]

Zhou et al., “Unified Vision-Language Pre-Training for Image Captioning and VQA,” Paper, Presented at the Conference on Artificial Intelligence, New York, NY, Feb. 7-12, 2020; Proceedings of the AAAI Conference on Artif… [cited by applicant]

International Preliminary Report on Patentability in International Appln. No. PCT/US2024/014283, mailed on Aug. 14, 2025, 7 pages. [cited by applicant]