IP Library › Granted Patent US 12,675,541
Granted Patent B2
US 12,675,541 · App. 18/584,580 · Granted Jul 7, 2026

Multimodal web navigation

Inventors: Hiroki Furuta (Tokyo, JP); Ofir Nachum (San Francisco, CA); Kuang-Huei Lee (San Francisco, CA); Izzeddin Gur (San Jose, CA); Shixiang Gu (San Francisco, CA)
Assignee: Google LLC
G06F16/954G06F40/284G06V30/413
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,675,541
App. No.
18/584,580
Filed
Feb 22, 2024
Granted
Jul 7, 2026
Kind
B2
Art Unit
2171
USPC
715/738
Abstract

Machine learning models for autonomously navigating the web and/or apps are provided that receive (i) visual input that includes one or more images representing the current, and optionally one or more prior, visual states of the website or app being navigated, and (ii) textual input that represents a user request to be implemented by the model, the current HTML or other code defining the website or app, and optionally additional text like prior model outputs that represent prior steps in the navigation of the website or app by the model. The visual input and textual input are transformed into respective sets of visual and textual tokens, and this multi-modal token set is applied to an encoder. The output of the encoder is applied to a decoder to generate textual output that represents an action to take, relative to the code defining the website or app, to pursue the user request.

Claims (53)

1 . A computer-implemented method comprising:

applying a first image of a website to a first machine learning model to generate a first set of visual tokens that represents the contents of the first image;

transforming a first textual input to generate a first set of textual tokens that represents the contents of the first textual input, wherein the first textual input includes at least a first portion of a code defining the website and a command to perform a task using the website;

applying the first set of visual tokens and the first set of textual tokens to a second machine learning model to generate a first intermediate output, wherein the second machine learning model comprises an encoder; and

applying the first intermediate output to a third machine learning model to generate a first command output, wherein the first command output represents an action to take relative to the code defining the website to effect the task, and wherein the third machine learning model comprises a decoder.

2 . The computer-implemented method of claim 1 , wherein the first textual input additionally comprises a previous command output of the third machine learning model, wherein a command based on the first command output was transmitted to a server hosting the website to result in the generation of the first image of the website and the first portion of code defining the website.

3 . The computer-implemented method of claim 1 , further comprising:

receiving, from a server hosting the website, the first portion of the code;

generating the first image of the website based on at least the first portion of the code; and

transmitting, to the server, a command based on the first command output.

4 . The computer-implemented method of claim 3 , wherein the first command output specifies a target object within the first portion of the code and an action to take toward the target object.

5 . The computer-implemented method of claim 4 , wherein the first command output specifies that the action to take toward the target object is one of clicking the target object or inputting text to the target object.

6 . The computer-implemented method of claim 3 , further comprising, subsequent to transmitting the command based on the first command output transmitting to the server:

receiving, from the server, a second portion of a code defining the website;

generating a second image of the website based on at least the second portion of the code;

applying the second image of the website to the first machine learning model to generate a second set of visual tokens that represents the contents of the second image;

transforming a second textual input to generate a second set of textual tokens that represents the contents of the second textual input, wherein the second textual input includes at least the first portion of the code, the command to perform the task using the website, the second portion of the code, and the first command output;

apply the first set of visual tokens, the second set of visual tokens, and the second set of textual tokens to the second machine learning model to generate a second intermediate output;

applying the second intermediate output to the third machine learning model to generate a second command output, wherein the second command output represents an action to take relative to the code defining the website to effect the task; and

transmitting, to the server, a command based on the second command output.

7 . The computer-implemented method of claim 1 , wherein the first textual input additionally includes at least one past command output generated by the third machine learning model prior to generating the first command output.

8 . The computer-implemented method of claim 1 , wherein the first, second, and third machine learning models were trained using a training dataset that included a plurality of sequences of actions, with each sequence of actions corresponding to a respective command to perform a task using a respective website, wherein each action of a particular sequence is associated with a respective image of the website for the particular sequence prior to taking the action.

9 . The computer-implemented method of claim 8 , wherein the second machine learning model and third machine learning model were pre-trained as an instruction-finetuned language model prior to being trained using the training dataset.

10 . The computer-implemented method of claim 1 , wherein the first machine learning model comprises a vision transformer, and wherein the second machine learning model and third machine learning model were at least partially pre-trained as an instruction-finetuned language model.

11 . A non-transitory computer readable medium having stored thereon program instructions executable by at least one processor to cause the at least one processor to perform a method comprising:

applying a first image of a website to a first machine learning model to generate a first set of visual tokens that represents the contents of the first image;

transforming a first textual input to generate a first set of textual tokens that represents the contents of the first textual input, wherein the first textual input includes at least a first portion of a code defining the website and a command to perform a task using the website;

applying the first set of visual tokens and the first set of textual tokens to a second machine learning model to generate a first intermediate output, wherein the second machine learning model comprises an encoder; and

applying the first intermediate output to a third machine learning model to generate a first command output, wherein the first command output represents an action to take relative to the code defining the website to effect the task, and wherein the third machine learning model comprises a decoder.

12 . The non-transitory computer readable medium of claim 11 , wherein the first textual input additionally comprises a previous command output of the third machine learning model, wherein a command based on the first command output was transmitted to a server hosting the website to result in the generation of the first image of the website and the first portion of code defining the website.

13 . The non-transitory computer readable medium of claim 11 , wherein the method further comprises:

receiving, from a server hosting the website, the first portion of the code;

generating the first image of the website based on at least the first portion of the code; and

transmitting, to the server, a command based on the first command output.

14 . The non-transitory computer readable medium of claim 13 , wherein the first command output specifies a target object within the first portion of the code and an action to take toward the target object.

15 . The non-transitory computer readable medium of claim 14 , wherein the first command output specifies that the action to take toward the target object is one of clicking the target object or inputting text to the target object.

16 . The non-transitory computer readable medium of claim 13 , wherein the method further comprises, subsequent to transmitting the command based on the first command output transmitting to the server:

receiving, from the server, a second portion of a code defining the website;

generating a second image of the website based on at least the second portion of the code;

applying the second image of the website to the first machine learning model to generate a second set of visual tokens that represents the contents of the second image;

transforming a second textual input to generate a second set of textual tokens that represents the contents of the second textual input, wherein the second textual input includes at least the first portion of the code, the command to perform the task using the website, the second portion of the code, and the first command output;

apply the first set of visual tokens, the second set of visual tokens, and the second set of textual tokens to the second machine learning model to generate a second intermediate output;

applying the second intermediate output to the third machine learning model to generate a second command output, wherein the second command output represents an action to take relative to the code defining the website to effect the task; and

transmitting, to the server, a command based on the second command output.

17 . The non-transitory computer readable medium of claim 11 , wherein the first textual input additionally includes at least one past command output generated by the third machine learning model prior to generating the first command output.

18 . The non-transitory computer readable medium of claim 11 , wherein the first machine learning model comprises a vision transformer, and wherein the second machine learning model and third machine learning model were at least partially pre-trained as an instruction-finetuned language model.

19 . A computer-implemented method comprising:

obtaining a training dataset that includes a plurality of sequences of actions, wherein each sequence of actions of the training dataset corresponds to a respective command to perform a task using a respective website, and wherein each action of a particular sequence of the plurality of sequences of actions is associated with a respective image of the website for the particular sequence prior to taking the action; and

using the training dataset to train first, second, and third machine learning models such that:

the first machine learning model is trained to generate, from an image of a website, a first set of visual tokens that represents the contents of the image;

the second machine learning model is trained to generate, from the first set of visual tokens and a first set of textual tokens, a first intermediate output, wherein the first set of textual tokens represents the contents of a textual input that includes at least a first portion of a code defining the website and a command to perform a task using the website, and wherein the second machine learning model comprises an encoder; and

the third machine learning model is trained to generate, from the intermediate output, a first command output that represents an action to take relative to the code defining the website to effect the task, wherein the third machine learning model comprises a decoder.

20 . The computer-implemented method of claim 19 , wherein the second machine learning model and third machine learning model were pre-trained as an instruction-finetuned language model prior to being trained using the training dataset.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 25, 2024
From: FURUTA, HIROKI; NACHUM, OFIR; LEE, KUANG-HUEI; GUR, IZZEDDIN; GU, SHIXIANG
To: GOOGLE LLC
Reel/Frame 066882/0686 →
Continuity (1)
Related Publication 20250272350A1 · Aug 28, 2025
References Cited (95)
US 11734375B2 · Faust · 2023 [cited by examiner]
US 20230178076A1 · Abramson · 2023 [cited by examiner]
US 20250094713A1 · Wang · 2025 [cited by examiner]
WebVoyager : Building an End-to-End Web Agent with Large Multimodal Models; Hongliang He et al. Jun. 6, 2024 (Year: 2024). [cited by examiner]
Px2Struct (Year: 2023). [cited by examiner]
Li et al., “SelfDoc: Self-Supervised Document Representation Learning”, Jun. 7, 2021. [cited by applicant]
Li et al., “Pre-Trained Language Models for Interactive Decision-Making”, Oct. 29, 2022. [cited by applicant]
Li et al., “Mapping Natural Language Instructions to Mobile UI Action Sequences”, Jun. 5, 2020. [cited by applicant]
Li et al., “Competition-Level Code Generation with AlphaCode”, Feb. 8, 2022. [cited by applicant]
Liu et al., “Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration”, Feb. 24, 2018. [cited by applicant]
Lu et al., “Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks”, Oct. 4, 2022. [cited by applicant]
Matsushima et al., “Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization”, Jun. 23, 2020. [cited by applicant]
Mazumder et al., “FLIN: A Flexible Natural Language Interface for Web Navigation” Apr. 13, 2021. [cited by applicant]
Meta Fundamental AI Research Diplomacy Team (FAIR), Bakhtin et al., “Human-level play in the game of Diplomacy by combining language models with strategic reasoning”, Science 378, 1067-1074, Dec. 9, 2022. [cited by applicant]
Nakano et al., “WebGPT: Browser-assisted question-answering with human feedback”, Jun. 1, 2022. [cited by applicant]
Nogueira et al., “End-to-End Goal-Driven Web Navigation”, May 20, 2016. [cited by applicant]
Quyang et al., “Training language models to follow instructions with human feedback”, Mar. 4, 2022. [cited by applicant]
Radford et al., “Language Models are Unsupervised Multitask Learners”, 24 pages. [cited by applicant]
Radford et al., “Learning Transferable Visual Models From Natural Language Supervision”, Feb. 26, 2021. [cited by applicant]
Rae et al., “Scaling Language Models: Methods, Analysis & Insights from Training Gopher”, DeepMind, Jan. 21, 2022. [cited by applicant]
Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, Journal of Machine Learning Research 21, Sep. 19, 2023. [cited by applicant]
Reed et al., “A Generalist Agent”, Transactions on Machine Learning Research, Nov. 11, 2022. [cited by applicant]
Roberts et al., “Scaling Up Models and Data with t5x and seqio”, Mar. 31, 2022. [cited by applicant]
Shah et al., “LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action”, Jul. 26, 2022. [cited by applicant]
Shi et al., “World of Bits: An Open-Domain Platform for Web-Based Agents”, 2017. [cited by applicant]
Shoeybi et al., “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism”, Mar. 13, 2020. [cited by applicant]
Shridhar et al. “Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation”, Nov. 11, 2022. [cited by applicant]
Shvo et al., “AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Reinforcement Learning”, Jun. 6, 2021. [cited by applicant]
Stiennon et al., “Learning to summarize from human feedback”, Feb. 15, 2022. [cited by applicant]
Tang et al., “Unifying Vision, Text, and Layout for Universal Document Processing”, Mar. 13, 2023. [cited by applicant]
Tran et al., “UL2: Unifying Language Learning Paradigms”, Feb. 28, 2023. [cited by applicant]
Toyama et al., “AndroidEnv: A Reinforcement Learning Platform for Android”, May 27, 2021. [cited by applicant]
Vaswani et al., “Attention Is All You Need”, Aug. 2, 2023. [cited by applicant]
Wang et al., “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers”, Jan. 5, 2023. [cited by applicant]
Wang et al., “LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding”, Feb. 28, 2022. [cited by applicant]
Wang et al., “WebFormer: The Web-page Transformer for Structure Information Extraction”, Feb. 1, 2022. [cited by applicant]
Wei et al., “Finetuned Language Models are Zero-Shot Learners”, Feb. 8, 2022. [cited by applicant]
Wei et al., “Emergent Abilities of Large Language Models”, Oct. 26, 2022. [cited by applicant]
Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, Jan. 10, 2023. [cited by applicant]
Xu et al., “LayoutLM: Pre-training of Text and Layout for Document Image Understanding”, Jun. 16, 2020. [cited by applicant]
Yao et al., “WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents”, Feb. 8, 2023. [cited by applicant]
Yao et al., “React: Synergizing Reasoning and Acting in Language Models”, Mar. 10, 2023. [cited by applicant]
Zaheer et al., “Learning to Navigate Wikipedia by Taking Random Walks”, Oct. 31, 2022. [cited by applicant]
Zeng et al., “Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language”, May 27, 2022. [cited by applicant]
Ziegler et al., “Fine-Tuning Language Models from Human Preferences”, Jan. 8, 2020. [cited by applicant]
Adolphs et al., “Boosting Search Engines with Interactive Agents”, Transactions on Machine Learning Research, Jun. 7, 2022. [cited by applicant]
Aghajanyan et al., “HTLM: Hyper-Text Pre-Training and Prompting of Language Models”, Jul. 14, 2021. [cited by applicant]
Aghajanyan et al., “CM3: A Causal Masked Multimodal Model of the Internet”, Jan. 19, 2022. [cited by applicant]
Ahn et al., “Do as I Can, Not as I Say: Grounding Language in Robotic Affordances”, Aug. 16, 2022. [cited by applicant]
Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning”, Nov. 15, 2022. [cited by applicant]
Appalaraju et al., “Docformer: End-to-End Transformer for Document Understanding”, International Conference on Computer Vision, Sep. 20, 2021. [cited by applicant]
Austin et al., “Program Synthesis with Large Language Models”, Aug. 16, 2021. [cited by applicant]
Bommasani et al., “On the Opportunities and Risks of Foundation Models”, Jul. 12, 2022. [cited by applicant]
Brohan et al., “RT-1: Robotics Transformer for Real-World Control at Scale”, Aug. 11, 2023. [cited by applicant]
Brown et al., “Language Models are Few-Shot Learners”, Jul. 22, 2020. [cited by applicant]
Burns et al., “A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility”, Aug. 15, 2022. [cited by applicant]
Chen et al., “Decision Transformer: Reinforcement Learning via Sequence Modeling”, Jun. 24, 2021. [cited by applicant]
Chen et al., “Evaluating Large Language Models Trained on Code”, Jul. 14, 2021. [cited by applicant]
Chen et al., “A System for General In-Hand Object Re-Orientation”, 5th Conference on Robot Learning, Nov. 4, 2021. [cited by applicant]
Chen et al., “PaLI: A Jointly-Scaled Multilingual Language-Image Model”, Jun. 5, 2023. [cited by applicant]
Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways”, Oct. 5, 2022. [cited by applicant]
Chung et al., “Scaling Instruction-Finetuned Language Models”, Dec. 6, 2022. [cited by applicant]
Interactive Agents Team, “Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learning”, DeepMind, Feb. 2, 2022. [cited by applicant]
Deng et al., “ImageNet: A Large-Scale Hierarchical Image Database”. [cited by applicant]
Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, May 24, 2019. [cited by applicant]
Diaz et al., “User-Driven Automation of Web Form Filling”, ICWE 2013, pp. 171-185. [cited by applicant]
Dosovitskiy et al., “An Image is Worth 16X16 Words: Transformers for Image Recognition at Scale”, Jun. 3, 2021. [cited by applicant]
Fan et al., “MINEDOJO: Building Open-Ended Embodied Agents with Internet-Scale Knowledge”, Nov. 22, 2022. [cited by applicant]
Furuta et al., “A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation”, Feb. 4, 2023. [cited by applicant]
Furuta et al., “Generalized Decision Transformer for Offline Hindsight Information Matching”, Feb. 4, 2022. [cited by applicant]
Ghosh et al., “Learning to Reach Goals via Iterated Supervised Learning”, Oct. 2, 2020. [cited by applicant]
Gu et al., “BRAXLINES: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximization”, Oct. 10, 2021. [cited by applicant]
Gu et al., “Open-Vocabulary Object Detection via Vision and Language Knowledge Distillation”, May 12, 2022. [cited by applicant]
Gur et al., “Learning to Navigate the Web”, Dec. 21, 2018. [cited by applicant]
Gur et al., “Environment Generation for Zero-Shot Compositional Reinforcement Learning”, Jan. 21, 2022. [cited by applicant]
Gur et al., “Understanding HTML with Large Language Models”, May 19, 2023. [cited by applicant]
He et al., “Deep Residual Learning for Image Recognition”, Dec. 10, 2015. [cited by applicant]
He et al., “ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces”, Jan. 25, 2021. [cited by applicant]
Huang et al., “Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents”, Mar. 8, 2022. [cited by applicant]
Huang et al., “Inner Monologue: Embodied Reasoning through Planning with Language Models”, Jul. 12, 2022. [cited by applicant]
Humphreys et al., “A Data-Driven Approach for Learning to Control Computers”, Nov. 11, 2022. [cited by applicant]
Iyer et al., “OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization”, Jan. 30, 2023. [cited by applicant]
Janner et al., “Offline Reinforcement Learning as One Big Sequence Modeling Problem”, Nov. 29, 2021. [cited by applicant]
Jaques et al., “Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog”, Jul. 8, 2019. [cited by applicant]
Jia et al., “DOM-Q-NET: Grounded RL on Structured Language”, Feb. 19, 2019. [cited by applicant]
Jiang et al., “VIMA: General Robot Manipulation with Multimodal Prompts”, May 28, 2023. [cited by applicant]
Kamath et al., “MDETR—Modulated Detection for End-to-End Multi-Modal Understanding”, Oct. 12, 2021. [cited by applicant]
Kaplan et al., “Scaling Laws for Neural Language Models”, Jan. 23, 2020. [cited by applicant]
Kojima et al., “Large Language Models are Zero-Shot Reasoners”, Jan. 29, 2023. [cited by applicant]
Kudo et al., “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing”, Aug. 19, 2018. [cited by applicant]
Lee et al., “Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding”, Jun. 15, 2023. [cited by applicant]
Lee et al., “Multi-Game Decision Transformers”, Oct. 15, 2022. [cited by applicant]
Lewis et al., “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension”, Oct. 29, 2019. [cited by applicant]
Li et al., “StructuralLM: Structural Pre-training for Form Understanding”, May 24, 2021. [cited by applicant]
Li et al., “MarkupLM: Pre-training of Text and Markup Language for Visually Rich Document Understanding”, Mar. 11, 2022. [cited by applicant]