IP Library › Granted Patent US 12,455,636
Granted Patent B2
US 12,455,636 · App. 18/726,744 · Granted Oct 28, 2025

Learned computer control using pointing device and keyboard actions

Inventors: Peter Conway Humphreys (London, GB); Timothy Paul Lillicrap (London, GB); Tobias Markus Pohlen (London, GB); Adam Anthony Santoro (London, GB)
Assignee: DeepMind Technologies Limited
G06F3/038G06F3/023G06F40/284G06N3/0442G06N3/092G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,455,636
App. No.
18/726,744
Granted
Oct 28, 2025
Kind
B2
Abstract

A computer-implemented method for controlling a particular computer to execute a task is described. The method includes receiving a control input comprising a visual input, the visual input including one or more screen frames of a computer display that represent at least a current state of the particular computer, processing the control input using a neural network to generate one or more control outputs that are used to control the particular computer to execute the task, in which the one or more control outputs include an action type output that specifies at least one of a pointing device action or a keyboard action to be performed to control the particular computer; determining one or more actions from the one or more control outputs; and executing the one or more actions to control the particular computer.

Claims (61)

1. A computer-implemented method for controlling a particular computer to execute a task, the method including:

receiving a control input including a visual input, the visual input including one or more screen frames of a computer display that represent at least a current state of the particular computer;

processing the control input using a neural network to generate one or more control outputs that are used to control the particular computer to execute the task, wherein the one or more control outputs comprise an action type output that specifies at least one of a pointing device action or a keyboard action to be performed to control the particular computer;

determining one or more actions from the one or more control outputs; and

executing the one or more actions to control the particular computer,

wherein the control input further comprises one or more language inputs, one or more previous controls, or both, and

wherein the neural network comprises a visual processing sub-network, one or more language processing sub-networks, a previous control processing sub-network, a multimodal transformer sub-network, and an output sub-network, and wherein processing the control input using the neural network to generate the one or more control outputs comprises:

processing, using the visual processing sub-network, the visual input to generate one or more visual embeddings;

processing each language input in the one or more language inputs using the language processing sub-network to generate a respective language embedding;

processing, using the previous control processing sub-network, the one or more previous controls to generate a previous control embedding;

combining, using a multimodal transformer sub-network, the one or more visual embeddings and the one or more language embeddings to generate a transformed embedding; and

processing, using the output sub-network, the transformed embedding and the previous control embedding to generate the one or more control outputs.

2. The method of claim 1 , wherein the visual processing sub-network comprises a image encoder sub-network and a flattening sub-network, and

wherein processing, using the visual processing sub-network, the visual input to generate one or more visual embeddings comprises:

processing, using the image encoder sub-network, the visual input to generate one or more encoded visual feature vectors, and

flattening the one or more encoded visual feature vectors by using the flattening sub-network to generate the one or more visual embeddings.

3. The method of claim 2 , wherein the image encoder sub-network is a ResNet which has four ResNet blocks.

4. The method of claim 1 , wherein each language processing sub-network comprises a tokenization layer and a language encoder sub-network, and

wherein processing the language input in the one or more language inputs using the language processing sub-network to generate the respective language embedding comprises:

splitting, using the tokenization layer, the respective language input into tokens and generating, for each token in the tokens, a corresponding token embedding, and

attending over the token embeddings by using the language encoder sub-network to generate the corresponding language embedding.

5. The method of claim 4 , wherein the language encoder sub-network is a 1-layer transformer with 4 heads.

6. The method of claim 4 , wherein attending over the token embeddings to generate the corresponding language embedding comprises:

processing the token embeddings to generate a key vector and a value vector,

processing an extra embedding to generate a query vector, and

processing the key vector, the value vector, and the query vector using a cross-attention mechanism to generate the corresponding language embedding.

7. The method of claim 1 , wherein generating the transformed embedding using the multimodal transformer sub-network comprises:

processing the one or more visual embeddings to generate a visual query vector, a visual key vector, and a visual value vector,

processing the one or more language embeddings to generate a language query vector, a language key vector, and a language value vector,

processing one or more additional embeddings to generate an extra query vector, an extra key vector, and an extra value vector,

using a self-attention mechanism to process the visual query vector, the visual key vector, the visual value vector, the language query vector, the language key vector, the language value vector, the extra query vector, the extra key vector, the extra value vector to generate the transformed embedding.

8. The method of claim 1 , wherein the output sub-network comprises one or more long short-term memory (LSTM) layers.

9. The method of claim 8 , wherein the output sub-network further comprises an attention policy sub-network configured to process an output of the one or more LSTM layers to generate a task field index output.

10. The method of claim 9 , wherein the attention policy sub-network is configured to:

use the corresponding output of the two LSTM layers as a query vector,

process the corresponding task field embeddings generated by the task field language processing sub-network to generate a key vector and a value vector, and

use a cross-attention mechanism to process the query vector, the key vector, and the value vector to generate the task field index output.

11. The method of claim 1 , further comprising: co-training the neural network using i) imitation learning to imitate one or more behaviors of human demonstration, and ii) reinforcement learning.

12. The method of claim 1 , further comprising:

training the neural network using imitation learning to imitate the behavior of human demonstration as shown in the visual input, and

training the neural network using reinforcement learning.

13. The method of claim 1 , wherein during training of the neural network, the visual input demonstrates how a human would execute the task on the particular computer.

14. The method of claim 1 , wherein processing, using the output sub-network, the transformed embedding and the previous control embedding to generate the one or more control outputs comprising: processing, using the output sub-network, a concatenation of the transformed embedding and the previous control embedding to generate the one or more control outputs.

15. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network configured to process a control input to generate one or more control outputs for controlling a particular computer to execute a task, the control input comprising (i) a visual input comprising one or more screen frames of a computer display that represent at least a current state of the particular computer, (ii) one or more language inputs, and (iii) one or more previous controls, the neural network comprising:

a visual processing sub-network configured to process the visual input to generate one or more visual embeddings;

one or more language processing sub-networks, each language processing sub-network configured to process a language input in the one or more language inputs to generate a respective language embedding;

a previous control processing sub-network to process the one or more previous controls to generate a previous control embedding;

a multimodal transformer sub-network configured to combine the one or more visual embeddings and the one or more language embeddings to generate a transformed embedding; and

an output sub-network configured to process the transformed embedding and the previous control embedding to generate the one or more control outputs.

16. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving a control input including a visual input, the visual input including one or more screen frames of a computer display that represent at least a current state of the particular computer;

processing the control input using a neural network to generate one or more control outputs that are used to control the particular computer to execute the task, wherein the one or more control outputs comprise an action type output that specifies at least one of a pointing device action or a keyboard action to be performed to control the particular computer;

determining one or more actions from the one or more control outputs; and

executing the one or more actions to control the particular computer,

wherein the control input further comprises one or more language inputs, one or more previous controls, or both, and

wherein the neural network comprises a visual processing sub-network, one or more language processing sub-networks, a previous control processing sub-network, a multimodal transformer sub-network, and an output sub-network, and wherein processing the control input using the neural network to generate the one or more control outputs comprises:

processing, using the visual processing sub-network, the visual input to generate one or more visual embeddings;

processing each language input in the one or more language inputs using the language processing sub-network to generate a respective language embedding;

processing, using the previous control processing sub-network, the one or more previous controls to generate a previous control embedding;

combining, using a multimodal transformer sub-network, the one or more visual embeddings and the one or more language embeddings to generate a transformed embedding; and

processing, using the output sub-network, the transformed embedding and the previous control embedding to generate the one or more control outputs.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 3, 2024
From: HUMPHREYS, PETER CONWAY; LILLICRAP, TIMOTHY PAUL; POHLEN, TOBIAS MARKUS; SANTORO, ADAM ANTHONY
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 068780/0616 →
Continuity (2)
Provisional Application 63304549 · Jan 28, 2022
Related Publication 20250224821A1 · Jul 10, 2025
References Cited (76)
US 10810993B2 · Chen · 2020 [cited by examiner]
US 10878505B1 · Blair · 2020 [cited by examiner]
US 11010645B2 · Xu · 2021 [cited by examiner]
US 11743588B1 · Shu · 2023 [cited by examiner]
US 11803710B1 · Love · 2023 [cited by examiner]
US 12189870B2 · Humphreys · 2025 [cited by examiner]
US 20190108448A1 · O'Malia et al. · 2019 [cited by applicant]
US 20190122258A1 · Bramberger et al. · 2019 [cited by applicant]
US 20190258713A1 · Kiros et al. · 2019 [cited by applicant]
US 20200285932A1 · Singh · 2020 [cited by examiner]
US 20200302294A1 · Kadav et al. · 2020 [cited by applicant]
US 20200380403A1 · Aggarwal et al. · 2020 [cited by applicant]
US 20210000277A1 · Fu et al. · 2021 [cited by applicant]
US 20210027774A1 · Kessler · 2021 [cited by examiner]
US 20210065712A1 · Holm · 2021 [cited by examiner]
US 20210110831A1 · Shillingford · 2021 [cited by examiner]
US 20210134312A1 · Koishida · 2021 [cited by examiner]
US 20220067546A1 · Amrani · 2022 [cited by examiner]
US 20220171936A1 · Wang et al. · 2022 [cited by applicant]
US 20220309364A1 · Perry et al. · 2022 [cited by applicant]
US 20220366152A1 · Zenkel et al. · 2022 [cited by applicant]
US 20220398497A1 · Shimizu · 2022 [cited by applicant]
US 20220404956A1 · Choi · 2022 [cited by applicant]
US 20230038673A1 · Masoud · 2023 [cited by applicant]
US 20230106873A1 · Qin · 2023 [cited by examiner]
US 20230120397A1 · Cronie · 2023 [cited by applicant]
US 20230192141A1 · Zhang · 2023 [cited by applicant]
US 20230244325A1 · Humphreys · 2023 [cited by examiner]
US 20230260536A1 · Xu · 2023 [cited by examiner]
US 20240265911A1 · Assael · 2024 [cited by examiner]
US 20240346814A1 · Bharadwaj · 2024 [cited by examiner]
CN 110379225A · 2019 [cited by applicant]
CN 112822845 · 2021 [cited by applicant]
KR 1020110074858A · 2011 [cited by applicant]
Abramson et al., “Creating multimodal interactive agents with imitation and self-supervised learning,” CoRR, Dec. 7, 2021, arXiv:2112.03763, 17 pages. [cited by applicant]
Ammanabrolu et al., “How to motivate your dragon: Teaching goal-driven agents to speak and act in fantasy worlds,” CoRR, Oct. 1, 2020, arXiv:2010.00685, 27 pages. [cited by applicant]
Bommasani et al., “On the opportunities and risks of foundation models,” CoRR, Aug. 16, 2021, arXiv:2108.07258, 214 pages. [cited by applicant]
Botvinick et al., “Reinforcement Learning, Fast and Slow,” Trends in Cognitive Sciences, May 2019, 23(5):408-422. [cited by applicant]
Brown et al., “Language models are few-shot learners,” CoRR, May 28, 2020, arXiv:2005.14165, 75 pages. [cited by applicant]
Chen et al., “Evaluating large language models trained on code,” CoRR, Jul. 7, 2021, arXiv:2107.03374, 35 pages. [cited by applicant]
Christiano et al., “Deep reinforcement learning from human preferences,” CoRR, Jun. 12, 2017, arXiv: 1706.03741, 17 pages. [cited by applicant]
Github.com [online], “google/sandboxed-api,” Nov. 2021, retrieved on Jun. 30, 2023, retrieved from URL<https://github.com/google/sandboxed-api/tree/main/sandboxed_api/sandbox2/>, 5 pages. [cited by applicant]
Github.com [online], “Security Considerations,” Jan. 2019, retrieved on Jun. 30, 2023, retrieved from URL<https://chromedriver.chromium.org/security-considerations/>, 1 page. [cited by applicant]
Gur et al. “Environment generation for zero-shot compositional reinforcement learning,” Advances in Neural Information Processing Systems 34, 2021, 13 pages. [cited by applicant]
Gur et al., “Learning to navigate the web,” CoRR, Dec. 21, 2018, arXiv:1812.09195, 12 pages. [cited by applicant]
Ibarz et al., “Reward learning from human preferences and demonstrations in Atari,” CoRR, Nov. 15, 2018, arXiv:1811.06521, 20 pages. [cited by applicant]
International Search Report and Written Opinion in International Appln. No. PCT/EP2023/052210, dated Apr. 3, 2023, 18 pages. [cited by applicant]
Jaques et al., “Way off-policy batch deep reinforcement learning of implicit human preferences in dialog,” CoRR, Jun. 30, 2019, arXiv:1907.00456, 16 pages. [cited by applicant]
Jia et al., “DOM-Q-NET: Grounded rl on structured language,” CoRR, Feb. 19, 2019, arXiv:1902.07257, 17 pages. [cited by applicant]
Kaplan et al., “Scaling laws for neural language models,” CoRR, Jun. 23, 2020, arXiv:2001.08361, 30 pages. [cited by applicant]
Kingma et al., “Adam: A method for stochastic optimization,” CoRR, Dec. 22, 2014, arXiv:1412.6980, 15 pages. [cited by applicant]
Li et al., “Dialogue learning with human-in-the-loop,” CoRR, Nov. 29, 2016, arXiv:1611.09823, 23 pages. [cited by applicant]
Li et al., “Mapping natural language instructions to mobile UI action sequences,” CoRR, May 7, 2020, arXiv:2005.03776, 13 pages. [cited by applicant]
Liu et al., “Reinforcement learning on web interfaces using workflowguided exploration,” CoRR, Feb. 24, 2018, arXiv:1802.08802, 15 pages. [cited by applicant]
Nakano et al., “Webgpt: Browser-assisted question-answering with human feedback,” CoRR, Dec. 17, 2021, arXiv:2112.09332, 32 pages. [cited by applicant]
Pomerleau, “ALVINN: An autonomous land vehicle in a neural network,” Advances in neural information processing systems, 1988, pp. 305-313. [cited by applicant]
Rae et al., “Scaling language models: Methods, analysis & insights from training gopher,” CoRR, Dec. 8, 2021, arXiv:2112.11446, 120 pages. [cited by applicant]
Shi et al., “World of bits: An open-domain platform for webbased agents,” Proceedings of the 34th International Conference on Machine Learning, 2017, pp. 3135-3144. [cited by applicant]
Shoeybi et al., “Megatron-lm: Training multibillion parameter language models using model parallelism,” CoRR, Sep. 17, 2019, arXiv:1909.08053, 15 pages. [cited by applicant]
Shvo et al., “Appbuddy: Learning to accomplish tasks in mobile apps via reinforcement learning, ” CoRR, May 31, 2021, arXiv:2106.00133, 17 pages. [cited by applicant]
Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature, 2016, 529(7587):484-489. [cited by applicant]
Song et al., “V-MPO: On-policy maximum a posteriori policy optimization for discrete and continuous control, ” CoRR, Sep. 26, 2019, arXiv:1909.12238, 19 pages. [cited by applicant]
Stiennon et al., “Learning to summarize from human feedback,” CoRR, Sep. 2, 2020, arXiv:2009.01325, 45 pages. [cited by applicant]
Sutton et al., “Reinforcement learning: An introduction,” MIT press, Nov. 13, 2018, 10 pages. [cited by applicant]
Toyama et al., “Androidenv: A reinforcement learning platform for android,” CoRR, May 27, 2021, arXiv:2105.13231, 13 pages. [cited by applicant]
Vaswani et al., “Attention is all you need,” Advances in neural information processing systems, 2017, 11 pages. [cited by applicant]
Vinyals et al., “Grandmaster level in starcraft II using multi-agent reinforcement learning,” Nature, Oct. 30, 2019, 575 (7782):350-354. [cited by applicant]
Wirth et al., “A survey of preference-based reinforcement learning methods,” Journal of Machine Learning Research, 2017, 18(136):1-46. [cited by applicant]
International Preliminary Report on Patentability in International Appln. No. PCT/EP2023/052210, dated Aug. 8, 2024, 11 pages. [cited by applicant]
Office Action in Australian Appln. No. 2023211162, dated Aug. 8, 2024, 4 pages. [cited by applicant]
Gur et al., “Leaming to navigate the web,” CoRR, Dec. 21, 2018, arXiv:1812.09195, 12 pages. [cited by applicant]
Shoeybi et al., “Megatron-Im: Training multibillion parameter language models using model parallelism,” CoRR, Sep. 17, 2019, arXiv:1909.08053, 15 pages. [cited by applicant]
Notice of Allowance in Japanese Appln. No. 2024-544791, mailed on Mar. 25, 2025, 5 pages (with English translation). [cited by applicant]
Notice of Acceptance in Australian Appln. No. 2023211162, mailed on May 2, 2025, 3 pages. [cited by applicant]
Office Action in Chinese Appln. No. 202380019005.6, mailed Aug. 20, 2025, 11 pages (with English translation). [cited by applicant]
Office Action in Korean Appln. No. 10-2024-7023307, mailed on Sep. 4, 2025, 3 pages. [cited by applicant]