IP Library › Granted Patent US 12,608,585
Granted Patent B2
US 12,608,585 · App. 17/872,528 · Granted Apr 21, 2026

Neural networks for selecting actions to be performed by a robotic agent

Inventors: Razvan Pascanu (Letchworth Garden, GB); Raia Thais Hadsell (London, GB); Mel Vecerik (London, GB); Thomas Rothoerl (London, GB); Andrei-Alexandru Rusu (London, GB); Nicolas Manfred Otto Heess (London, GB)
Assignee: GDM HOLDING LLC
G06N3/008G06N3/092G06N3/10G06N3/045G06N3/08G06N3/082G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,608,585
App. No.
17/872,528
Granted
Apr 21, 2026
Kind
B2
Abstract

A system includes a neural network system implemented by one or more computers. The neural network system is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation. The neural network system includes: (i) a sequence of deep neural networks (DNNs), in which the sequence of DNNs includes a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, and (ii) a first robot-trained DNN that is configured to receive the observation and to process the observation to generate the policy output.

Claims (77)

1 . A computer-implemented method comprising:

receiving an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task;

processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), and

wherein the sequence of DNNs comprises:

a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, wherein the simulation-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, and

a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulation-trained DNN and to process the observation and the one or more layer outputs to generate the policy output; and

selecting an action to be performed by the robotic agent in response to the observation using the policy output,

wherein (i) the simulation-trained DNN that provides layer outputs to the first robot-trained DNN and (ii) the first robot-trained DNN that receives layer outputs from the simulation-trained DNN are initialized with different capacities and the first robot-trained DNN that receives layer outputs from the simulation-trained DNN has fewer parameters than the simulation-trained DNN that provides layer outputs to the first robot-trained DNN.

2 . The method of claim 1 ,

wherein the simulation-trained DNN comprises a first plurality of indexed layers, and the simulation-trained DNN, implemented by one or more computers, is configured to process the observation through each layer in the first plurality of indexed layers to generate a respective layer output for each layer in the first plurality of indexed layers; and

wherein the first robot-trained DNN comprises a second plurality of indexed layers, and the first robot-trained DNN, implemented by the one or more computers, is configured to process the observation through each layer in the second plurality of indexed layers to generate the policy output.

3 . The method of claim 2 , wherein, for each of one or more layers of the second plurality of indexed layers, a corresponding layer in the first plurality of indexed layers having a same index is a neural network layer of a same type but of a larger dimension than the layer in the second plurality of indexed layers.

4 . The method of claim 2 , wherein the first plurality of indexed layers and the second plurality of indexed layers each include a respective recurrent neural network layer.

5 . The method of claim 4 , wherein the recurrent neural network layer in the second plurality of indexed layers, implemented by the one or more computers, is configured to receive as input (i) a layer output of a layer preceding the recurrent neural network layer in the first plurality of indexed layers, (ii) an internal state of the recurrent neural network layer in the first plurality of indexed layers, and (iii) a layer output of a layer preceding the recurrent neural network layer in the second plurality of indexed layers.

6 . The method of claim 2 , wherein each of one or more layers in the second plurality of indexed layers, implemented by the one or more computers, is configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, and (ii) a layer output generated by a preceding layer of the simulation-trained DNN, and wherein each of the one or more layers in the second plurality of indexed layers is further configured to:

apply a respective first set of parameters to the layer output generated by the preceding layer of the first robot-trained DNN; and

apply a respective second set of parameters to the layer output generated by the preceding layer of the simulation-trained DNN,

wherein a preceding layer is a layer whose index is one less than the index of the layer.

7 . The method of claim 2 , wherein the policy output defines a respective change in position of each of a plurality of degrees of freedom of the robotic agent.

8 . The method of claim 7 , wherein the plurality of degrees of freedom include one or more joints of the robotic agent and one or more actuators of the robotic agent.

9 . The method of claim 2 , wherein the sequence of DNNs further comprises:

a second robot-trained DNN, wherein:

the second robot-trained DNN comprises a third plurality of indexed layers, and

one or more of the layers in the third plurality of indexed layers, implemented by the one or more computers, are each configured to receive as input (i) a layer output generated by a preceding layer of the first robot-trained DNN, (ii) a layer output generated by a preceding layer of the simulation-trained DNN, and (iii) a layer output generated by a preceding layer of the second robot-trained DNN.

10 . The method of claim 9 , wherein:

the second robot-trained DNN, implemented by the one or more computers, is configured to receive different data characterizing the current state in conjunction with the observation; and

the second robot-trained DNN, implemented by the one or more computers, is configured to process the different data through the third plurality of indexed layers to generate a second policy output that defines an action to be performed by the robotic agent to perform a second, different robotic task.

11 . The method of claim 10 , wherein the observation is visual data and the different data is proprioceptive data.

12 . The method of claim 1 , wherein the first robot-trained DNN has fewer layers than the simulation-trained DNN.

13 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task;

processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), and

wherein the sequence of DNNs comprises:

a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, wherein the simulation-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, and

a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulation-trained DNN and to process the observation and the one or more layer outputs to generate the policy output; and

selecting an action to be performed by the robotic agent in response to the observation using the policy output,

wherein (i) the simulation-trained DNN that provides layer outputs to the first robot-trained DNN and (ii) the first robot-trained DNN that receives layer outputs from the simulation-trained DNN are initialized with different capacities and the first robot-trained DNN that receives layer outputs from the simulation-trained DNN has fewer parameters than the simulation-trained DNN that provides layer outputs to the first robot-trained DNN.

14 . A system comprising:

one or more computers; and

one or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task;

processing the observation using a neural network system to generate a policy output, wherein the neural network system comprises a sequence of deep neural networks (DNNs), and

wherein the sequence of DNNs comprises:

a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, wherein the simulation-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, and

a first robot-trained DNN that has been trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulation-trained DNN and to process the observation and the one or more layer outputs to generate the policy output; and

selecting an action to be performed by the robotic agent in response to the observation using the policy output,

wherein (i) the simulation-trained DNN that provides layer outputs to the first robot-trained DNN and (ii) the first robot-trained DNN that receives layer outputs from the simulation-trained DNN are initialized with different capacities and the first robot-trained DNN that receives layer outputs from the simulation-trained DNN has fewer parameters than the simulation-trained DNN that provides layer outputs to the first robot-trained DNN.

15 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein the neural network system, implemented by the one or more computers, is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation,

wherein the sequence of DNNs comprises:

a simulation-trained DNN, the simulation-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, wherein the simulation-trained DNN is trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task; and

a first robot-trained DNN that is trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulation-trained DNN and to process the observation and the one or more layer outputs to generate the policy output; and

wherein the operations comprise:

training the first robot-trained DNN on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed,

wherein (i) the simulation-trained DNN that provides layer outputs to the first robot-trained DNN and (ii) the first robot-trained DNN that receives layer outputs from the simulation-trained DNN are initialized with different capacities and the first robot-trained DNN that receives layer outputs from the simulation-trained DNN has fewer parameters than the simulation-trained DNN that provides layer outputs to the first robot-trained DNN.

16 . The one or more non-transitory computer-readable media of claim 15 , wherein an output layer of the first robot-trained DNN, implemented by the one or more computers, is configured to:

receive as input (i) a layer output generated by a layer preceding the output layer of the first robot-trained DNN, and (ii) a layer output generated by a layer preceding an output layer of the simulation-trained DNN;

apply a first set of parameters to the layer output generated by a layer preceding the output layer the first robot-trained DNN; and

apply a second set of parameters to the layer output generated by a layer preceding the output layer of the simulation-trained DNN, and

wherein the operations further comprise:

initializing values of the second set of parameters to match trained values of parameters of the output layer of the simulation-trained DNN.

17 . The one or more non-transitory computer-readable media of claim 16 , wherein the operations further comprise:

initializing values of the first set of parameters to zero.

18 . The one or more non-transitory computer-readable media of claim 16 , wherein the operations further comprise:

initializing values of parameters of layers of the first robot-trained DNN other than the output layer of the first robot-trained DNN to random values.

19 . The one or more non-transitory computer-readable media of claim 15 , wherein the operations further comprise:

training the simulation-trained DNN on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task to determine trained values of parameters of the simulation-trained DNN.

20 . The one or more non-transitory computer-readable media of claim 19 , wherein training the simulation-trained DNN comprises training the simulation-trained DNN on the interactions of the simulated version of the robotic agent using a first reinforcement learning technique.

21 . The one or more non-transitory computer-readable media of claim 20 , wherein training the first robot-trained DNN comprises training the first robot-trained DNN on the interactions of the robotic agent using a second reinforcement learning technique.

22 . A system comprising:

one or more computers; and

one or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network system comprising a sequence of deep neural networks (DNNs) to determine trained values of parameters of one or more of the DNNs in the sequence of DNNs, wherein the neural network system, implemented by the one or more computers, is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation,

wherein the sequence of DNNs comprises:

a simulation-trained DNN, the simulation-trained DNN is configured to receive the observation and process the observation to generate one or more layer outputs, wherein the simulation-trained DNN is trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task; and

a first robot-trained DNN that is trained on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed, wherein the first robot-trained DNN is configured to receive the observation and one or more layer outputs generated by the simulation-trained DNN and to process the observation and the one or more layer outputs to generate the policy output; and

wherein the operations comprise:

training the first robot-trained DNN on interactions of the robotic agent with the real-world environment to perform the robotic task to determine trained values of parameters of the first robot-trained DNN while holding trained values of the parameters of the simulation-trained DNN fixed,

wherein (i) the simulation-trained DNN that provides layer outputs to the first robot-trained DNN and (ii) the first robot-trained DNN that receives layer outputs from the simulation-trained DNN are initialized with different capacities and the first robot-trained DNN that receives layer outputs from the simulation-trained DNN has fewer parameters than the simulation-trained DNN that provides layer outputs to the first robot-trained DNN.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 16, 2022
From: PASCANU, RAZVAN; HADSELL, RAIA THAIS; VECERIK, MEL; ROTHOERL, THOMAS; RUSU, ANDREI-ALEXANDRU; HEESS, NICOLAS MANFRED OTTO
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 060821/0408 →
Continuity (5)
Continuation 16829237 · Mar 25, 2020
Continuation 16380125 · Apr 10, 2019
Continuation PCTUS2017055894 · Oct 10, 2017
Provisional Application 62406363 · Oct 10, 2016
Related Publication 20220355472A1 · Nov 10, 2022
References Cited (76)
US 8793205B1 · Fisher et al. · 2014 [cited by applicant]
US 9314924B1 · Laurent et al. · 2016 [cited by applicant]
US 20140257803A1 · Yu et al. · 2014 [cited by applicant]
US 20150100530A1 · Mnih · 2015 [cited by examiner]
US 20160026914A1 · Yu · 2016 [cited by examiner]
US 20160292568A1 · Schaul et al. · 2016 [cited by applicant]
US 20160307098A1 · Goel · 2016 [cited by examiner]
US 20170337464A1 · Rabinowitz · 2017 [cited by examiner]
US 20170364829A1 · Fyffe · 2017 [cited by examiner]
US 20180032867A1 · Son · 2018 [cited by examiner]
US 20190025917A1 · Francis · 2019 [cited by applicant]
US 20190061147A1 · Luciw · 2019 [cited by examiner]
US 20190188567A1 · Yao · 2019 [cited by examiner]
US 20220355472A1 · Pascanu · 2022 [cited by examiner]
CN 102402712 · 2012 [cited by applicant]
CN 103279039 · 2013 [cited by applicant]
CN 110023965A · 2019 [cited by applicant]
JP H05197821 · 1993 [cited by applicant]
JP 2011143523 · 2020 [cited by applicant]
WO WO2016025412A1 · 2016 [cited by applicant]
Devin et al., “Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer” Sep. 22, 2016, arXiv: 1609.07088v1, pp. 1-8. (Year: 2016). [cited by examiner]
Zhang et al., “Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control” Nov. 13, 2015, arXiv: 1511.03791v2, pp. 1-8. (Year: 2015). [cited by examiner]
Zhu et al., “Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning” Sep. 16, 2016, arXiv: 1609.05143v1, pp. 1-8. (Year: 2016). [cited by examiner]
Rusu et al., “Progressive Neural Networks” Jun. 15, 2016, arXiv: 1606.04671v1, pp. 1-12. (Year: 2016). [cited by examiner]
Finn et Levine, “Deep Visual Foresight for Planning Robot Motion” Oct. 3, 2016, arXiv: 1610.00696v1, pp. 1-8. (Year: 2016). [cited by examiner]
Li et Hoiem, “Learning without Forgetting” Jun. 29, 2016, arXiv: 1606.09282v1, pp. 1-16. (Year: 2016). [cited by examiner]
Tzeng et al., “Towards Adapting Deep Visuomotor Representations from Simulated to Real Enviironments” Nov. 23, 2015, arXiv: 1511.07111v1, pp. 1-10. (Year: 2015). [cited by examiner]
Gu et al., “Deep Reinforcement Learning for Robotic Manipulation” Oct. 3, 2016, arXiv: 1610.00633v1, pp. 1-9. (Year: 2016). [cited by examiner]
Parisotto et al., “Actor-Mimic Deep Multitask and Transfer Reinforcement Learning” Feb. 22, 2016, arXiv: 1511.06342v4, pp. 1-16. ( Year: 2016). [cited by examiner]
Garnelo et al., “Towards Deep Symbolic Reinforcement Learning” Oct. 1, 2016, arXiv: 1609.05518v2, pp. 1-13. (Year: 2016). [cited by examiner]
Peng et al., “Terrain-Adaptive Locomotion Skills Using Deep Reinforcement Learning” Jul. 2016, pp. 1-12. (Year: 2016). [cited by examiner]
Pinto et Gupta, “Learning to Push by Grasping: Using multiple tasks for effective learning” Sep. 28, 2016, arXiv: 1609.09025v1, pp. 1-8. (Year: 2016). [cited by examiner]
Han et al., “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding” Feb. 15, 2016, arXiv: 1510.00149v5, pp. 1-14. (Year: 2016). [cited by examiner]
Han et al., “Learning both Weights and Connections for Efficient Neural Networks” Oct. 30, 2015, arXiv: 1506.02626v3, pp. 1-9. ( Year: 2015). [cited by examiner]
Barrett et al., “Transfer learning for reinforcement learning on a physical robot,” Ninth International Conference on Autonomous Agents and Multiagent Systems—Adaptive Learning Agents Workshop, May 2010, 6 pages. [cited by applicant]
Bousmalis et al., “Domain separation networks,” 29th Conference on Neural Information Processing Systems, Aug. 22, 2016, 15 pages. [cited by applicant]
Braylan et al [online], “Resuse of neural modules for general video game playing,” Dec. 2015, retrieved Apr. 10, 2019, URL<https://arxiv.org/pdf/1512.01537.pdf> 7 pages. [cited by applicant]
Cutler, “Efficient Reinforcement Learning for Robots using Informative Simulated Priors,” 2015 IEEE International Conference on Robotics and Automation, Jul. 2, 2015, pp. 2605-2612. [cited by applicant]
Devin et al [online], “Learning Modular Network Policies for Multi-Task and Multi-Robot Transfer,” Sep. 2016, retrieved on Apr. 10, 2019, URL <https://arxiv.org/pdf/1609.07088.pdf>, 8 pages. [cited by applicant]
Foerster et al., “Learning to Communicate with Deep Multi-Agent Reinforcement Learning,” May 24, 2016, arXiv, pp. 1-13. [cited by applicant]
Ganin et al., “Domain-adversarial training of neural networks,” Journal of Machine Learning Research, Apr. 2016, 35 pages. [cited by applicant]
Gu et al., “Continuous Deep Q-Learning with Model-based Acceleration,” International Conference on Machine Learning, 2016, 10 pages. [cited by applicant]
Hadsell et al [online], “Progressive nets for simulation to robot transfer,” RSS 2016 Workshop on Limits and Potentials of Deep Learning in Robotics, Jun. 2016, retrieved on Apr. 10, 2019, URL<http://juxi.net/workshop/d… [cited by applicant]
Hausknecht et al., “Deep Reinforcement Learning in Parameterized Action Space,” arXiv, Feb. 16, 2016, pp. 1-12. [cited by applicant]
Heess et al., “Learning continuous control policies by stochastic value gradients,” Advances in Neural Information Processing Systems 28: Annual Conferences on Nueral Information Processing Systems 2015, Dec. 2015, 9 pa… [cited by applicant]
James et al., “3D simulation for robot arm control with deep q-learning,” ArXiv E-prints, Dec. 2016, 6 pages. [cited by applicant]
JP Decision to Grant a Patent in Japanese Appln. No. 2019-519210, dated Mar. 29, 2021, 5 pages (with English translation). [cited by applicant]
JP Office Action in Japanese Appln. No. 2019-519210, dated Jul. 27, 2020, 12 pages (with English translation). [cited by applicant]
Kono et al., “Hierarchical Transfer Learning with Heterogeneous Multiple Autonomous Agents,” Transactions of the Society of instrument and Control Engineers, The Society of instrument and Control Engineers, Jun. 30, 201… [cited by applicant]
Lample et al., “Playing FPS Games with Deep Reinforcement Learning,” arXiv, Sep. 18, 2016, pp. 1-7. [cited by applicant]
Levine et al., “End-to-end training of deep visuomotor policies,” CoRR, Apr. 2016, 40 pages. [cited by applicant]
Levine et al., “Learning contact rich manipulation skills with guided policy search,” IEEE International Conference on Robotics and Automation, Feb. 2015, 8 pages. [cited by applicant]
Levine et al., “Learning hand-eye coordination for robotic grasping with deep learning and large scale data collection,” CoRR, Aug. 2016, 12 pages. [cited by applicant]
Levine et al., “Learning neural network policies with guided policy search under unknown dynamics,” Advances in Neural Information Processing Systems 27, 2014, 11 pages. [cited by applicant]
Lillicrap et al., “Continuous control with deep reinforcement learning,” CoRR, Feb. 2016, 14 pages. [cited by applicant]
Long et al., “Learning transferable features with deep adaptation networks,” Proceedings of the 32nd International Conference on Machine Learning, May 2015, 9 pages. [cited by applicant]
Mnih et al., “Asynchronous methods for deep reinforcement learning,” Int'l Conf. on Machine Learning, Jun. 2016, 19 pages. [cited by applicant]
Mnih et al., “Human level control through deep reinforcement learning,” Macmillan Publishers Limited, Feb. 2015, 13 pages. [cited by applicant]
Office Action in European Appln. No. 17788409.5, dated Nov. 10, 2021, 7 pages. [cited by applicant]
PCT International Search Report and Written Opinion in the International Appln. No PCT/US2017/055894, dated Jan. 17, 2018, 15 pages. [cited by applicant]
Peng et al., “Learning deep object detectors from 3D models,” 2015 IEEE International Conference on Computer Visions, Oct. 2015, 9 pages. [cited by applicant]
Pinto et al., “Supersizing self-supervision: learning to grasp from 50K tries and 700 robot hours,” ICRA, Sep. 2015, 8 pages. [cited by applicant]
Pinto et al., “The curious robot: learning visual representations via physical interaction,” CoRR, Jul. 2016, 17 pages. [cited by applicant]
Rasu et al [online], “Progressive Neural Networks,” arXiv, Sep. 2016, retrieved on Apr. 10, 2019, URL<https://arxiv.org/pdf/1606.04671.pdf>, 14 pages. [cited by applicant]
Rusu et al., “Progressive Neural Networks,” arXiv, Sep. 2016, 14 pages. [cited by applicant]
Schulman et al., “High-dimensional continuous control using generalized advantage estimation,” Proceedings of the International Conference on Learning Representations, Oct. 2018, 14 pages. [cited by applicant]
Schulman et al., “Trust region policy organization,” Proceeding of the 32nd International Conference on Machine Learning, Apr. 2017, 16 pages. [cited by applicant]
Su et al., “Render for CNN: viewpoint estimation in images using CNNs trained with rendered 3D model views,” 2015 IEEE International Conference on Computer Visions, Dec. 2015, 9 pages. [cited by applicant]
Todorov et al., “MuJoCo: A physics engine for model-based control,” International Conference on Intelligent Robots and Systems, 2012, 8 pages. [cited by applicant]
Tokuie et al., “Deep Learning from the Viewpoint of Optimization,” Operations Research, Japan Operations Research Society, Apr. 1, 2015, 60(4):191-197 (with English abstract). [cited by applicant]
Tzeng et al., “Deep domain confusion: maximizing for domain invariance,” CoRR, Dec. 2014, 9 pages. [cited by applicant]
Tzeng et al., “Simultaneous deep transfer across domains and tasks,” IEEE International Conference on Computer Vision, Dec. 2015, 9 pages. [cited by applicant]
Tzeng et al., “Towards adapting deep visuomotor representations from simulated to real environments,” CoRR, Apr. 2016, 10 pages. [cited by applicant]
Yosinski et al., “How transferable are features in deep neural networks?” Advances in Neural Information Processing Systems, Advances in Neural Information Processing Systems, Nov. 2014, 14 pages. [cited by applicant]
Zhu et al [online], “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” Sep. 2016, retrieved Apr. 10, 2019, URL<https://arxiv.org/pdf/1609.05143.pdf>, 8 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201780074261, dated Aug. 17, 2022, 14 pages (with English translation). [cited by applicant]