IP Library › Granted Patent US 12,226,913
Granted Patent B2
US 12,226,913 · App. 17/755,587 · Granted Feb 18, 2025

Methods and systems to remotely operate robotic devices

Inventors: Ajay U. Mandlekar (Stanford, CA); Yuke Zhu (Stanford, CA); Animesh Garg (Stanford, CA); Silvio Savarese (Stanford, CA); Fei-Fei Li (Stanford, CA)
Assignee: The Board of Trustees of the Leland Stanford Junior University
B25J9/1689B25J9/1682B25J13/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,226,913
App. No.
17/755,587
Granted
Feb 18, 2025
Kind
B2
Abstract

Methods and systems to remotely operate robotic devices are provided. A number of embodiments allow users to remotely operate robotic devices using generalized consumer devices (e.g., cell phones). Additional embodiments provide for a platform to allow communication between consumer devices and the robotic devices. Further embodiments allow for training robotic devices to operate autonomously by training the robotic device with machine learning algorithms using data collected from scalable methods of controlling robotic devices.

Claims (34)

1. A system for remotely operating a robotic device, comprising:

a plurality of network-connected robotic devices configured to perform a task;

a coordination server configured to manage a plurality of users by performing the steps of:

matching each of a first set of the plurality of users to a corresponding network-connected robotic device from the plurality of network-connected robotic devices;

creating a teleoperation process for each of the first set of the plurality of users, wherein the teleoperation process receives at least one control command from each user and transmits each user's at least one control command to the corresponding network-connected robotic device;

matching each of a second set of the plurality of users to a corresponding virtual version of the network-connected robotic devices in a simulated domain; and

creating a teleoperation process for each of the second set of the plurality of users, wherein the teleoperation process receives at least one control command from each user and transmits each user's at least one control command to the corresponding virtual version of the network-connected robotic device; and

a data collection server configured to collect data of the control commands of the first and second set of the plurality of users from the teleoperation processes.

2. The system of claim 1 , wherein the system comprises a plurality of network-connected robotic devices configured to accomplish a task, and the coordination server is configured to connect a user to each of the robotic devices in the plurality of robotic devices.

3. The system of claim 2 , wherein the coordination server places a lock on the control of each physical robot arm so that only one user may operate a robotic device at a time.

4. The system of claim 1 , wherein the teleoperation server process implements a low-pass filter to reject high-frequency user input from the user.

5. The system of claim 1 , further comprising a networked control device to allow the plurality of users to control the robotic device.

6. The system of claim 5 , wherein the control device is a smartphone.

7. A method for training robots for independent operation, comprising:

obtaining a plurality of network-connected robotic devices;

connecting managing a plurality of users seeking to control the network-connected robotic devices, wherein the user has control over the network-connected robotic device managing of users comprises:

matching each of a first set of the plurality of users to a corresponding network-connected robotic device from the plurality of network-connected robotic devices;

creating a teleoperation process for each of the first set of the plurality of users, wherein the teleoperation process receives at least one control command from each user and transmits each user's at least one control command to the corresponding network-connected robotic device;

matching each of a second set of the plurality of users to a corresponding virtual version of the network-connected robotic devices in a simulated domain; and

creating a teleoperation process for each of the second set of the plurality of users, wherein the teleoperation process receives at least one control command from each user and transmits each user's at least one control command to the corresponding virtual version of the network-connected robotic device;

training a policy based on data of the control commands of the first and second set of users; and

deploying the policy on an untrained network-connected robotic device allowing the untrained network-connected robotic device to operate independently of a human operator.

8. The method of claim 7 , further comprising collecting data of the control commands of the first and second set of users.

9. The method of claim 8 , wherein the collecting of data is accomplished by specifying a task for the user to complete using the network-connected robotic device, and solutions completed by the user are saved as data.

10. The method of claim 9 , wherein the task requires fine-grained dexterity and high-level planning.

11. The method of claim 7 , wherein the plurality of users is managed using a coordination server.

12. The method of claim 11 , wherein the coordination server implements a mutual exclusion principle to limit the number of users on a platform.

13. The method of claim 7 , wherein the teleoperation processes include a low-pass filter to reject high-frequency input from the user.

14. The method of claim 7 , wherein the plurality of users control the network-connected robotic device via a generalized consumer device.

15. The method of claim 14 , wherein the generalized consumer device is a smartphone.

16. The system of claim 1 , wherein the coordination server creates the teleoperation process in a server that is geographically near the network-connected robotic device.

17. The system of claim 1 , wherein the coordination server connects a user from the first set of the plurality of users to a network-connected robotic device in response to determining a server being geographically near the network-connected robotic device to create the teleoperation process between the network-connected robotic device and the user.

18. The method of claim 11 , wherein the coordination server creates the teleoperation process in a server that is geographically near the network-connected robotic device.

19. The method of claim 11 , wherein the coordination server connects a user from the first set of the plurality of users to a network-connected robotic device in response to determining a server being geographically near the network-connected robotic device to create the teleoperation process between the network-connected robotic device and the user.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 9, 2023
From: MANDLEKAR, AJAY U.; ZHU, YUKE; GARG, ANIMESH; SAVARESE, SILVIO; LI, FEI-FEI
To: THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY
Reel/Frame 062318/0678 →
Continuity (2)
Provisional Application 62929663 · Nov 1, 2019
Related Publication 20230226696A1 · Jul 20, 2023
References Cited (61)
US 7720572B2 · Ziegler · 2010 [cited by examiner]
US 8265793B2 · Cross et al. · 2012 [cited by applicant]
US 8588976B2 · Mangaser et al. · 2013 [cited by applicant]
US 8793205B1 · Fisher et al. · 2014 [cited by applicant]
US 9403273B2 · Payton et al. · 2016 [cited by applicant]
US 9457468B1 · Elazary et al. · 2016 [cited by applicant]
US 9592604B2 · Hoffman et al. · 2017 [cited by applicant]
US 9744669B2 · Wicks · 2017 [cited by examiner]
US 10061896B2 · Jordan et al. · 2018 [cited by applicant]
US 10079012B2 · Klimanis · 2018 [cited by examiner]
US 11878422B2 · Oyekanlu · 2024 [cited by examiner]
US 20090177323A1 · Ziegler · 2009 [cited by examiner]
US 20180157973A1 · El-yaniv et al. · 2018 [cited by applicant]
US 20190249394A1 · Mehra · 2019 [cited by examiner]
US 20220371192A1 · Oyekanlu · 2022 [cited by examiner]
KR 101751297B1 · 2017 [cited by applicant]
WO 2021087455A1 · 2021 [cited by applicant]
Robot Network 13.05 User Guide (Year: 2023). [cited by examiner]
International Preliminary Report on Patentability for International Application No. PCT/US2020/058542, Report issued May 3, 2022, Mailed May 12, 2022, 6 Pgs. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2020/058542, Search completed Feb. 21, 2021, Mailed Mar. 18, 2021, 13 Pgs. [cited by applicant]
Abbeel et al., “Autonomous Helicopter Aerobatics through Apprenticeship Learning”, The International Journal of Robotics Research, Jun. 23, 2010, vol. 29, No. 13, pp. 1608-1639, https://doi.org/10.1177/0278364910371999. [cited by applicant]
Argall et al., “A survey of robot learning from demonstration”, Robotics and Autonomous Systems, May 31, 2009, vol. 57, Issue 5, pp. 469-483, https://doi.org/10.1016/j.robot.2008.10.024. [cited by applicant]
Aytar et al., “Playing Hard Exploration Games by Watching Youtube”, Advances in Neural Information Processing Systems, 2018, pp. 2935-2945. [cited by applicant]
Boularias et al., “Relative entropy inverse reinforcement learning”, in Proceedings of the 14th Int'l Conference on Artificial Intelligence and Statistics (AISTATS), 2011, pp. 182-189. [cited by applicant]
Danielczuk et al., “Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter ”, arXiv:1903.01588v1 [cs.RO], Mar. 4, 2019, 9 pgs. [cited by applicant]
Deng et al., “ImageNet: A Large-Scale Hierarchical Image Database”, IEEE Computer Vision and Pattern Recognition, 2009, 8 pgs. [cited by applicant]
Forbes et al., “Robot Programming by Demonstration with Crowdsourced Action Fixes”, Second AAAI Conference on Human Computation and Crowdsourcing, 2014, 10 pgs. [cited by applicant]
Fritsche et al., “First-Person Teleoperation of a Humanoid Robot”, IEEE—RAS 15th Int'l Conference on Humanoid Robots (Humanoids), 2015, 6 pgs. [cited by applicant]
Goldberg, “Beyond the Web: Excavating the Real World via Mosaic”, in Second International WWW Conference, Chicago, Illinois, Oct. 17-21, 1994, 12 pgs. [cited by applicant]
Goldfeder et al., “The Columbia Grasp Database”, IEEE International Conference on Robotics and Automation, 2009, pp. 1710-1716. [cited by applicant]
Hart et al., “Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research”, Advances in Psychology, vol. 52, Elsevier, 1988, pp. 139-183. [cited by applicant]
James et al., “Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 12627… [cited by applicant]
Kalashnikov et al., “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”, arXiv:1806.10293v3 [cs.LG], Nov. 28, 2018, 23 pgs. [cited by applicant]
Kasper et al., “The KIT object models database: An object model database for object recognition, localization and manipulation in service robotics”, The International Journal of Robotics Research, 2012, vol. 31, No. 8, … [cited by applicant]
Kehoe et al., “A Survey of Research on Cloud Robotics and Automation”, IEEE Transactions on Automation Science and Engineering, 2015, vol. 12, No. 2, pp. 1-11. [cited by applicant]
Kent et al., “A Comparison of Remote Robot Teleoperation Interfaces for General Object Manipulation”, Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, ACM, 2017, pp. 371-379, https:/… [cited by applicant]
Kofman et al., “Teleoperation of a Robot Manipulator Using a Vision-Based Human-Robot Interface”, IEEE transactions on Industrial Electronics, Oct. 2005, vol. 52, Issue 5, pp. 1206-1219, first published Sep. 26, 2005, D… [cited by applicant]
Krishnan et al., “SWIRL: A Sequential Windowed Inverse Reinforcement Learning Algorithm for Robot Tasks With Delayed Rewards”, The International Journal of Robotics Research, 2019, 16 pgs. [cited by applicant]
Kurzweil, “Tracking the Acceleration of Intelligence, Stories on Progress”, printed Aug. 23, 2021 from http://www.kurzweilai.net/crowdsourcing-for-robots, published Jun. 30, 2014, 4 pgs. [cited by applicant]
Levine et al., “Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection”, arXiv:1603.02199v4 [cs.LG], Aug. 28, 2016, 12 pgs. [cited by applicant]
Lipton et al., “Baxter's Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing”, arXiv:1703.01270v1 [cs.RO], Mar. 3, 2017, 8 pgs. [cited by applicant]
Mahler et al., “Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics”, arXiv:1703.09312v3, Aug. 8, 2017, 12 pgs. [cited by applicant]
Mandlekar et al., “RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation”, arXiv:1811.02790v1 [cs.RO], Nov. 7, 2018, 15 pgs. [cited by applicant]
Marin et al., “A Multimodal Interface to Control a Robot Arm via the Web: A Case Study on Remote Programming”, IEEE Transactions on Industrial Electronics, Dec. 5, 2005, vol. 52, Issue 6, pp. 1506-1520, DOI:10.1109/TIE.… [cited by applicant]
Ng et al., “Algorithms for Inverse Reinforcement Learning”, Icml, 2000, pp. 663-670. [cited by applicant]
Penaloza et al., “Robot Reinforcement Learning using Crowdsourced Rewards”, Sep. 23, 2013, Retrieved from: http://roboearth.ethz.ch/wp-content/uploads/2013/03/final-7.pdf, 6 pgs. [cited by applicant]
Pinto et al., “Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours”, arXiv:1509.06825v1 [cs.LG], Sep. 23, 2015, 8 pgs. [cited by applicant]
Pomerleau, “ALVINN: An autonomous land vehicle in a neural network”, Advances in Neural Information Processing Systems, Jan. 1989, pp. 305-313. [cited by applicant]
Rajpurkar et al., “Know What You Don't Know: Unanswerable Questions for SQUAD”, arXiv:1806.03822v1 [cs.CL], Jun. 11, 2018, 9 pgs. [cited by applicant]
Ross et al., “Learning Monocular Reactive UAV Control in Cluttered Natural Environments”, arXiv, Robotics and Automation (ICRA), 2013 IEEE International Conference on, IEEE, Nov. 2012, 8 pgs. [cited by applicant]
Schulman et al., “Learning from Demonstrations Through the Use of Non-Rigid Registration”, Robotics Research, Springer, 2016, pp. 339-354. [cited by applicant]
Sermanet et al., “Time-Contrastive Networks: Self-Supervised Learning from Video”, arXiv:1704.06888v3 [cs.CV], Mar. 20, 2018, 15 pgs. [cited by applicant]
Sharma et al., “Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation”, arXiv:1810.07121v1 [cs.RO], Oct. 16, 2018, 10 pgs. [cited by applicant]
Sorokin et al., “People Helping Robots Helping People: Crowdsourcing for Grasping Novel Objects”, retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.5042&rep=rep1&type=pdf, Nov. 19, 2018, 6 pgs. [cited by applicant]
Sung et al., “Robobarista: Object Part based Transfer of Manipulation Trajectories from Crowd-Sourcing in 3D Pointclouds”, Robotics Research, Springer, 2018, 6 pgs. [cited by applicant]
Toris et al., “The Robot Management System: A Framework for Conducting Human-Robot Interaction Studies Through Crowdsourcing”, Journal of Human-Robot Interaction, 2014, vol. 3, No. 2, pp. 25-49, DOI 10.5898/JHRI.3.2.Tor… [cited by applicant]
Vecerik et al., “Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards”, arXiv:1707.08817v1 [cs.AI], Jul. 27, 2017, 10 pgs. [cited by applicant]
Whitney et al., “Comparing Robot Grasping Teleoperation across Desktop and Virtual Reality With ROS Reality”, Robotics Research, The 18th International Symposium ISRR, Nov. 2019, pp. 335-350, DOI:10.1007/978-3-030-28619… [cited by applicant]
Yan et al., “Learning grasping interaction with geometry-aware 3D representations”, arXiv:1708.07303v1 [cs.RO], Aug. 24, 2017, 10 pgs. [cited by applicant]
Yu et al., “More than a Million Ways to be Pushed. A High-Fidelity Experimental Dataset of Planar Pushing”, arXiv:1604.04038v2 [cs.RO], Aug. 4, 2016, 8 pgs. [cited by applicant]
Zhang et al., “Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation”, arXiv:1710.04615v2 [cs.LG], Mar. 6, 2018, 9 pgs. [cited by applicant]
Cited By (1)
US 12,420,426