IP Library › Granted Patent US 12,399,567
Granted Patent B2
US 12,399,567 · App. 16/932,067 · Granted Aug 26, 2025

Vision-based teleoperation of dexterous robotic system

Inventors: Ankur Handa (Seattle, WA); Karl Van Wyk (Lynnwood, WA); Wei Yang (Seattle, WA); Yu-Wei Chao (Seattle, WA); Dieter Fox (Seattle, WA); Qian Wan (Seattle, WA)
Assignee: NVIDIA Corporation
G06F3/017B25J9/1612B25J9/1628B25J9/1689B25J9/1697B25J13/08B25J13/084B25J15/0009G06T1/0014G06V10/764G06V10/82G06V20/56G06V20/59G06V20/64G06V40/11G06V40/28
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,399,567
App. No.
16/932,067
Filed
Jul 17, 2020
Granted
Aug 26, 2025
Kind
B2
Art Unit
3658
USPC
700/259
Abstract

A human pilot controls a robotic arm and gripper by simulating a set of desired motions with the human hand. In at least one embodiment, one or more images of the pilot's hand are captured and analyzed to determine a set of hand poses. In at least one embodiment, the set of hand poses is translated to a corresponding set of robotic-gripper poses. In at least one embodiment, a set of motions is determined that perform the set of robotic-gripper poses, and the robot is directed to perform the set of motions.

Claims (75)

1. A computer-implemented method comprising:

determining, from an image of a hand, at least one first joint angle of a first joint of at least one digit of a model of the hand;

determining a distance and a direction from the at least one digit to another digit of the model of the hand;

translating the at least one first joint angle of the first joint to at least one second joint angle of a corresponding second joint of a robotic hand based at least in part on the distance, the direction, and a cost function that maintains at least a minimum separation between at least a pair of robotic digits of the robotic hand;

determining a set of movements that repositions the robotic hand from a first pose to a second pose based at least in part on the at least one second joint angle of the corresponding second joint; and

causing the robotic hand to perform the set of movements.

2. The computer-implemented method of claim 1 , further comprising:

obtaining the image from a depth camera;

determining a point cloud of the hand from the image; and

determining the at least one first joint angle of the first joint of the at least one digit of the model of the hand using the point cloud.

3. The computer-implemented method of claim 2 , wherein:

the depth camera is an RGB depth camera, a radar imager, a medical imaging system, or a LIDAR system.

4. The computer-implemented method of claim 1 , further comprising:

determining a third pose of the model of the hand, wherein the third pose includes information that identifies a location of each of five fingers of the hand; and

the third pose includes information that identifies one or more joint locations of the hand.

5. The computer-implemented method of claim 1 , wherein:

the at least one first joint angle of the first joint is translated to the at least one second joint angle of the corresponding second joint by at least performing kinematic retargeting of the at least one first joint angle of the first joint.

6. The computer-implemented method of claim 1 , wherein the set of movements are determined using Reimannian motion policies.

7. The computer-implemented method of claim 1 , wherein the robotic hand is an articulated robotic hand, a robotic gripper, or a probe.

8. The computer-implemented method of claim 1 , wherein:

the robotic hand includes one or more tactile sensors that provide tactile sensor information; and

the tactile sensor information includes a 2-dimensional array of force values for a digit of the robotic hand.

9. The computer-implemented method of claim 1 , wherein translating the at least one first joint angle to the at least one second joint angle comprises minimizing the cost function.

10. The computer-implemented method of claim 1 , wherein the distance and the direction are determined as components of a vector, and the at least one first joint angle is translated to the at least one second joint angle based at least in part on the vector.

11. A system comprising:

one or more processors; and

computer-readable memory storing executable instructions that, as a result of being executed by the one or more processors, cause the system to:

determine, from an image of an appendage performing a task, at least one first joint angle of a first joint of at least one digit of the appendage;

determine a distance and a direction from the at least one digit to another digit of the appendage;

determine at least one second joint angle of a second joint of a robotic gripper based at least in part on the distance, the direction, and a cost function that maintains at least a minimum separation between individually movable portions of the robotic gripper;

determine a set of movements that repositions the robotic gripper from a first pose to a second pose based at least in part on the at least one second joint angle of the second joint of the robotic gripper; and

perform the set of movements to position the robotic gripper in the second pose to cause the robotic gripper to perform the task.

12. The system of claim 11 , wherein the executable instructions cause the system to further:

generate a point cloud from the image of the appendage; and

determine the at least one first joint angle of the first joint of the at least one digit of the appendage from the point cloud.

13. The system of claim 11 , wherein the set of movements are performed in accordance with Reimannian motion policies.

14. The system of claim 11 , wherein the appendage is a human hand or human foot.

15. The system of claim 11 , wherein the executable instructions cause the system to apply a scale factor to the at least one first joint angle of the first joint to determine the at least one second joint angle of the second joint of the robotic gripper.

16. The system of claim 11 , wherein the executable instructions cause the system to determine:

a third pose of the appendage, wherein the third pose of the appendage specifies a set of joint angles of the first joint comprising the at least one first joint angle of the first joint.

17. The system of claim 11 , wherein the second pose specifies a set of target joint angles of the second joint of the robotic gripper and a position of the robotic gripper.

18. The system of claim 11 , wherein the at least one second joint angle of the second joint of the robotic gripper is determined using a subset of the at least one first joint angle of the first joint of the at least one digit of the appendage.

19. The system of claim 11 , wherein the cost function comprises at least the distance and the direction.

20. Non-transitory computer-readable media storing instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to:

determine, from an image of a human performing a task, at least one first joint angle of a joint of at least one digit of a model of an appendage of the human;

determine a distance and a direction from the at least one digit to another digit of the model of the appendage;

translate the at least one first joint angle of the first joint to at least one second joint angle of a second joint of a robotic hand based at least in part on the distance, the direction, and a cost function that maintains at least a minimum separation between at least a pair of robotic digits of the robotic hand;

determine a set of movements that repositions the robotic hand from a first pose to a second pose based at least in part on the at least one second joint angle of the second joint of the robotic hand; and

cause the robotic hand to perform the set of movements.

21. The non-transitory computer-readable media of claim 20 , wherein the instructions cause the computer system to further:

obtain the image from a depth camera;

determine a point cloud of the appendage from the image; and

determine the at least one first joint angle of the first joint based at least in part on the point cloud.

22. The non-transitory computer-readable media of claim 21 , wherein the depth camera is an RGB depth camera, a radar imager, a medical imaging system, or a LIDAR system.

23. The non-transitory computer-readable media of claim 20 , wherein the instructions cause the computer system to determine a third pose of the model of the appendage,

the third pose includes information that identifies a location of each of five fingers of a hand, and

the third pose includes information that identifies one or more joint locations of the hand.

24. The non-transitory computer-readable media of claim 20 , wherein the instructions cause the computer system to further:

determine a point cloud of the appendage from the image, wherein the point cloud provides three-dimensional data describing a hand from which the at least one first joint angle of the first joint can be determined.

25. The non-transitory computer-readable media of claim 20 , wherein:

the at least one first joint angle of the first joint is translated to the at least one second joint angle of the second joint of the robotic hand by at least performing kinematic retargeting of the at least one first joint angle of the first joint of the at least one digit of the model of the appendage.

26. The non-transitory computer-readable media of claim 20 , wherein:

the robotic hand includes one or more tactile sensors that provide tactile sensor information; and

the tactile sensor information includes a 2-dimensional array of force values for a digit of the robotic hand.

27. The non-transitory computer-readable media of claim 20 , wherein the robotic hand is an articulated robotic hand, a robotic gripper, or a probe.

28. The non-transitory computer-readable media of claim 20 , wherein the instructions cause the computer system to further:

determine a point cloud of the appendage from the image; and

determine a joint structure and joint angles of the first joint of the at least one digit of the model of the appendage of the human.

29. The non-transitory computer-readable media of claim 20 , wherein the first pose of the robotic hand is a present pose of the robotic hand.

30. A robot comprising:

an arm that includes one or more articulated members connected via one or more servo motors;

a robotic appendage connected to the arm;

one or more processors; and

the non-transitory computer-readable media of claim 20 connected to the one or more processors.

31. The non-transitory computer-readable media of claim 20 , wherein the cost function is to avoid collisions between at least the pair of robotic digits.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 17, 2020
From: HANDA, ANKUR; VAN WYK, KARL; YANG, WEI; CHAO, YU-WEI; FOX, DIETER; WAN, QIAN
To: NVIDIA CORPORATION
Reel/Frame 053242/0459 →
Continuity (2)
Provisional Application 62903671 · Sep 20, 2019
Related Publication 20210086364A1 · Mar 25, 2021
References Cited (202)
US 6229230B1 · Wing · 2001 [cited by applicant]
US 8989902B1 · Crawford · 2015 [cited by examiner]
US 9321176B1 · Sun et al. · 2016 [cited by applicant]
US 10062354B2 · Dutt · 2018 [cited by applicant]
US 10186081B2 · Taylor · 2019 [cited by examiner]
US 10406686B2 · Boca et al. · 2019 [cited by applicant]
US 10757861B2 · Robertson et al. · 2020 [cited by applicant]
US 10768708B1 · Sills · 2020 [cited by examiner]
US 20030210259A1 · Liu et al. · 2003 [cited by applicant]
US 20130211592A1 · Kim · 2013 [cited by applicant]
US 20150239127A1 · Barajas et al. · 2015 [cited by applicant]
US 20150253864A1 · Parkhomenko et al. · 2015 [cited by applicant]
US 20160335790A1 · Fleishman · 2016 [cited by examiner]
US 20190318659A1 · Hamadani et al. · 2019 [cited by applicant]
US 20200030973A1 · Goto · 2020 [cited by examiner]
US 20200306959A1 · Huang et al. · 2020 [cited by applicant]
US 20200394500A1 · Hill et al. · 2020 [cited by applicant]
US 20220080581A1 · Wang et al. · 2022 [cited by applicant]
US 20230226696A1 · Mandlekar et al. · 2023 [cited by applicant]
US 20240180624A1 · Xu et al. · 2024 [cited by applicant]
CN 104589356A · 2015 [cited by applicant]
CN 109189206A · 2019 [cited by applicant]
CN 109934155A · 2019 [cited by applicant]
DE 112004002219T5 · 2006 [cited by applicant]
DE 602005006126T2 · 2009 [cited by applicant]
DE 102013113459B4 · 2015 [cited by applicant]
DE 102016009030B4 · 2019 [cited by applicant]
DE 112019001507T5 · 2020 [cited by applicant]
DE 102020124285A1 · 2021 [cited by applicant]
EP 3771522A1 · 2021 [cited by applicant]
JP 2153922A · 1992 [cited by examiner]
JP H0446778A · 1992 [cited by applicant]
JP 2011110620A · 2011 [cited by applicant]
JP 2012223864A · 2012 [cited by applicant]
JP 2015071206A · 2015 [cited by applicant]
KR 20090118153A · 2009 [cited by applicant]
WO 2019021058A2 · 2019 [cited by applicant]
Zhu et al., “Kinematic Self Retargeting: A Framework for Human Pose Estimation”, Computer Vision and Understanding 114, 2010, 14 Pages (Year: 2010). [cited by examiner]
Ratliff et al., “Riemannian Motion Policies”, arXiv:1801.02854v3, 2018, 15 Pages (Year: 2018). [cited by examiner]
Su et al., “Learning to Switch Between Sensorimotor Primitives Using Multimodal Haptic Signals”, International Conference on Simulation of Adaptive Behavior, 2017, 14 Pages (Year: 2017). [cited by examiner]
Antotsiou et al., “Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation”, ECCV 2018, 15 pages (Year: 2018). [cited by examiner]
German Office Action for Application No. 102020127508.0, mailed Jan. 19, 2022, 12 pages. [cited by applicant]
Abd et al., “Direction of Slip Detection for Adaptive Grasp Force Control with a Dexterous Robotic Hand,” IEEE/ASME International Conference on Advanced Intelligent Mechatronics, 2018, 22 pages. [cited by applicant]
Andrychowicz et al., “Learning Dexterous In-Hand Manipulation,” Aug. 28, 2018, 27 pages. [cited by applicant]
Antotsiou et al., “Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation,” Proceedings of the European Conference on Computer Vision, 2018, 15 pages. [cited by applicant]
A'Lvarez et al., “Tactile-Based in-Hand Object Pose Estimation,” Iberian Robotics Conference, Springer, 2017, 13 pages. [cited by applicant]
Bimbo et al., “Global Estimation of an Objects Pose using Tactile Sensing,” Advanced Robotics, Feb. 20, 2015, 15 pages. [cited by applicant]
Bimbo et al., “In-Hand Object Pose Estimation using Covariance-Based Tactile to Geometry Matching,” IEEE Robotics and Automation Letters, Jan. 2016, 8 pages. [cited by applicant]
Cao et al., “Openpose: Realtime Multi-person 2D Pose Estimation using Part Affinity Fields,” Dec. 18, 2018, 14 pages. [cited by applicant]
Chebotar et al., “BIGS: Biotac Grasp Stability Dataset,” ICRA 2016 Workshop on Grasping and Manipulation Datasets, 2016, 2 pages. [cited by applicant]
Chebotar et al., “Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience,” 2019 IEEE International Conference on Robotics and Automation (ICRA), May 20, 2019, 9 pages. [cited by applicant]
Chebotar et al., “Learning Robot Tactile Sensing for Object Manipulation,” IROS, 2014, 8 pages. [cited by applicant]
Cheng et al., “RMPflow : A Computational Graph for Automatic Motion Policy Generation,” Apr. 5, 2019, 45 pages. [cited by applicant]
Choi et al., “Using Vision for Pre-and Postgrasping Object Localization for Soft Hands,” International Symposium on Experimental Robotics, Springer, 2016, 12 pages. [cited by applicant]
Corcoran et al., “A Measurement Model for Tracking Hand-Object State During Dexterous Manipulation,” ICRA, 2010, 7 pages. [cited by applicant]
Cyberglove Systems “CyberGlove Systems,” retrieved from Internet on May 20, 2021, from www.cyberglovesystems.com., 2017, 5 pages. [cited by applicant]
Deng et al., “PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking,” May 22, 2019, retreived Oct. 22, 2020, from https://arxiv.org/pdf/1905.09304.pdf, 10 pages. [cited by applicant]
Diftler et al., “Robonaut 2—Initial Activities on-board the ISS,” IEEE Aerospace Conference, 2012, 12 pages. [cited by applicant]
Ding et al., “In-Hand Grasping Pose Estimation using Particle Filters in Combination with Haptic Rendering Models,” International Journal of Humanoid Robotics, 2018, 20 pages. [cited by applicant]
Dorner, “Chasing the Colour Glove: Visual Hand Tracking,” Ph.D. Dissertation, Simon Fraser University, 1994, 105 pages. [cited by applicant]
Duff et al., “Physical Simulation for Monocular 3D Model Based Tracking,” ICRA, 2011, 8 pages. [cited by applicant]
Eade, “Lie Groups for 2D and 3D Transformations,” retrieved on May 19, 2021, from http://ethaneade.com/lie.pdf, Dec. 2013, 25 pages. [cited by applicant]
Ge et al., “Point-to-Point Regression Pointnet for 3D Hand Pose Estimation,” ECCV, 2018, 17 pages. [cited by applicant]
Han et al., “Online Optical Marker-Based Hand Tracking with Deep Labels,” ACM TOG, 2018, 10 pages. [cited by applicant]
Handa et al., “DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System,” Oct. 14, 2019, 18 pages. [cited by applicant]
Jang et al., “Hierarchical Fingertip Space: A Unified Framework for Grasp Planning and In-Hand Grasp Adaptation,” IEEE Transactions on Robotics, 32(4): 2016, 13 pages. [cited by applicant]
Hansen et al., “Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES),” Evolutionary Computation, 11(1): 2003, 18 pages. [cited by applicant]
HaptX, “HaptX,” retrieved from Internet on May 20, 2021, from https:/haptx.com/., 11 pages. [cited by applicant]
Hasson et al., “Learning Joint Reconstruction of Hands and Manipulated Objects,” In Proc. CVPR, , dated Apr. 11, 2019, pp. 11807-11816. [cited by applicant]
He et al., “Deep Residual Leaming for Image Recognition,” CVPR, 2016, 9 pages. [cited by applicant]
Hebert et al., “Fusion of Stereo Vision, Force-Torque, and Joint Sensors for Estimation of In-Hand Object Location,” ICRA, 2011, 7 pages. [cited by applicant]
Hinterstoisser et al., “Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes,” ACCV, 2012, 14 pages. [cited by applicant]
Hu et al., “Tetrahedral Meshing in the Wild,” ACM Trans. Graph, 37(4): Jul. 2018, 14 pages. [cited by applicant]
IEEE, “IEEE Standard 754-2008 (Revision of IEEE Standard 754-1985): IEEE Standard for Floating-Point Arithmetic,” Aug. 29, 2008, 70 pages. [cited by applicant]
Imgaug, “Imgaug,” retrieved from the Internet on May 20, 2021, from https://github.com/aleju/imgaug, Jun. 1, 2020, 25 pages. [cited by applicant]
Zatt et al., “Tracking Objects with Point Clouds from Vision and Touch,” ICRA, 2017, 8 pages. [cited by applicant]
Jaderberg et al., “Population Based Training of Neural Networks,” Nov. 28, 2017, 21 pages. [cited by applicant]
Javdani et al., “Efficient Touch Based Localization Through Submodularity,” ICRA, 2013, 8 pages. [cited by applicant]
Johansson et al., “Coding and use of Tactile Signals from the Fingertips in Object Manipulation Tasks,” Nature reviews, Neuroscience, vol. 10, May 2009, pp. 345-359. [cited by applicant]
Johansson et al., “Roles of Glabrous Skin Receptors and Sensorimotor Memory in Automatic Control of Precision Grip when Lifting Rougher or More Slippery Objects,” Experimental Brain Research, vol. 56, No. 3, Oct. 1984, … [cited by applicant]
Johansson et al.,“Tactile Sensory Coding in the Glabrous Skin of the Human Hand,” Trends in Neurosciences, vol. 6, 1983, 6 pages. [cited by applicant]
Johnson, “The Nlopt Nonlinear-Optimization Package,” retreived from the Internet May 20, 2021, from https://github.com/stevengj/nlopt, Nov. 18, 2020, 2 pages. [cited by applicant]
Kokic et al., “Learning to Estimate Pose and Shape of Hand-Held Objects from RGB Images,” Nov. 11, 2019, 8 pages. [cited by applicant]
Koval et al., “Pose Estimation for Planar Contact Manipulation with Manifold Particle Filters,” The International Journal of Robotics Research, 2015, 8 pages. [cited by applicant]
Kraft, “A Software Package for Sequential Quadratic Programming,” Forschungsbericht-Deutsche Forschungs- und Versuchsanstalt fur Luft- und Raumfahrt, 1988, 33 pages. [cited by applicant]
Kraft, “Algorithm 733: Tomp-Fortran Modules for Optimal Control Calculations,” TOMS, 1994, 20 pages. [cited by applicant]
Kumar et al., “MuJoCo HAPTIX: A Virtual Reality System for Hand Manipulation,” Humanoids, 2015, 7 pages. [cited by applicant]
Lambert et al., “Joint Inference of Kinematic and Force Trajectories with Visuo-Tactile Sensing,” Mar. 8, 2019, 7 pages. [cited by applicant]
Lee et al., “Making Sense of Vision and Touch: Self-supervised Learning of Multimodal Representations for Contact-Rich Tasks,” Oct. 24, 2018, 8 pages. [cited by applicant]
Li et al. “Localization and Manipulation of Small Parts Using GelSight Tactile Sensing,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, Sep. 14, 2014, pp. 3988-3993. [cited by applicant]
Li et al., “A Comparative Study of Contact Models for Contact-Aware State Estimation,” IROS, 2015, 6 pages. [cited by applicant]
Li et al., “DeepIM: Deep Iterative Matching for 6D Pose Estimation,” ECCV, 2018, 16 pages. [cited by applicant]
German Office Action for Application No. 102020124285.9, dated Jun. 21, 2021, 7 pages. [cited by applicant]
Li et al., “Vision-Based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network,” ICRA, 2019, 7 pages. [cited by applicant]
Liang et al., “GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning,” CoRL, Oct. 24, 2018, 14 pages. [cited by applicant]
Liu et al., “A Glove-based System for Studying Hand-Object Manipulation via Joint Pose and Force Sensing,” IROS, 2017, 8 pages. [cited by applicant]
Liu et al., “High-Fidelity Grasping in Virtual Reality using a Glove-based System,” ICRA, 2019, 7 pages. [cited by applicant]
Luo et al., “Robotic Tactile Perception of Object Properties: A Review,” Mechatronics, Nov. 10, 2017, 17 pages. [cited by applicant]
Lynch et al., “Learning Latent Plans from Play,” Dec. 20, 2019, 17 pages. [cited by applicant]
Macklin et al., “Non-Smooth Newton Methods for Deformable Multi-Body Dynamics,” Jul. 10, 2019, 21 pages. [cited by applicant]
N.I. of Standards and Technology, “Robotic Grasping and Manipulation for Assembly,” retrieved from the Internet on May 20, 2021, from https://www.nist.gov/el/intelligent-systems-division-73500/robotic-grasping-and-manip… [cited by applicant]
Norton et al., “Analysis of Human-Robot Interaction at the DARPA Robotics Challenge Finals,” The International Journal of Robotics Research, 36(5-7): 2017, 31 pages. [cited by applicant]
Oberweger et al., “Generalized Feedback Loop for Joint Hand-Object Pose Estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Mar. 25, 2019, 15 pages. [cited by applicant]
Oberweger et al., “Hands Deep in Deep Learning for Hand Pose Estimation,” dated Feb. 24, 2015, 10 pages. [cited by applicant]
Orocos Kinematics and Dynamics, “Orocos Kinematics and Dynamics Library,” retrieved from the Internet on May 20, 2021, from https://github.com/orocos/orocos_kinematics_dynamics, 2 pages. [cited by applicant]
Peters et al., “Relative Entropy Policy Search,” Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010, 6 pages. [cited by applicant]
Petrovskaya et al., “Global Localization of Objects via Touch,” IEEE Transactions on Robotics, 2011, 17 pages. [cited by applicant]
Pezzementi et al., “Object Mapping, Recognition, and Localization from Tactile Geometry,” ICRA, 2011, 7 pages. [cited by applicant]
Pfanne et al., “Fusing Joint Measurements and Visual Features for In-Hand Object Pose Estimation,” IEEE Robotics and Automation Letters, Jun. 2018, 8 pages. [cited by applicant]
Platt et al., “Using Bayesian Filtering to Localize Flexible Materials During Manipulation,” IEEE Transactions on Robotics, 2011, 13 pages. [cited by applicant]
Qi et al., “Pointnet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,” Jun. 7, 2017, 14 pages. [cited by applicant]
Rajeswaran et al., “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations,” RSS, Jun. 26, 2018, 9 pages. [cited by applicant]
Ratliff et al. “Riemannian Motion Policies,” arXiv:1801.02854v3, dated Jul. 25, 2018, 15 pages. [cited by applicant]
Romero et al., “Embodied Hands: Modeling and Capturing Hands and Bodies Together,” ACM SIGGRAPH, 2017, 17 pages. [cited by applicant]
Saund et al., “Touch Based Localization of Parts for High Precision Manufacturing,” ICRA, 2017, 8 pages. [cited by applicant]
Schmidt et al. “Depth-Based Tracking with Physical Constraints for Robot Manipulation,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), May 26, 2015, pp. 119-126. [cited by applicant]
Schmidt et al., “Dart: Dense Articulated Real-Time Tracking,” RSS, IEEE, 2014, 9 pages. [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201609, issued Jan… [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201806, issued Jan… [cited by applicant]
Stenger et al., “Model-Based Hand Tracking using a Hierarchical Bayesian Filter,” PAMI, 2006, 13 pages. [cited by applicant]
Sterbis et al., “Transcontinental Telesurgical Nephrectomy using the Da Vinci Robot in a Porcine Model,” Urology, 71(5): 2008, 3 pages. [cited by applicant]
Su et al., “Force Estimation and Slip Detection/Classification for Grip Control using a Biomimetic Tactile Sensor,” Humanoids, IEEE, 2015, 7 pages. [cited by applicant]
Sundaralingam et al., “Robust Learning of Tactile Force Estimation through Robot Interaction,” International Conference on Robotics and Automation, Mar. 5, 2019, 8 pages. [cited by applicant]
Sundaram et al., “Learning the Signatures of the Human Grasp using a Scalable Tactile Glove,” Nature, 569(7758): 2019, 19 pages. [cited by applicant]
Theobalt et al., “Pitching a Baseball: Tracking High-Speed Motion with Multi-Exposure Images,” ACM TOG, 2004, 8 pages. [cited by applicant]
Tompson et al., “Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks,” ACM Transactions on Graphics, 2014, 10 pages. [cited by applicant]
Van Wyk et al., “Robotic Grasping and Manipulation Competition: Future Tasks to Support the Development of Assembly Robotics,” Robotic Grasping and Manipulation Challenge, Springer, 2016, 11 pages. [cited by applicant]
Vardi et al., “The Multivariate L1-Median and Associated Data Depth,” Proceedings of the National Academy of Sciences, 2000, 4 pages. [cited by applicant]
Vezzani et al., “Memory Unscented Particle Filter for 6-DOF Tactile Localization,” IEEE Transactions on Robotics, 2016, 15 pages. [cited by applicant]
Villegas et al., “Neural Kinematic Networks for Unsupervised Motion Retargetting,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, 10 pages. [cited by applicant]
Wang et al., “Real-Time Hand-Tracking with a Color Glove,” ACM TOG, 2009, 8 pages. [cited by applicant]
Wyk et al., “Comparative Peg-in-Hole Testing of a Force-Based Manipulation Controlled Robotic Hand,” IEEE Transaction on Robotics, 34(2): Apr. 2018, 8 pages. [cited by applicant]
Xhalon et al., “Online In-Hand Object Localization,” IROS, 2013, 8 pages. [cited by applicant]
Xiang et al., “PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes,” May 26, 2018, 10 pages. [cited by applicant]
Yu et al., “Realtime State Estimation with Tactile and Visual Sensing. Application to Planar Manipulation,” ICRA, 2018, 8 pages. [cited by applicant]
Yuan et al., “Big-Hand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis,” CVPR, 2017, 9 pages. [cited by applicant]
Zhang et al., “A Dynamic Bayesian Approach to Real-Time Estimation and Filtering in Grasp Acquisition,” ICRA, 2013, 8 pages. [cited by applicant]
Zhang et al., “Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation,” ICRA, IEEE, Mar. 6, 2018, 9 pages. [cited by applicant]
Zhang, “Making Convolutional Networks Shift-Invariant Again,” ICML, 2019, 11 pages. [cited by applicant]
Zhu et al., “Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost,” ICRA, Oct. 14, 2018, 8 pages. [cited by applicant]
Zhuang et al., “Shared Human-Robot Proportional Control of a Dexterous Myoelectric Prosthesis,” Nature Machine Intelligence, 2019, 24 pages. [cited by applicant]
Ajili et al., “Gesture Recognition for Humanoid Robot Teleoperation,” 26th IEEE International Symposium on Robot and Human Interactive Communication, 2017, 7 pages. [cited by applicant]
Aronson et al., “Gaze Complements Control Input for Goal Prediction During Assisted Teleoperation,” Robotics Science and Systems, 2022, 12 pages. [cited by applicant]
Arunachalam et al., “Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation,” 2022, 9 pages. [cited by applicant]
Arunachalam et al., “Holo-Dex: Teaching Dexterity with Immersive Mixed Reality,” 2022, 12 pages. [cited by applicant]
Chen et al., “ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning,” ICRA, 2022, 19 pages. [cited by applicant]
Coumans et al., “PyBullet, A Python Module for Physics Simulation for Games,” Robotics and Machine Learning, retrieved http://pybullet.org, 2016, 10 pages. [cited by applicant]
Danchilla et al., “Three.js Framework,” Beginning WebGL for HTML5, 2012, 31 pages. [cited by applicant]
Deits, “Meshcat,” GitHub, retrieved from https://github.com/rdeits/, 2018, 3 pages. [cited by applicant]
Du et al., “Markerless kinect-based hand tracking for robot teleoperation,” International Journal of Advanced Robotic Systems, 9(2):36, 2012, 10 pages. [cited by applicant]
Elsner et al., “PARTI-A Haptic Virtual Reality Control Station for Model-Mediated Robotic Applications” Frontiers in Virtual Reality, Jul. 2022, 10 pages. [cited by applicant]
Fang et al., “Vision-based Posture-Consistent Teleoperation of Robotic Arm Using Multi-stage Deep Neural Network,” Robotics and Autonomous Systems, 2020, 8 pages. [cited by applicant]
Gan et al., “ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation,” 2020, 23 pages. [cited by applicant]
Gharaybeh et al., “Telerobotic Control in Virtual Reality,” IEEE, 2019, 8 pages. [cited by applicant]
Handa et al., “DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System,” 2019, 18 pages. [cited by applicant]
Hedayati et al., “Improving Collocated Robot Teleoperation with Augmented Reality,” International Conference on Human Robot Interaction, 2018, 10 pages. [cited by applicant]
Huang et al., “A Novel Training and Collaboration Integrated Framework for Human-Agent Teleoperation,” Sensors, 21(24): 2021, 15 pages. [cited by applicant]
IEEE “IEEE Standard for Floating-Point Arithmetic”, Microprocessor Standards Committee of the IEEE Computer Society, IEEE Std 754-2008, dated Jun. 12, 2008, 70 pages. [cited by applicant]
Jang et al., “BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning,” Feb. 4, 2022, 23 Pages. [cited by applicant]
Khadir et al., “Teleoperator Imitation with Continuous-time Safety,” 2019, 9 pages. [cited by applicant]
Kofman et al., “Robot-Manipulator Teleoperation by Markerless Vision-Based Hand-Arm Tracking,” International Journal of Optomechatronics, 2007, 28 pages. [cited by applicant]
Kofman et al., “Teleoperation of a Robot Manipulator Using a Vision-Based Human-Robot,” IEEE transactions on industrial electronics, 52(5): Oct. 2005, 14 pages. [cited by applicant]
Kumar et al., “MuJoCo HAPTIX: A Virtual Reality System for Hand Manipulation,” International Conference on Humanoid Robots, 2015, 7 pages. [cited by applicant]
Li et al., “A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision,” IEEE Transactions on Cybernetics, 2022, 12 pages. [cited by applicant]
Li et al., “A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU,” 2020, 7 pages. [cited by applicant]
Liang et al., “ In-Hand Object Pose Tracking via Contact Feedback and GPU-Accelerated Robotic Simulation,” IEEE International Conference on Robotics and Automation, 2020, 7 pages. [cited by applicant]
Lipton et al., “Baxter's Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing” IEEE Robotics and Automation Letters, 3(1): 2017, 8 pages. [cited by applicant]
Liu et al., “Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time,” Jun. 9, 2021, 13 Pages. [cited by applicant]
Makoviychuk et al., “Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning,” Aug. 25, 2021, 32 pages. [cited by applicant]
Mandlekar et al., “Human-in-the-Loop Imitation Learning using Remote Teleoperation,” Dec. 12, 2020, 8 Pages. [cited by applicant]
Mandlekar et al., “Roboturk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation,” Conference on Robot Learning, Nov. 7, 2018, 15 pages. [cited by applicant]
Meeker et al., “A Continuous Teleoperation Subspace with Empirical and Algorithmic Mapping Algorithms for Non-Anthropomorphic Hands,” IEEE Transactions on Automation Science and Engineering, 19(1): 2020, 14 pages. [cited by applicant]
Mizera et al., “Evaluation of Hand-Tracking Systems in Teleoperation and Virtual Dexterous Manipulation,” IEEE Sensors Journal, 20(3): 2020, 14 pages. [cited by applicant]
Mosbach et al., “Accelerating Interactive Human-like Manipulation Learning with GPU-based Simulation and High-quality Demonstrations,” Humanoids, Nov. 2022, 7 pages. [cited by applicant]
Muelling et al., “Autonomy Infused Teleoperation with Application to BCI Manipulation,” 2015, 10 pages. [cited by applicant]
Niemeyer et al., “Telerobotics,” Springer Handbook of Robotics, 2016, 17 pages. [cited by applicant]
Pavlakos et al., “Expressive Body Capture: 3D Hands, Face, and Body from a Single Image,” CVPR, 2019, pp. 10975-10985. [cited by applicant]
Ponomareva et al., “GraspLook: a VR-based Telemanipulation System with R-CNN-driven Augmentation of Virtual Environment,” International Conference on Advanced Robotics, 2021, 6 pages. [cited by applicant]
Qin et al., “Object-Independent Human-to-Robot Handovers using Real Time Robotic Vision” 2020, 7 pages. [cited by applicant]
Radosavovic et al., “Real-World Robot Learning with Masked Visual Pre-training,” 2022, 13 pages. [cited by applicant]
Rakita et al., “A Motion Retargeting Method for Effective Mimicry-based Teleoperation of Robot Arms,” ACM/IEEE International Conference on Human-Robot Interaction, 2017, 10 pages. [cited by applicant]
Rakita et al., “Remote Telemanipulation with Adapting Viewpoints in Visually Complex Environments,” Robotics: Science and Systems XV, 2019, 10 pages. [cited by applicant]
Rong et al., “FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration,” 2020, 13 pages. [cited by applicant]
Rosen et al., “Mixed Reality as a Bidirectional Communication Interface for Human-Robot Interaction,” IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020, 8 pages. [cited by applicant]
Salvato et al., “Predicting Hand-Object Interaction for Improved Haptic Feedback in Mixed Reality,” Robotics and Automation Letters, 7(2): 2022, 7 pages. [cited by applicant]
Shaw et al., “VideoDex: Learning Dexterity from Internet Videos,” 2022, 17 pages. [cited by applicant]
Sivakumar et al., “Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube,” 2022, 15 pages. [cited by applicant]
Son et al., “Human-Centered Design and Evaluation of Haptic Cueing for Teleoperation of Multiple Mobile Robots,” IEEE Transactions on Cybernetics, 43(2): 2013, 13 pages. [cited by applicant]
Sundaralingam et al., “Curobo: Parallelized collision free robot motion generation,” Proceedings of the IEEE International Conference on Robotics and Automation, Jun. 2023, 8 pages. [cited by applicant]
Todorov et al., “MuJoCo : A Physics Engine for Model-Based Control,” 2012, 8 pages. [cited by applicant]
Tung et al., “Learning Multi-Arm Manipulation Through Collaborative Teleoperation,” Dec. 12, 2020, 7 Pages. [cited by applicant]
Xiang et al., “SAPIEN: A SimulAted Part-based Interactive ENvironment,” Mar. 19, 2020, 16 pages. [cited by applicant]
Zhang et al., “A Feasibility Study on an Intuitive Teleoperation System Combining IMU with sEMG Sensors,” Intelligent Robotics and Applications: 11th International Conference, 2018, 11 pages. [cited by applicant]
Zhang et al., “Human Robot Motion Retargeting via Neural Latent Optimization,” 2021, 8 pages. [cited by applicant]
Zhang et al., “MediaPipe Hands: On-device Real-time Hand Tracking,” 2020, 5 pages. [cited by applicant]
Zhao et al., “Combining Marker-based Mocap and RGB-D Camera for Acquiring High-fidelity Hand Motion Data” Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 2012, 10 pages. [cited by applicant]
Office Action for Chinese Application No. 202010973239.2, mailed Mar. 27, 2024, 29 pages. [cited by applicant]
Lei et al., “Inverse Kinematics Algorithm for Industrial Robot of ABB1410 Based on Screw Theory,” Machinery Design & Manufacture, 2016, 4 pages. [cited by applicant]
Office Action for Chinese Application No. 202010973239.2, mailed Jun. 25, 2024, 17 pages. [cited by applicant]
Yang et al., “Online Retargeting for Multi-lateral Teleoperation,” International Conference on Ubiquitous Robots and Ambient Intelligence, 2013, 3 pages. [cited by applicant]
Ankur Handa et al., U.S. Appl. No. 16/863,111, filed Apr. 30, 2020. [cited by applicant]
Notice of Decision to Grant for Chinese Application No. 202010973239.2, mailed Aug. 22, 2024, 6 Pages. [cited by applicant]