IP Library › Granted Patent US 12,427,672
Granted Patent B2
US 12,427,672 · App. 17/900,843 · Granted Sep 30, 2025

Robotic manipulator with visual guidance and tactile sensing

Inventors: Abdulla Amer Hasan Ayyad (Abu Dhabi, AE); Rajkumar Muthusamy (Abu Dhabi, AE); Mohamad Abdul Mouti Halwani (Abu Dhabi, AE); Yahya Hashem Zweiri (Abu Dhabi, AE); Seneviratne Mudigansalage Seneviratne (Abu Dhabi, AE); Andre Dewald Swart (Abu Dhabi, AE)
Assignee: Khalifa University of Science and Technology and The Aerospace Holding Company LLC
B25J9/1697B25J13/082B25J13/084B25J19/023
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,427,672
App. No.
17/900,843
Granted
Sep 30, 2025
Kind
B2
Abstract

A robotic manipulator includes one or multiple end effectors that can engage with an object, and one or multiple cameras that simultaneously observe each end effector, and the surrounding environment. For example, an end effector can include a contact surface including tactile markers which can deform when the end effector contacts the object.

Claims (40)

1. A robotic manipulator, comprising:

a first end effector comprising a contact surface configured to make physical contact with an object, wherein the contact surface has an exterior surface which makes contact with the object and an interior surface which does not directly contact the object; and

an event camera that is couplable with the robotic manipulator and configured to receive an interior optical signal associated with a visually-detectable deformation of visual markers distributed on the interior surface of the contact surface in response to making contact with the object, wherein the event camera is configured to receive an exterior optical signal of an exterior environment at least partially surrounding the first end effector, and the interior optical signal is reflected within a first channel of the first end effector to the event camera and the exterior optical signal is reflected within a second channel of a second end effector to the event camera.

2. The robotic manipulator of claim 1 , wherein the contact surface comprises a flexible or semi-flexible material and is further configured to deform in response to contacting the object.

3. The robotic manipulator of claim 1 , wherein the first end effector further comprises a mirror positioned in the first channel to direct the interior optical signal associated with the object from the contact surface to the event camera.

4. The robotic manipulator of claim 1 , wherein the first end effector further comprises a light source positioned to direct light to the contact surface and illuminate the visual markers.

5. The robotic manipulator of claim 1 , further comprising:

a machining tool coupled with the robotic manipulator; and

the second end effector.

6. The robotic manipulator of claim 5 , wherein at least one of:

the second end effector is configured to engage with the object;

the second end effector comprises a lens configured to focus light from the exterior environment and a mirror configured to direct the light from the exterior environment to the event camera;

the event camera is a first camera and is aligned with the first end effector and the robotic manipulator further comprises a second camera aligned with the second end effector; or

the robotic manipulator further comprises a third end effector, and wherein the second and third end effectors comprise respective second and third contact surfaces configured to engage with the object.

7. A robotic manipulator, comprising:

a machining tool;

a first end effector comprising a contact surface configured to engage with an object, wherein the contact surface has an exterior surface which makes contact with the object and an interior surface which does not directly contact the object;

an event camera coupled with the robotic manipulator and configured to receive an interior optical signal associated with a visually-detectable deformation of visual markers distributed on the interior surface of the contact surface in response to making contact with the object, wherein the event camera is configured to receive an optical signal of an environment at least partially surrounding the first end effector; and

a second end effector positioned on an opposing side of the machining tool from the first end effector and comprising a lens configured to focus exterior light and a mirror positioned to direct the focused exterior light to the event camera, wherein the interior optical signal is reflected within a first channel of the first end effector to the event camera and an exterior optical signal is reflected within a second channel of the second end effector to the event camera.

8. The robotic manipulator of claim 7 , wherein the first end effector comprises a light source positioned to emit light to the contact surface and illuminate the visual markers.

9. The robotic manipulator of claim 7 , wherein the event camera comprises a plurality of photodiodes that are configured to asynchronously output data values corresponding to light intensity values of the photodiodes.

10. A method, comprising:

engaging a contact surface of a first end effector of a robotic manipulator to an object, wherein the contact surface has an exterior surface which makes contact with the object and an interior surface which does not directly contact the object; and

receiving, from an event camera that is coupled with the robotic manipulator, an interior optical signal associated with visually-detectable deformation of visual markers distributed on the interior surface of the contact surface in response to making contact with the object, wherein the event camera receives an exterior optical signal of an environment at least partially surrounding the first end effector, and the interior optical signal is reflected within a first channel of the first end effector to the event camera and the exterior optical signal is reflected within a second channel of a second end effector to the event camera; and

in response to receiving the interior optical signal or the exterior optical signal, controlling the first end effector to i) align the contact surface, ii) change a force of the contact surface, or iii) both.

11. The method of claim 10 , further comprising:

deforming a flexible or semi-flexible material of the contact surface in response to contacting the object.

12. The method of claim 10 , further comprising directing the interior optical signal associated with the object from the contact surface to the event camera.

13. The method of claim 12 , further comprising determining an amount of deformation of the contact surface based on a variation of the visual markers on the contact surface as represented in the interior optical signal.

14. The method of claim 13 , further comprising:

controlling a light source to direct light to the contact surface; and

illuminating the visual markers.

15. The method of claim 14 , further comprising:

coupling a machining tool with the robotic manipulator; and

coupling the second end effector with the robotic manipulator.

16. The robotic manipulator of claim 1 , wherein the first channel and the second channel are parallel.

17. The robotic manipulator of claim 1 , wherein the first channel and the second channel are different lengths.

18. The robotic manipulator of claim 17 , wherein the second channel includes at least one lens at a distal end of the second end effector.

19. The robotic manipulator of claim 1 , wherein the contact surface contours in a convex shape away from a distal end of the first end effector.

20. The robotic manipulator of claim 1 , wherein the first channel and the second channel connect at a region containing one or more lenses, wherein the one or more lenses are configured to receive the interior optical signal and the exterior optical signal.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 22, 2022
From: AYYAD, ABDULLA; MUTHUSAMY, RAJKUMAR; HALWANI, MOHAMAD ABDUL MOUTI; ZWEIRI, YAHYA HASHEM; SENEVIRATNE, SENEVIRATNE MUDIGANSALAGE; SWART, ANDRE DEWALD
To: KHALIFA UNIVERSITY OF SCIENCE AND TECHNOLOGY; THE AEROSPACE HOLDING COMPANY LLC
Reel/Frame 061185/0023 →
Continuity (2)
Provisional Application 63240285 · Sep 2, 2021
Related Publication 20230073681A1 · Mar 9, 2023
References Cited (77)
US 20160107316A1 · Alt · 2016 [cited by examiner]
US 20170328795A1 · Alt · 2017 [cited by examiner]
US 20180302562A1 · Newcombe · 2018 [cited by examiner]
US 20210023714A1 · Zhang et al. · 2021 [cited by applicant]
US 20210023715A1 · Zhang · 2021 [cited by examiner]
US 20210107166A1 · Yerazunis · 2021 [cited by examiner]
US 20210187735A1 · Homberg · 2021 [cited by examiner]
US 20230018498A1 · Kaehler · 2023 [cited by applicant]
US 20230294306A1 · Nicholas · 2023 [cited by examiner]
US 20230330859A1 · Tee · 2023 [cited by examiner]
CA 3214123A1 · 2022 [cited by applicant]
CN 112809679A · 2021 [cited by examiner]
CN 116038666A · 2023 [cited by applicant]
GB 2134071A · 1984 [cited by applicant]
GB 2605423A · 2022 [cited by examiner]
WO WO2022025893A1 · 2022 [cited by examiner]
WO WO2022207853A1 · 2022 [cited by examiner]
U.S. Appl. No. 17/900,770 , “Non-Final Office Action”, May 28, 2024, 9 pages. [cited by applicant]
“Application Manual”, ABB, 2018, 292 pages. [cited by applicant]
“Force Sensors”, Kistler, 2019, 42 pages. [cited by applicant]
“Force/Torque Sensors”, JR3 Inc., Accessed from Internet on Nov. 7, 2020, 1 page. [cited by applicant]
“Multi-Axis Force/Torque Sensor”, ATI Industrial Automation, 2014, 44 pages. [cited by applicant]
“Neuromorphic Sensing and Computing”, Yole Dveloppement, Available Online at https://yole-i-micronews-com.osu.eu-west-2.outscale.com/uploads/2019/ 09/YD19039 Neuromorphic Sensing Computing 2019 sample, 2019, 44 pages. [cited by applicant]
“Prophesee and Sony Develop a Stacked Event-Based Vision Sensor with the Industry's Smallest Pixels and Highest HDR”, Available Online at: https://www.prophesee.ai/2020/02/19/prophesee-sony-stacked-event-based-vision-se… [cited by applicant]
Abad et al., “Visuotactile Sensors With Emphasis on Gelsight Sensor: A Review”, IEEE Sensors Journal, vol. 20, No. 14, Mar. 2020, pp. 7628-7638. [cited by applicant]
Aceto et al., “A Survey on Information and Communication Technologies for Industry 4.0: State-of-the-art, Taxonomies, Perspectives, and Challenges”, IEEE Communications Surveys & Tutorials, vol. 21, No. 4, Aug. 2019, pp… [cited by applicant]
Al Khawli et al., “Introducing Data Analytics to the Robotic Drilling Process”, Industrial Robot, vol. 45, No. 3, Jun. 2018, pp. 371-378. [cited by applicant]
Alzarok et al., “3D Visual Tracking of an Articulated Robot in Precision Automated Tasks”, Sensors, vol. 17, No. 1, Jan. 2017, pp. 1-23. [cited by applicant]
Assaf et al., “Seeing by Touch: Evaluation of a Soft Biologically-Inspired Artificial Fingertip in Real-time Active Touch”, Sensors, vol. 14, No. 2, Feb. 2014, pp. 2561-2577. [cited by applicant]
Caggiano et al., “Study on Thrust Force and Torque Sensor Signals in Drilling of AI/CFRP Stacks for Aeronautical Applications”, Procedia CIRP, vol. 79, Jan. 2019, pp. 337-342. [cited by applicant]
Cantwell et al., “The Impact Resistance of Composite Materials—A Review”, Composites, vol. 22, No. 5, Sep. 1991, pp. 347-362. [cited by applicant]
Chen et al., “Robotic Grinding of a Blisk With Two Degrees of Freedom Contact Force Control”, The International Journal of Advanced Manufacturing, 2019, pp. 461-474. [cited by applicant]
Chen et al., “Tactile Sensors for Friction Estimation and Incipient Slip Detection Toward Dexterous Robotic Manipulation: A Review”, IEEE Sensors Journal, vol. 18, No. 22, Nov. 2018, pp. 9049-9064. [cited by applicant]
Chorley et al., “Development of a Tactile Sensor Based on Biologically Inspired Edge Encoding”, 2009 International Conference on Advanced Robotics, Jun. 2009, pp. 1-6. [cited by applicant]
De La Puente et al., “Grasping Objects From the Floor in Assistive Robotics: Real World Implications and Lessons Learned”, IEEE Access, vol. 7, Aug. 2019, pp. 123725-123735. [cited by applicant]
Devlieg et al., “High-Accuracy Robotic Drilling/Milling of 737 Inboard Flaps”, SAE International Journal of Aerospace, vol. 4, No. 2, 2011, pp. 1373-1379. [cited by applicant]
Ding et al., “Research and Application on Force Control of Industrial Robot Polishing Concave Curved Surfaces”, Proceedings of the Institution of Mechanical Engineers Part B Journal of Engineering Manufacture, vol. 233,… [cited by applicant]
Domroes et al., “Application and Analysis of Force Control Strategies to Deburring and Grinding”, Modern Mechanical Engineering, vol. 3, No. 2, Jun. 2013, pp. 11-18. [cited by applicant]
Frommknecht et al., “Multi-Sensor Measurement System for Robotic Drilling”, Robotics and Computer-Integrated Manufacturing, vol. 47, Dec. 2015, pp. 1-8. [cited by applicant]
Hristu et al., “The Performance of a Deformable-Membrane Tactile Sensor: Basic Results on Geometrically Defined Tasks”, Proceedings—IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No… [cited by applicant]
Hughes et al., “Soft Manipulators and Grippers: A Review”, Frontiers in Robotics and AI, vol. 3, Nov. 2016, pp. 1-12. [cited by applicant]
Indiveri et al., “Neuromorphic Vision Sensors”, Science, 288, No. 5469, May 2000, pp. 1189-1190. [cited by applicant]
Ito et al., “Shape Sensing by Vision Based Tactile Sensor for Dexterous Handling of Robot Hands”, 2010 IEEE International Conference on Automation Science and Engineering, Aug. 21-24, 2010, pp. 574-579. [cited by applicant]
Ji et al., “Industrial Robotic Machining: A Review”, The International Journal of Advanced Manufacturing Technology, vol. 103, No. 1-4, Apr. 2019, pp. 1239-1255. [cited by applicant]
Johnson et al., “Retrographic Sensing for the Measurement of Surface Texture and Shape”, 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2009, pp. 1070-1077. [cited by applicant]
Jung , “Biohybrid Systems: Nerves, Interfaces and Machines”, John Wiley & Sons, Oct. 19, 2011, 223. [cited by applicant]
Karim et al., “Challenges and Obstacles in Robot-Machining”, 2013 44th International Symposium on Robotic, Oct. 2013, 4 pages. [cited by applicant]
Kumagai et al., “Event-Based Tactile Image Sensor for Detecting Spatio-Temporal Fast Phenomena in Contacts”, 2019 IEEE World Haptics Conference, WHC, Jul. 2019, pp. 343-348. [cited by applicant]
Li et al., “Localization and Manipulation of Small Parts Using Gelsight Tactile Sensing”, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Sep. 14-18, 2014, pp. 3988-3993. [cited by applicant]
Lichtsteiner et al., “A 128×128 120 dB 15 US Latency Asynchronous Temporal Contrast Vision Sensor”, IEEE Journal of Solid-state Circuits, IEEE Service Center, vol. 43, No. 2, Feb. 1, 2008, pp. 566-576. [cited by applicant]
Liu et al., “Neuromorphic Sensory Systems”, Current Opinion in Neurobiology, vol. 20, No. 3, Jun. 2010, pp. 288-295. [cited by applicant]
Muthusamy et al., “Neuromorphic Event-Based Slip Detection and Suppression in Robotic Grasping and Manipulation”, IEEE Access, vol. 8, Aug. 2020, pp. 153364-153384. [cited by applicant]
Muthusamy et al., “Neuromorphic Eye-in-Hand Visual Servoing”, IEEE Access, vol. 9, Apr. 2021, pp. 55853-55870. [cited by applicant]
Naeini et al., “A Novel Dynamic-Vision-Based Approach for Tactile Sensing Applications”, IEEE Transactions on Instrumentation and Measurement, vol. 69, No. 5, May 2020, pp. 1881-1893. [cited by applicant]
Nakagawa-Silva et al., “A Bio-Inspired Slip Detection and Reflex-like Suppression Method for Robotic Manipulators”, IEEE Sensors Journal, vol. 19, No. 24, Dec. 2019, pp. pp. 12443-12453. [cited by applicant]
Olsson et al., “Cost-Efficient Drilling Using Industrial Robots With High-bandwidth Force Feedback”, Robotics and Computer-Integrated Manufacturing, vol. 26, No. 1, Feb. 2010, pp. 24-38. [cited by applicant]
Rigi et al., “A Novel Event-Based Incipient Slip Detection Using Dynamic Active-Pixel Vision Sensor (DAVIS)”, Sensors, vol. 18, No. 2, Jan. 2018, pp. 1-17. [cited by applicant]
Romano et al., “Human-Inspired Robotic Grasp Control With Tactile Sensing”, IEEE Transactions on Robotics, vol. 27, No. 6, Dec. 2011, pp. 1067-1079. [cited by applicant]
Rosa et al., “Analysis and Implementation of a Force Control Strategy for Drilling Operations With an Industrial Robot”, Journal of the Brazilian Society of Mechanical Sciences and Engineering, vol. 39, No. 11, Sep. 201… [cited by applicant]
She et al., “Exoskeleton-covered Soft Finger With Vision-Based Proprioception and Tactile Sensing”, 2020 IEEE International Conference on Robotics and Automation (ICRA), May 31-Aug. 31, 2020, pp. 10075-10081. [cited by applicant]
Sun et al., “Design and Performance Analysis of an Industrial Robot Arm for Robotic Drilling Process”, he Industrial Robot; Bedford, vol. 46, No. 1, Apr. 12, 2019, pp. 7-16. [cited by applicant]
Tiwana et al., “A Review of Tactile Sensing Technologies With Applications in Biomedical Engineering”, Sensors and Actuators A: Physical, vol. 179, Jun. 2012, pp. 17-31. [cited by applicant]
Trueb et al., “Towards Vision-based Robotic Skins: A Data-Driven, Multi-Camera Tactile Sensor”, 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), Oct. 2019, 6 pages. [cited by applicant]
Vallbo et al., “Properties of Cutaneous Mechanoreceptors in the Human Hand Related to Touch Sensation”, Human Neurobiology, vol. 3, No. 1, Feb. 1984, pp. 3-14. [cited by applicant]
Vanarse et al., “A Review of Current Neuromorphic Approaches for Vision, Auditory, and Olfactory Sensors”, Frontiers in neuroscience, vol. 10, No. 115, Mar. 29, 2016, pp. 1-6. [cited by applicant]
Wang et al., “Real-Time Soft Body 3d Proprioception via Deep Vision-Based Sensing”, IEEE Robotics and Automation Letters, vol. 5, No. 2, Apr. 2020, pp. 3382-3389. [cited by applicant]
Ward-Cherrier et al., “A Miniaturised Neuromorphic Tactile Sensor Integrated With an Anthropomorphic Robot Hand”, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct. 2020, pp. 9883-9889. [cited by applicant]
Ward-Cherrier et al., “NeuroTac: A Neuromorphic Optical Tactile Sensor applied to Texture Recognition” 2020 IEEE International Conference on Robotics and Automation (ICRA), May 2020, pp. 2654-2660. [cited by applicant]
Ward-Cherrier et al., “The Tactip Family: Soft Optical Tactile Sensors With 3d-Printed Biomimetic Morphologies”, Soft Robotics, vol. 5, No. 2, 2018, pp. 216-227. [cited by applicant]
Xie et al., “Design of Robotic End-effector for Milling Force Control”, IOP Conference Series: Materials Science and Engineering, vol. 423, May 2018, pp. 1-6. [cited by applicant]
Xu et al., “A Visual Seam Tracking System for Robotic Arc Welding”, The International Journal of Advanced Manufacturing Technology, vol. 37, No. 1, Apr. 2008, pp. 70-75. [cited by applicant]
Yamaguchi et al., “Combining Finger Vision and Optical Tactile Sensing: Reducing and Handling Errors While Cutting Vegetables”, 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), Nov. 15-17, 201… [cited by applicant]
Yamaguchi et al., “Grasp Adaptation Control With Finger Vision: Verification With Deformable and Fragile Objects”, 35th Annual Conference of the Robotics Society of Japan (RSJ2017), Sep. 2017, 3 pages. [cited by applicant]
Yuan et al., “GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force”, Sensors, vol. 17, No. 12, Nov. 2017, pp. 1-21. [cited by applicant]
Yuan et al., “Shape-Independent Hardness Estimation Using Deep Learning and a Gelsight Tactile Sensor”, 2017 IEEE International Conference on Robotics and Automation (ICRA), Apr. 2017, 8 pages. [cited by applicant]
Yussof et al., “Sensorization of Robotic Hand Using Optical Three-Axis Tactile Sensor: Evaluation With Grasping and Twisting Motions”, Journal of Computer Science, vol. 6, No. 8, Aug. 2010, pp. 955-962. [cited by applicant]
U.S. Appl. No. 17/900,770 , “Notice of Allowance”, Nov. 25, 2024, 7 pages. [cited by applicant]