IP Library › Granted Patent US 12,280,495
Granted Patent B2
US 12,280,495 · App. 17/900,770 · Granted Apr 22, 2025

Robotic manipulator with camera and tactile sensing

Inventors: Rajkumar Muthusamy (Abu Dhabi, AE); Huda Alyammahi (Abu Dhabi, AE); Muhammad Umer Hameed Shah (Abu Dhabi, AE); Omar Fuad Faris (Abu Dhabi, AE); Irfan Hussain (Abu Dhabi, AE); Yahya Hashem Zweiri (Abu Dhabi, AE); Dongming Gan (Abu Dhabi, AE); Seneviratne Mudigansalage Seneviratne (Abu Dhabi, AE)
Assignee: Khalifa University of Science and Technology
B25J19/023B25J13/084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,280,495
App. No.
17/900,770
Granted
Apr 22, 2025
Kind
B2
Abstract

A robotic manipulator is described. The robotic manipulator includes a rigid or semi-rigid end effector that engages with objects and a sensor that detects data associated with the object. For example, the sensor can include an optical sensor or a vision-based tactile sensor that detects data associated with the object.

Claims (25)

1. A robotic manipulator, comprising:

an end effector configured to engage with an object;

sensors positioned on the robotic manipulator, each of the sensors configured to detect data associated with the object; and

an actuator coupled with the end effector and configured to move a portion of the end effector relative to the robotic manipulator;

wherein the sensors comprise a tactile sensor configured to engage with the object; and

wherein the tactile sensor comprises one or more transparent rigid layers, configured to engage with the object and allow light to travel through the one or more layers.

2. The robotic manipulator of claim 1 , wherein the sensors comprise an optical sensing system comprising an optical sensor and optic mirrors positioned to direct light to the optical sensor.

3. The robotic manipulator of claim 2 , wherein at least one optic mirror is positioned on the moving portion of the end effector.

4. The robotic manipulator of claim 3 , wherein the actuator is further configured to move the portion of the end effector between a closed position where the end effector is engaged with the object and an open position where the end effector is positioned adjacent to the object, wherein when the end effector is in the closed position, the at least one optic mirror is aligned to direct light reflected by the object to the optical sensor.

5. The robotic manipulator of claim 1 , wherein the sensors comprise a neuromorphic vision-based sensor or an event-based camera.

6. The robotic manipulator of claim 1 , wherein the one or more transparent rigid layers comprise an engagement surface configured to deform in response to engagement with the object.

7. The robotic manipulator of claim 1 , wherein the sensors further comprise an optical sensing system comprising an optical sensor and one or more optic mirrors configured to direct light traveling through the one or more layers to the optical sensor.

8. The robotic manipulator of claim 1 , wherein the sensors comprise an optical sensor that is configured to detect event based data associated with the object.

9. The robotic manipulator of claim 8 , wherein the end effector is a first end effector and the robotic manipulator further comprises a second end effector attached to the robotic manipulator in a fixed position and comprising an engagement area configured to engage with the object.

10. The robotic manipulator of claim 1 , wherein the sensors comprise an optical sensor positioned to detect data associated with the object and a tactile sensor configured to engage with the object.

11. The robotic manipulator of claim 1 , wherein the end effector comprises a compliant, rigid, or semi-rigid structure.

12. The robotic manipulator of claim 1 , wherein the end effector is a first end effector and the robotic manipulator further comprises a second end effector coupled with the robotic manipulator.

13. The robotic manipulator of claim 12 , wherein the first and second end effectors have respective first and second engagement surfaces, the first and second engagement surfaces being configured to engage with the object.

14. The robotic manipulator of claim 13 , wherein the first and second engagement surfaces comprise a marker-based occluded surface, a marker-less opaque surface, or a transparent soft surface.

15. A robotic manipulator, comprising:

an end effector configured to engage with an object, the end effector having a compliant, rigid, or semi-rigid structure and comprising an engagement surface configured to engage with an object, the engagement surface comprising a flexible material; and

an optical sensor positioned to detect data associated with the object through the engagement surface.

16. The robotic manipulator of claim 15 , wherein the optical sensor is configured to detect vision or tactile information using an event-based optical sensor.

17. The robotic manipulator of claim 15 , further comprising a light source coupled with end effector and configured to illuminate the object.

18. The robotic manipulator of claim 15 , wherein the optical sensor comprises an event-based camera configured to detect illumination changes associated with the object.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 22, 2022
From: MUTHUSAMY, RAJKUMAR; ALYAMMAHI, HUDA; SHAH, MUHAMMAD UMER HAMEED; FARIS, OMAR FUAD; HUSSAIN, IRFAN; ZWEIRI, YAHYA HASHEM; GAN, DONGMING; SENEVIRATNE, SENEVIRATNE MUDIGANSALAGE
To: KHALIFA UNIVERSITY OF SCIENCE AND TECHNOLOGY
Reel/Frame 061185/0478 →
Continuity (2)
Provisional Application 63241212 · Sep 7, 2021
Related Publication 20230072207A1 · Mar 9, 2023
References Cited (76)
US 20160107316A1 · Alt et al. · 2016 [cited by applicant]
US 20170328795A1 · Alt et al. · 2017 [cited by applicant]
US 20180302562A1 · Newcombe et al. · 2018 [cited by applicant]
US 20210023714A1 · Zhang · 2021 [cited by examiner]
US 20210023715A1 · Zhang et al. · 2021 [cited by applicant]
US 20210107166A1 · Yerazunis et al. · 2021 [cited by applicant]
US 20210187735A1 · Homberg et al. · 2021 [cited by applicant]
US 20230018498A1 · Kaehler · 2023 [cited by examiner]
US 20230294306A1 · Nicholas · 2023 [cited by applicant]
US 20230330859A1 · Tee et al. · 2023 [cited by applicant]
CA 3214123A1 · 2022 [cited by examiner]
CN 112809679A · 2021 [cited by applicant]
CN 116038666A · 2023 [cited by examiner]
GB 2134071A · 1984 [cited by examiner]
GB 2605423A · 2022 [cited by applicant]
WO 2022025893A1 · 2022 [cited by applicant]
WO 2022207853A1 · 2022 [cited by applicant]
“Application Manual”, ABB, 2018, 292 pages. [cited by applicant]
“Force Sensors”, Kistler, 2019, 42 pages. [cited by applicant]
“Force/Torque Sensors”, JR3 Inc., Accessed from Internet on Nov. 7, 2020, 1 page. [cited by applicant]
“Multi-Axis Force/Torque Sensor”, ATI Industrial Automation, 2014, 44 pages. [cited by applicant]
“Neuromorphic Sensing and Computing”, Yole Dveloppement, Available Online at https://yole-i-micronews-com.osu.eu-west-2.outscale.com/uploads/2019/ 09/YD19039 Neuromorphic Sensing Computing 2019 sample, 2019, 44 pages. [cited by applicant]
“Prophesee and Sony Develop a Stacked Event-Based Vision Sensor with the Industry's Smallest Pixels and Highest HDR”, Available Online at: https://www.prophesee.ai/2020/02/19/prophesee-sony-stacked-event-based-vision-se… [cited by applicant]
Abad et al., “Visuotactile Sensors With Emphasis on Gelsight Sensor: A Review”, IEEE Sensors Journal, vol. 20, No. 14, Mar. 2020, pp. 7628-7638. [cited by applicant]
Aceto et al., “A Survey on Information and Communication Technologies for Industry 4.0: State-of-the-art, Taxonomies, Perspectives, and Challenges”, IEEE Communications Surveys & Tutorials, vol. 21, No. 4, Aug. 2019, pp… [cited by applicant]
Al Khawli et al., “Introducing Data Analytics to the Robotic Drilling Process”, Industrial Robot, vol. 45, No. 3, Jun. 2018, pp. 371-378. [cited by applicant]
Alzarok et al., “3D Visual Tracking of an Articulated Robot in Precision Automated Tasks”, Sensors, vol. 17, No. 1, Jan. 2017, pp. 1-23. [cited by applicant]
Assaf et al., “Seeing by Touch: Evaluation of a Soft Biologically-Inspired Artificial Fingertip in Real-time Active Touch”, Sensors, vol. 14, No. 2, Feb. 2014, pp. 2561-2577. [cited by applicant]
Caggiano et al., “Study on Thrust Force and Torque Sensor Signals in Drilling of AI/CFRP Stacks for Aeronautical Applications”, Procedia CIRP, vol. 79, Jan. 2019, pp. 337-342. [cited by applicant]
Cantwell et al., “The Impact Resistance of Composite Materials—A Review”, Composites, vol. 22, No. 5, Sep. 1991, pp. 347-362. [cited by applicant]
Chen et al., “Robotic Grinding of a Blisk With Two Degrees of Freedom Contact Force Control”, The International Journal of Advanced Manufacturing, 2019, pp. 461-474. [cited by applicant]
Chen et al., “Tactile Sensors for Friction Estimation and Incipient Slip Detection Toward Dexterous Robotic Manipulation: A Review”, IEEE Sensors Journal, vol. 18, No. 22, Nov. 2018, pp. 9049-9064. [cited by applicant]
Chorley et al., “Development of a Tactile Sensor Based on Biologically Inspired Edge Encoding”, 2009 International Conference on Advanced Robotics, Jun. 2009, pp. 1-6. [cited by applicant]
De La Puente et al., “Grasping Objects From the Floor in Assistive Robotics: Real World Implications and Lessons Learned”, IEEE Access, vol. 7, Aug. 2019, pp. 123725-123735. [cited by applicant]
Devlieg et al., “High-Accuracy Robotic Drilling/Milling of 737 Inboard Flaps”, SAE International Journal of Aerospace, vol. 4, No. 2, 2011, pp. 1373-1379. [cited by applicant]
Ding et al., “Research and Application on Force Control of Industrial Robot Polishing Concave Curved Surfaces”, Proceedings of the Institution of Mechanical Engineers Part B Journal of Engineering Manufacture, vol. 233,… [cited by applicant]
Domroes et al., “Application and Analysis of Force Control Strategies to Deburring and Grinding”, Modern Mechanical Engineering, vol. 3, No. 2, Jun. 2013, pp. 11-18. [cited by applicant]
Frommknecht et al., “Multi-Sensor Measurement System for Robotic Drilling”, Robotics and Computer-Integrated Manufacturing, vol. 47, Dec. 2015, pp. 1-8. [cited by applicant]
Hristu et al., “The Performance of a Deformable-Membrane Tactile Sensor: Basic Results on Geometrically Defined Tasks”, Proceedings—IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No… [cited by applicant]
Hughes et al., “Soft Manipulators and Grippers: A Review”, Frontiers in Robotics and AI, vol. 3, Nov. 2016, pp. 1-12. [cited by applicant]
Indiveri et al., “Neuromorphic Vision Sensors”, Science, 288, No. 5469, May 2000, pp. 1189-1190. [cited by applicant]
Ito et al., “Shape Sensing by Vision Based Tactile Sensor for Dexterous Handling of Robot Hands”, 2010 IEEE International Conference on Automation Science and Engineering, Aug. 21-24, 2010, pp. 574-579. [cited by applicant]
Ji et al., “Industrial Robotic Machining: A Review”, The International Journal of Advanced Manufacturing Technology, vol. 103, No. 1-4, Apr. 2019, pp. 1239-1255. [cited by applicant]
Johnson et al., “Retrographic Sensing for the Measurement of Surface Texture and Shape”, 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2009, pp. 1070-1077. [cited by applicant]
Jung , “Biohybrid Systems: Nerves, Interfaces and Machines”, John Wiley & Sons, Oct. 19, 2011, 223. [cited by applicant]
Karim et al., “Challenges and Obstacles in Robot-Machining”, 2013 44th International Symposium on Robotic, Oct. 2013, 4 pages. [cited by applicant]
Kumagai et al., “Event-Based Tactile Image Sensor for Detecting Spatio-Temporal Fast Phenomena in Contacts”, 2019 IEEE World Haptics Conference, WHC, Jul. 2019, pp. 343-348. [cited by applicant]
Li et al., “Localization and Manipulation of Small Parts Using Gelsight Tactile Sensing”, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Sep. 14-18, 2014, pp. 3988-3993. [cited by applicant]
Lichtsteiner et al., “A 128×128 120 dB 15 μS Latency Asynchronous Temporal Contrast Vision Sensor”, IEEE Journal of Solid-state Circuits, IEEE Service Center, vol. 43, No. 2, Feb. 1, 2008, pp. 566-576. [cited by applicant]
Liu et al., “Neuromorphic Sensory Systems”, Current Opinion in Neurobiology, vol. 20, No. 3, Jun. 2010, pp. 288-295. [cited by applicant]
Muthusamy et al., “Neuromorphic Event-Based Slip Detection and Suppression in Robotic Grasping and Manipulation”, IEEE Access, vol. 8, Aug. 2020, pp. 153364-153384. [cited by applicant]
Muthusamy et al., “Neuromorphic Eye-in-Hand Visual Servoing”, IEEE Access, vol. 9, Apr. 2021, pp. 55853-55870. [cited by applicant]
Naeini et al., “A Novel Dynamic-Vision-Based Approach for Tactile Sensing Applications”, IEEE Transactions on Instrumentation and Measurement, vol. 69, No. 5, May 2020, pp. 1881-1893. [cited by applicant]
Nakagawa-Silva et al., “A Bio-Inspired Slip Detection and Reflex-like Suppression Method for Robotic Manipulators”, IEEE Sensors Journal, vol. 19, No. 24, Dec. 2019, pp. pp. 12443-12453. [cited by applicant]
Olsson et al., “Cost-Efficient Drilling Using Industrial Robots With High-bandwidth Force Feedback”, Robotics and Computer-Integrated Manufacturing, vol. 26, No. 1, Feb. 2010, pp. 24-38. [cited by applicant]
Rigi et al., “A Novel Event-Based Incipient Slip Detection Using Dynamic Active-Pixel Vision Sensor (DAVIS)”, Sensors, vol. 18, No. 2, Jan. 2018, pp. 1-17. [cited by applicant]
Romano et al., “Human-Inspired Robotic Grasp Control With Tactile Sensing”, IEEE Transactions on Robotics, vol. 27, No. 6, Dec. 2011, pp. 1067-1079. [cited by applicant]
Rosa et al., “Analysis and Implementation of a Force Control Strategy for Drilling Operations With an Industrial Robot”, Journal of the Brazilian Society of Mechanical Sciences and Engineering, vol. 39, No. 11, Sep. 201… [cited by applicant]
She et al., “Exoskeleton-covered Soft Finger With Vision-Based Proprioception and Tactile Sensing”, 2020 IEEE International Conference on Robotics and Automation (ICRA), May 31-Aug. 31, 2020, pp. 10075-10081. [cited by applicant]
Sun et al., “Design and Performance Analysis of an Industrial Robot Arm for Robotic Drilling Process”, he Industrial Robot; Bedford, vol. 46, No. 1, Apr. 12, 2019, pp. 7-16. [cited by applicant]
Tiwana et al., “A Review of Tactile Sensing Technologies With Applications in Biomedical Engineering”, Sensors and Actuators A: Physical, vol. 179, Jun. 2012, pp. 17-31. [cited by applicant]
Trueb et al., “Towards Vision-based Robotic Skins: A Data-Driven, Multi-Camera Tactile Sensor”, 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), Oct. 2019, 6 pages. [cited by applicant]
Vallbo et al., “Properties of Cutaneous Mechanoreceptors in the Human Hand Related to Touch Sensation”, Human Neurobiology, vol. 3, No. 1, Feb. 1984, pp. 3-14. [cited by applicant]
Vanarse et al., “A Review of Current Neuromorphic Approaches for Vision, Auditory, and Olfactory Sensors”, Frontiers in neuroscience, vol. 10, No. 115, Mar. 29, 2016, pp. 1-6. [cited by applicant]
Wang et al., “Real-Time Soft Body 3d Proprioception via Deep Vision-Based Sensing”, IEEE Robotics and Automation Letters, vol. 5, No. 2, Apr. 2020, pp. 3382-3389. [cited by applicant]
Ward-Cherrier et al., “A Miniaturised Neuromorphic Tactile Sensor Integrated With an Anthropomorphic Robot Hand”, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct. 2020, pp. 9883-9889. [cited by applicant]
Ward-Cherrier et al., “NeuroTac: A Neuromorphic Optical Tactile Sensor applied to Texture Recognition”, 2020 IEEE International Conference on Robotics and Automation (ICRA), May 2020, pp. 2654-2660. [cited by applicant]
Ward-Cherrier et al., “The Tactip Family: Soft Optical Tactile Sensors With 3d-Printed Biomimetic Morphologies”, Soft Robotics, vol. 5, No. 2, 2018, pp. 216-227. [cited by applicant]
Xie et al., “Design of Robotic End-effector for Milling Force Control”, IOP Conference Series: Materials Science and Engineering, vol. 423, May 2018, pp. 1-6. [cited by applicant]
Xu et al., “A Visual Seam Tracking System for Robotic Arc Welding”, The International Journal of Advanced Manufacturing Technology, vol. 37, No. 1, Apr. 2008, pp. 70-75. [cited by applicant]
Yamaguchi et al., “Combining Finger Vision and Optical Tactile Sensing: Reducing and Handling Errors While Cutting Vegetables”, 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), Nov. 15-17, 201… [cited by applicant]
Yamaguchi et al., “Grasp Adaptation Control With Finger Vision: Verification With Deformable and Fragile Objects”, 35th Annual Conference of the Robotics Society of Japan (RSJ2017), Sep. 2017, 3 pages. [cited by applicant]
Yuan et al., “GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force”, Sensors, vol. 17, No. 12, Nov. 2017, pp. 1-21. [cited by applicant]
Yuan et al., “Shape-Independent Hardness Estimation Using Deep Learning and a Gelsight Tactile Sensor”, 2017 IEEE International Conference on Robotics and Automation (ICRA), Apr. 2017, 8 pages. [cited by applicant]
Yussof et al., “Sensorization of Robotic Hand Using Optical Three-Axis Tactile Sensor: Evaluation With Grasping and Twisting Motions”, Journal of Computer Science, vol. 6, No. 8, Aug. 2010, pp. 955-962. [cited by applicant]
U.S. Appl. No. 17/900,843, “Final Office Action”, Feb. 27, 2025, 26 pages. [cited by applicant]
Cited By (1)
US 12,397,420