IP Library › Granted Patent US 12,415,273
Granted Patent B2
US 12,415,273 · App. 17/689,534 · Granted Sep 16, 2025

Control of robotic arms through hybrid inverse kinematics and intuitive reference frame

Inventors: Lei Wu (Tampa, FL); Redwan Alqasemi (Wesley Chapel, FL); Rajiv Dubey (Tampa, FL)
Assignee: University of South Florida
B25J9/1664B25J3/00B25J9/023B25J9/06B25J9/1607B25J9/1689
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,415,273
App. No.
17/689,534
Granted
Sep 16, 2025
Kind
B2
Abstract

Disclosed is an intuitive robotic device control reference frame with hybrid inverse kinematics having a natural and easier human-robot interface. A system for controlling a robot can include a robotic device that performs tasks, an input device that receives user input indicating desired movement of the robotic device to perform the tasks, and a controller that controls the robotic device based on the user input. The controller can receive signals identifying the user input, determine a ground reference frame and an end-effector reference frame for controlling movement of the robotic device, generate, based on the ground reference frame and the end-effector reference frame, an intuitive reference frame for controlling movement of the robotic device, determine controls to move the robotic device in the intuitive reference frame, generate instructions for controlling movement of the robotic device, and execute the instructions to control movement of the robotic device.

Claims (79)

1. A system for controlling a robotic device, the system comprising:

a robotic device configured to perform one or more tasks;

an input device configured to receive user input indicating desired movement of the robotic device to perform the one or more tasks;

a ground reference frame determiner;

an end-effector reference frame determiner;

an intuitive reference frame determiner;

a processor; and

a controller configured to control the robotic device based on the user input indicating the desired movement of the robotic device, the controller configured to:

receive signals identifying the user input from the input device;

determine a ground reference frame using the ground reference frame determiner and the processor configured to track the real-time location of the ground reference frame for controlling movement of the robotic device based on user input and a base location of the robotic device relative to a ground surface;

determine an end-effector reference frame using the end-effector reference frame determiner and the processor configured to track the real-time location of the end-effector reference frame for controlling movement of the robotic device, wherein the end-effector reference frame is based on a current joint configuration of the robotic device;

generate, based on the ground reference frame and the end-effector reference frame, an intuitive reference frame for controlling movement of the robotic device;

assign an axis of the intuitive reference frame to an x axis of the end-effector reference frame;

assign a y axis of the intuitive reference frame based on a cross product of a z axis of the ground reference frame and the x axis of the intuitive reference frame;

determine the intuitive reference frame in real-time using the intuitive reference frame determiner which determines the movement of the robotic device based on user input, ground reference frame, and end-effector reference frame using hybrid inverse kinematics;

generate instructions to operate the robotic device according to the user input in the intuitive reference frame; and

execute the instructions to operate the robotic device according to the user input in the intuitive reference frame.

2. The system of claim 1 , wherein the controller is configured to:

receive user input indicating movement along the x axis of the end-effector reference frame; and

determine, based on the user input, corresponding controls to move the robotic device in a forward or a backward direction in the intuitive reference frame.

3. The system of claim 1 , wherein the controller is configured to:

receive user input indicating movement along an axis perpendicular to the x axis of the end-effector reference frame and parallel to a ground surface; and

determine, based on the user input, corresponding controls to move the robotic device in a left or a right direction in the intuitive reference frame.

4. The system of claim 1 , wherein the controller is configured to:

receive user input indicating movement along the z axis of the ground reference frame; and

determine, based on the user input, corresponding controls to move the robotic device in an up or a down direction in the intuitive reference frame.

5. The system of claim 1 , wherein the controller is configured to:

receive user input indicating movement around the x axis of the end-effector reference frame; and

determine, based on the user input, corresponding controls to move the robotic device in a roll direction in the intuitive reference frame.

6. The system of claim 1 , wherein the controller is configured to:

receive user input indicating movement around an axis perpendicular to the x axis of the end-effector reference frame and parallel to a ground surface; and

determine, based on the user input, corresponding controls to move the robotic device in a pitch direction in the intuitive reference frame.

7. The system of claim 1 , wherein the controller is configured to:

receive user input indicating movement around the z axis of the ground reference frame; and

determine, based on the user input, corresponding controls to move the robotic device in a yaw direction in the intuitive reference frame.

8. The system of claim 1 , wherein the robotic device is controlled in the end-effector reference frame or the ground reference frame.

9. The system of claim 1 , wherein determining the end-effector reference frame comprises transforming the ground reference frame based on the current joint configuration of the robotic device.

10. The system of claim 1 , wherein the end-effector reference frame changes with the current joint configuration of the robotic device.

11. The system of claim 1 , wherein generating the intuitive reference frame comprises assigning a z axis of the intuitive reference frame based on a cross product of the x axis of the intuitive reference frame and the y axis of the intuitive reference frame.

12. A method for controlling movement of a robotic device, the method comprising:

receiving user input using a controller and an input device wherein the user input indicates the desired movement of a robotic device;

determining a ground reference frame using a ground reference frame determiner and a processor which track the real-time location of the ground reference frame based on user input and a base location of the robotic device relative to a ground surface;

determining an end-effector reference frame using an end-effector determiner and a processor which track the real-time location of the end-effector reference, wherein the end-effector reference is determined by transforming the ground reference frame based on a current joint configuration of the robotic device;

automatically generating, in real-time, an intuitive reference frame for controlling movement of the robotic device using the controller based on the ground reference frame;

automatically generating, in real-time, an x axis of the intuitive reference frame to an axis of the end-effector reference frame using the controller;

automatically generating, in real-time, a y axis of the intuitive reference frame based on a cross product of a z axis of the ground reference frame and the x axis of the intuitive reference frame using the controller;

determining, by the controller, controls to move the robotic device in the intuitive reference frame;

generating, by the controller, instructions for controlling movement of the robotic device in the intuitive reference frame, end-effector reference frame, or the ground reference frame; and

executing, by the controller, the generated instructions to control movement of the robotic device in the intuitive reference frame, end-effector reference frame, or the ground reference frame.

13. The method of claim 12 , further comprising executing, by the controller, the instructions to control movement of the robotic device.

14. The method of claim 12 , further comprising:

receiving, by the controller and from the input device, user input indicating movement along the x axis of the end-effector reference frame; and

determining, by the controller and based on the user input, corresponding controls to move the robotic device in a forward or a backward direction in the intuitive reference frame.

15. The method of claim 12 , further comprising:

receiving, by the controller and from the input device, user input indicating movement along an axis perpendicular to the x axis of the end-effector reference frame and parallel to a ground surface; and

determining, by the controller and based on the user input, corresponding controls to move the robotic device in a left or a right direction in the intuitive reference frame.

16. The method of claim 12 , further comprising:

receiving, by the controller and from the input device, user input indicating movement along the z axis of the ground reference frame; and

determining, by the controller and based on the user input, corresponding controls to move the robotic device in an up or a down direction in the intuitive reference frame.

17. The method of claim 12 , further comprising:

receiving, by the controller and from the input device, user input indicating movement around the x axis of the end-effector reference frame; and

determining, by the controller and based on the user input, corresponding controls to move the robotic device in a roll direction in the intuitive reference frame.

18. The method of claim 12 , further comprising:

receiving, by the controller and from the input device, user input indicating movement around an axis perpendicular to the x axis of the end-effector reference frame and parallel to a ground surface; and

determining, by the controller and based on the user input, corresponding controls to move the robotic device in a pitch direction in the intuitive reference frame.

19. A system for controlling a robotic device, the system comprising:

a robotic device configured to perform one or more tasks;

an input device configured to receive user input indicating desired movement of the robotic device to perform the one or more tasks;

a ground reference frame determiner;

a processor; and

a controller configured to control the robotic device based on the user input indicating the desired movement of the robotic device, the controller configured to:

receive signals identifying the user input from the input device;

determine a ground reference using the ground reference frame determiner and the processor configured to track the real-time location of the ground reference frame for controlling movement of the robotic device based on user input and a base location of the robotic device relative to a ground surface;

determine an end-effector reference frame using the processor to track the real-time location of the end-effector reference, wherein the end-effector reference frame is determined by transforming the ground reference frame based on a current joint configuration of the robotic device;

generate, based on the ground reference frame and the end-effector reference frame, an intuitive reference frame for controlling movement of the robotic device;

wherein the intuitive reference frame comprises assigning a z axis of the intuitive reference frame based on a cross product of ax axis of the intuitive reference frame and a y axis of the intuitive reference frame;

generate instructions for controlling the robotic device in one or more of the ground reference frame, the end-effector reference frame, and the intuitive reference frame; and

execute the generated instructions to control movement of the robotic device according to one or more of the ground reference frame, the end-effector reference frame, and the intuitive reference frame.

20. The system of claim 19 , wherein generating the intuitive reference frame comprises assigning a y axis of the intuitive reference frame based on a cross product of a z axis of the ground reference frame and the x axis of the intuitive reference frame.

Assignments (2)
CONFIRMATORY LICENSE Recorded Jan 29, 2025
From: UNIVERSITY OF SOUTH FLORIDA
To: NATIONAL SCIENCE FOUNDATION
Reel/Frame 070042/0349 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 30, 2022
From: WU, LEI; ALQASEMI, REDWAN; DUBEY, RAJIV
To: UNIVERSITY OF SOUTH FLORIDA
Reel/Frame 060367/0350 →
Continuity (2)
Provisional Application 63160282 · Mar 12, 2021
Related Publication 20220288780A1 · Sep 15, 2022
References Cited (33)
US 7904202B2 · Hoppe · 2011 [cited by applicant]
US 8483882B2 · Abdallah et al. · 2013 [cited by applicant]
US 10617480B2 · Brisson · 2020 [cited by applicant]
US 20130325032A1 · Schena · 2013 [cited by examiner]
US 20170333142A1 · Itkowitz · 2017 [cited by examiner]
US 20190099183A1 · Leimbach et al. · 2019 [cited by applicant]
US 20200060777A1 · Nowlin et al. · 2020 [cited by applicant]
US 20200360097A1 · DiMaio · 2020 [cited by examiner]
US 20210394217A1 · Chevron · 2021 [cited by examiner]
US 20220226987A1 · Bacher · 2022 [cited by examiner]
US 20230363830A1 · Polchin · 2023 [cited by examiner]
Allard et al., “Transfer learning for semg hand gestures recognition using convolutional neural networks,” 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2017, pp. 1663-1668. [cited by applicant]
Alqasemi, “Maximizing manipulation capabilities of persons with disabilities using a smart 9-degree-of-freedom wheelchair mounted robotic arm system,” University of South Florida Graduate Theses and Dissertation, Mar. 2… [cited by applicant]
Baldi et al., “Design of a wearable interface for lightweight robotic arm for people with mobility impairments,” 2017 International Conference on Rehabilitation Robotics (ICORR), 2017, pp. 1567-1573. [cited by applicant]
Campeau-Lecours et al., “Intuitive adaptive orientation control for enhanced human-robot interaction,” IEEE Transactions on Robotics, 2019, vol. 35:509-520. [cited by applicant]
Chiu et al., “A wireless steady state visually evoked potential-based bci eating assistive system,” 2017 International Joint Conference on Neural Networks (IJCNN), 2017, 3003-3007. [cited by applicant]
Chung et al., “Performance evaluation of a mobile touchscreen interface for assistive robotic manipulators: A pilot study,” Topics in spinal cord injury rehabilitation, 2017, 23 2:131-139. [cited by applicant]
Du et al., “A markerless human-robot interface using particle filter and kalman filter for dual robots,” IEEE Transactions on Industrial Electronics, 2015, 62(4):2257-2264. [cited by applicant]
Fall et al., “Wireless semgbased body-machine interface for assistive technology devices,” IEEE Journal of Biomedical and Health Informatics, 2017, 21:967-977. [cited by applicant]
Featherstone, “Position and velocity transformations between robot end-effector coordinates and joint angles,” The International Journal of Robotics Research, 1983, 2(2):35-4535-45, 1983. [cited by applicant]
Herlant et al., “Assistive teleoperation of robot arms via automatic time-optimal mode switching,” The Eleventh ACM/IEEE International Conference on Human Robot Interaction, 2016, 35-42. [cited by applicant]
Introduction to Robotics: Mechanics and Control, Addison-Wesley series in electrical and computer engineering: control engineering, Pearson/Prentice Hall, 1986. [cited by applicant]
Jain et al., “Assistive robotic manipulation through shared autonomy and a body-machine interface,” 2015 IEEE International Conference on Rehabilitation Robotics (ICORR), 2015, pp. 526-531. [cited by applicant]
Lin et al., “The implementation of augmented reality in a robotic teleoperation system,” 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR), 2016, pp. 134-139. [cited by applicant]
Mandlekar et al., “Roboturk: A crowdsourcing platform for robotic skill learning through imitation,” 2nd Conference on Robot Learning (CoRL 2018), Nov. 2018, pp. 1-15. [cited by applicant]
McMullen et al., “Demonstration of a semi-autonomous hybrid brain-machine interface using human intracranial eeg, eye tracking, and computer vision to control a robotic upper limb prosthetic,” IEEE Transactions on Neura… [cited by applicant]
Meattini et al., “An semg-based human-robot interface for robotic hands using machine learning and synergies,” IEEE Transactions on Components, Packaging and Manufacturing Technology, 2018, 8(7):1149-1158. [cited by applicant]
Mohammed et al., “Brainwaves driven human-robot collaborative assembly,” CIRP annals, 2018, 67(1):13-16. [cited by applicant]
Muelling et al., “Autonomy infused teleoperation with application to brain computer interface controlled manipulation,” Autonomous Robots, 2017, 41:1401-1422. [cited by applicant]
Vu et al., “A new 6-dof haptic device for teleoperation of 6-dof serial robots,” IEEE Transactions on Instrumentation and Measurement, 2011, 60:3510-3523. [cited by applicant]
Wilson et al., “Relative end-effector control using cartesian position based visual servoing,” IEEE Transactions on Robotics and Automation, 1996, 12(5):684-696. [cited by applicant]
Wu et al., “Development of smartphone based human-robot interfaces for individuals with disabilities,” IEEE Robotics and Automation Letters, 2020, 5(4):5835-5841. [cited by applicant]
Yang et al., “Interfacedesign of a physical human-robot interaction system for humanimpedance adaptive skill transfer,” IEEE Transactions on Automation Science and Engineering, 2017, 15(1):329-340. [cited by applicant]