Granted Patent B2

US 12,403,590 · App. 17/819,685 · Granted Sep 2, 2025

Inventors: Tara Boroushaki (Cambridge, MA); Isaac S. Perper (Cambridge, MA); Fadel Adib (Cambridge, MA)

Assignee: Massachusetts Institute of Technology

B25J9/1612B25J13/006B25J19/023H01Q1/2208

Abstract

Claims (44)

1. A robotic system comprising:

A robotic member having an end effector;

An imaging device coupled to the robotic member proximate to or on the end effector, the imaging device configured to provide visual information;

A radio frequency (RF) system configured to collect and provide information, the RF system including an antenna coupled to the robotic member proximate to or on the end effector; and

A controller in communication with the robotic member, the imaging device and the RF system, the controller configured to geometrically fuse RF distance information provided thereto by the RF system and visual information provided thereto by the imaging device to reduce uncertainty about a location of a target object even when the object is fully occluded,

Wherein the controller is further configured to:

Iteratively geometrically fuse the RF distance information and the visual information to reduce uncertainty about the target object location;

Determine whether the uncertainty is smaller than a threshold value; and

In response to the uncertainty being smaller than the threshold value, cause the end effector to manipulate the target object.

2. The robotic system of claim 1 wherein the end effector comprises a gripper and wherein the imaging device is coupled to a portion of the gripper.

3. The robotic system of claim 1 wherein the end effector comprises a gripper and wherein the antenna is coupled to a portion of the gripper.

4. The robotic system of claim 1 , wherein the controller is further configured to cause movement of the end effector to a position that reduces uncertainty about the target object location.

5. The robotic system of claim 1 , wherein the controller is further configured to identify an occluding object.

6. The robotic system of claim 5 , wherein the controller is further configured to cause the end effector to manipulate the occluding object.

7. The robotic system of claim 1 , wherein the controller is further configured to:

cause the end effector to grasp the target object;

receive an RF signal from the target object; and

verify, using the received RF signal, that the end effector has picked up the target object.

8. The robotic system of claim 1 , wherein the controller is further configured to process the fused RF distance and visual information using a reinforcement-learning network to localize, maneuver toward, and grasp the target object.

9. A reinforcement-learning network comprising:

means for receiving fused RF distance and visual information;

means for processing the fused RF distance and visual information to efficiently localize, maneuver toward, and grasp a target object; and

a reinforcement-learning agent configured to be rewarded based on a distance traveled to a location of the target object and a number of vantage points required until successful localization of the target object.

10. The reinforcement-learning network of claim 9 , further comprising an RF-visual encoder configured to encode RF distance information and visual information to be provided as a feature input to the reinforcement-learning network.

11. An apparatus for locating a target object, the apparatus comprising:

An imaging device configured to collect visual data and provide visual information, the imaging device including a camera coupled to the apparatus;

A radio frequency (RF) system configured to collect and provide RF information, the RF system including an antenna coupled to the apparatus; and

A processing module in communication with the imaging device and the RF system, the processing module configured to:

Estimate a range to the target object using the RF information, the range including one or more distance estimates between a location of the antenna and a location of the target object;

Generate an occupancy map of an environment of the imaging device using the visual information;

Geometrically fuse the range obtained from the RF information with the occupancy map from the visual information; and

Use the fused information to identify one or more candidate regions for a location of the target object,

Wherein the imaging device and the RF system are mounted on a robotic end effector, and wherein the processing module is further configured to:

Iteratively geometrically fuse the range obtained from the RF information and the visual information to reduce the uncertainty about the target object location;

Determine whether the uncertainty is smaller than a threshold value; and

In response to the uncertainty being smaller than the threshold value, cause the end effector to manipulate the target object.

12. The apparatus of claim 11 , wherein the imaging device and the RF system are mounted on a robot.

13. The apparatus of claim 11 , wherein the processing module is further configured to process the fused RF distance and visual information using a reinforcement-learning network to localize, maneuver toward, and grasp the target object.

14. The apparatus of claim 13 , wherein the reinforcement-learning network includes a reinforcement-learning agent configured to be rewarded based on a distance traveled to a location of the target object and a number of vantage points required until successful localization of the target object.

15. The apparatus of claim 11 , wherein the occupancy map includes a 3D point cloud.

16. The apparatus of claim 11 , wherein the processing module is further configured to:

cause the end effector to grasp the target object;

receive an RF signal from the target object; and

verify, using the received RF signal, that the end effector has picked up the target object.

Assignments (1)

Continuity (2)

References Cited (84)