Granted Patent B2

US 12,380,587 · App. 17/812,154 · Granted Aug 5, 2025

Inventors: Barrett Clark (Seattle, WA); Oytun Akman (Oakland, CA); Matthew Brown (Seattle, WA); Ronald Poelman (Wassenaar, NL); Brian Philip Mathews (Burlingame, CA); Emmanuel Gallo (Oakland, CA); Ali Shafiekhani (San Francisco, CA)

Assignee: Bright Machines, Inc.

G06T7/70B25J9/1697G06T1/0014G06T7/0004G06T7/60G06T11/203G06T15/00G05B2219/37559G05B2219/40589G06T2207/20092G06T2207/30164

Abstract

Claims (64)

1. A method for vision-based tool localization (VTL) in a robotic assembly system including one or more calibrated sensors and a robot arm holding a tool, the method comprising:

capturing a plurality of images of a tool contact area of the tool from a plurality of different vantage points;

evaluating a first vantage point of the plurality of different vantage points by:

rendering an outline of the tool contact area based on an estimated position of the tool;

overlaying the outline over an image from the first vantage point of the tool contact area; and

minimizing a distance between the rendered outline and the image;

determining an estimated position of the tool contact area with respect to the robot arm based on the evaluating;

refining the estimated position of the tool contact area based on a second evaluation from a second vantage point of the plurality of different vantage points; and

providing the refined position of the tool contact area with respect to the robot arm to the robotic assembly system to enable accurate control of the tool by the robotic assembly system.

2. The method of claim 1 , further comprising:

identifying a tool center point; and

when the tool center point is not the tool contact area, calculating the tool center point.

3. The method of claim 1 , further comprising:

utilizing a second approach for identifying the tool contact area, wherein the second approach comprises one of:

requesting user input to select the tool contact area; and

space carving.

4. The method of claim 1 , wherein the plurality of images are taken by a plurality of cameras, and the tool contact area does not move.

5. The method of claim 1 , wherein the plurality of images are taken by one camera, and the tool contact area is moved between the images being taken.

6. The method of claim 1 , further comprising:

computing an offset between the estimated position and the refined position.

7. The method of claim 1 wherein the outline is a two-dimensional outline.

8. The method of claim 1 , wherein the outline is a three-dimensional outline.

9. The method of claim 1 , further comprising:

rendering a two-dimensional outline based on the estimated position; and

refining a three-dimensional shape position based on the two-dimensional outline.

10. A method for vision-based tool localization (VTL) in a robotic assembly system including one or more calibrated sensors and a robotic arm holding a tool, the method comprising:

capturing a plurality of images of a tool contact area of the tool from a plurality of different vantage points;

determining an estimated position of the tool contact area based on user input, indicating a position of the tool contact area;

refining the estimated position of the tool contact area based on another determination from another vantage point; and

providing the refined position of the tool contact area with respect to the robotic arm to the robotic assembly system to enable accurate control of the tool by the robotic assembly system.

11. The method of claim 10 , further comprising:

prompting user to make a selection by clicking on a designated point on the tool contact area;

identifying coordinates from the click; and

utilizing the coordinates from a plurality of clicks to compute the refined position of the tool contact area.

12. The method of claim 11 , further comprising:

providing helper data based on prior selections to the user.

13. The method of claim 11 , further comprising:

zooming in toward the tool contact area, prior to prompting the user to make the selection.

14. The method of claim 11 , further comprising:

mapping a sight line from a camera to the designated point; and

utilizing a plurality of sight lines to identify the refined position of the tool contact area.

15. A method for vision-based tool localization (VTL) in a robotic assembly system including one or more calibrated sensors, the method comprising:

capturing a plurality of images of a tool contact area of a tool on a robot arm from a plurality of different vantage points;

selecting a plurality of approaches to determine a tool contact area location;

determining an estimated position of the tool contact area using a first approach, wherein the first approach comprises one of: user selection, image overlay, and space carving;

refining the estimated position of the tool contact area with respect to the robot arm holding the tool based on a second approach, comprising a different one of: user selection, image overlay, and space carving; and

providing the refined position of the tool contact area with respect to the robot arm to the robotic assembly system to enable accurate control of the tool by the robotic assembly system.

16. The method of claim 15 , wherein the user selection comprises:

prompting a user to make a selection by clicking on a designated point on the tool contact area;

identifying coordinates from the click; and

utilizing the coordinates from a plurality of clicks to compute the tool contact area location.

17. The method of claim 15 , wherein the space carving comprises:

wherein determining the estimated position of the tool contact area comprises carving out empty space that is not the tool contact area; and

wherein refining the estimated position comprises carving out the empty space that is not the tool contact area from an image with a different vantage point.

18. The method of claim 15 , wherein determining the estimated position comprises:

rendering an outline based on an estimated position of the tool contact area;

overlaying the outline over an image of the tool contact area; and

minimizing a distance between the rendered outline and the image.

19. The method of claim 15 , further comprising:

weighting an accuracy for each of the plurality of approaches; and

utilizing the accuracy-weighted approaches to solve for the refined position.

20. The method of claim 19 , further comprising:

identifying one of the plurality of approaches with an accuracy rating below a threshold; and

suggesting that a user re-run the identified approach.

Assignments (6)

Continuity (2)

References Cited (123)