IP Library Granted Patent US 10,173,324
Granted Patent B2
US 10,173,324 · App. 14/941,823 · Granted Jan 8, 2019

Facilitating robot positioning

Inventors: Jianjun Wang (West Hartford, CT); Biao Zhang (West Hartford, CT); Carlos Martinez (South Windsor, CT); Carlos Winsor Morato (Avon, CT); Remus Boca (Simsbury, CT)
Assignee: ABB Schweiz AG
B25J9/1697B25J9/1692G06T7/74G06T2200/04G06T2207/10028G06T2207/30244Y10S901/09
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,173,324
App. No.
14/941,823
Granted
Jan 8, 2019
Kind
B2
Abstract

Robot positioning is facilitated by obtaining, for each time of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on a comparison of multiple three-dimensional images of a scene of an environment, the obtaining providing a plurality of indications of poses of the camera system; obtaining, for each time of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot; and determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot.

Claims (54)

1. A method comprising:

obtaining, for each time of a plurality of times of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on a comparison of multiple three-dimensional images of a scene of an environment, the obtaining providing a plurality of indications of poses of the camera system;

obtaining, for each time of a plurality of times of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot;

determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot; and

stitching together images of at least part of the scene of the environment using an image registration method or using measured robot positions and the determined indication of the reference point of the camera system.

2. The method of claim 1 , wherein the using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot comprises building a plurality of equations relating the plurality of poses of the camera system across the first sampling schedule to the plurality of poses of the robot across the second sampling schedule, the relating being a function of the reference coordinate frame and the reference point of the camera system relative to the pose of robot, and wherein the determining comprises finding a solution for the plurality of equations to determine the indication of the reference coordinate frame and the indication of the reference point of the camera system.

3. The method of claim 2 , wherein the building the plurality of equations comprises, for each indication of a pose of the camera system, of the plurality of indications of poses of the camera system, identifying a respective corresponding pose of the robot, the respective corresponding pose of the robot based on one or more indications of poses of the robot of the plurality of indications of poses of the robot.

4. The method of claim 3 , wherein the first sampling schedule and second sampling schedule differ by a time-offset, and an identified respective corresponding pose of the robot for an indication of the pose of the camera system comprises an interpolated pose of the robot interpolated based on the one or more indications of poses of the robot being a pose of the robot at a time that differs from a time of obtaining one of the plurality of indications of poses of the robot by an amount of time equal to time-offset.

5. The method of claim 4 , wherein the time-offset is initially unknown, and wherein the finding the solution for the plurality of equations comprises either finding a solution for the time-offset with the finding the solution for the plurality of equations to determine the indication of the reference coordinate frame and the indication of the reference point of the camera system, or first finding a solution for the time-offset and using that solution for the time-offset in finding the solution for the plurality of equations.

6. The method of claim 1 , wherein pose of the robot is represented by pose of a mounting flange of the robot.

7. The method of claim 1 , wherein pose of the robot is represented by joint measurements of the robot.

8. The method of claim 1 , wherein the obtained plurality of indications of poses of the camera system and obtained plurality of indications of poses of the robot are obtained based on the robot obtaining images of a scene of an environment.

9. The method of claim 1 , wherein the comparison is performed absent knowledge of distance between features of the scene.

10. The method of claim 1 , wherein the scene comprises features of unknown characteristics.

11. The method of claim 1 , wherein the determined indication of the reference point of the camera system serves as an automated calibration for the reference point of the camera system.

12. The method of claim 1 , further comprising:

identifying a position of a feature in a scene space of a scene imaged by the robot;

using the determined indication of the reference coordinate frame and a relation between the determined indication of the reference coordinate frame and the identified position of the feature in the scene space to identify a position of the feature in robot space; and

providing the identified position of the feature in robot space to inform positioning of the robot.

13. The method of claim 1 , wherein the obtaining the plurality of indications of poses of the camera system and the obtaining the plurality of indications of poses of the robot is associated with robot movement and imaging of a first scan region of an environment in which the robot operates, and the determining determines the indication of the reference coordinate frame and the indication of the reference point of the camera system for the first scan region, and wherein the method further comprises:

repeating the obtaining the plurality of indications of poses of the camera system and the obtaining the plurality of indications of poses of the robot in association with robot movement and imaging of at least one other scan region of the environment; and

repeating the determining to determine an indication of a reference coordinate frame and an indication of a reference point of the camera system for the at least one other scan region.

14. A computer system comprising:

a controller configured to execute program instructions to perform the acts of:

obtaining, for each time of a plurality of times of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on a comparison of multiple three-dimensional images of a scene of an environment, the obtaining providing a plurality of indications of poses of the camera system;

obtaining, for each time of a plurality of times of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot;

determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot; and

stitching together images of at least part of the scene of the environment using an image registration method or using measured robot positions and the determined indication of the reference point of the camera system.

15. The computer system of claim 14 , wherein the using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot comprises building a plurality of equations relating the plurality of poses of the camera system across the first sampling schedule to the plurality of poses of the robot across the second sampling schedule, the relating being a function of the reference coordinate frame and the reference point of the camera system relative to the pose of robot, and wherein the determining comprises finding a solution for the plurality of equations to determine the indication of the reference coordinate frame and the indication of the reference point of the camera system.

16. The computer system of claim 14 , wherein the scene comprises features of unknown characteristics.

17. The computer system of claim 14 , wherein the controller is configured to execute the program instructions to perform the acts of:

identifying a position of a feature in a scene space of a scene imaged by the robot;

using the determined indication of the reference coordinate frame and a relation between the determined indication of the reference coordinate frame and the identified position of the feature in the scene space to identify a position of the feature in robot space; and

providing the identified position of the feature in robot space to inform positioning of the robot.

18. A computer program product comprising:

a computer readable storage medium readable by a processor and storing instructions for execution by the processor for performing a method comprising:

obtaining, for each time of a plurality of times of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on a comparison of multiple three-dimensional images of a scene of an environment, the obtaining providing a plurality of indications of poses of the camera system;

obtaining, for each time of a plurality of times of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot;

determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot; and

stitching together images of at least part of the scene of the environment using an image registration method or using measured robot positions and the determined indication of the reference point of the camera system.

19. The computer program product of claim 18 , wherein the using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot comprises building a plurality of equations relating the plurality of poses of the camera system across the first sampling schedule to the plurality of poses of the robot across the second sampling schedule, the relating being a function of the reference coordinate frame and the reference point of the camera system relative to the pose of robot, and wherein the determining comprises finding a solution for the plurality of equations to determine the indication of the reference coordinate frame and the indication of the reference point of the camera system.

20. The computer program product of claim 18 , wherein the scene comprises features of unknown characteristics.

21. A method, comprising:

obtaining a plurality of three-dimensional images of a scene of an environment;

comparing the plurality of three-dimensional images of the scene;

obtaining, for each time of a plurality of times of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on the comparing of the plurality of three-dimensional images, the obtaining providing a plurality of indications of poses of the camera system;

obtaining, for each time of a plurality of times of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot;

determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot;

using an image registration method and also using measured robot positions and the determined indication of the reference point of the camera system to stitch together images of at least part of the scene of the environment;

identifying a position of a feature in a scene space of the scene;

determining a relation between the determined indication of the reference coordinate frame and the identified position of the feature in the scene space;

identifying the position of the feature in a robot space based on the identified position of a feature in a scene space of the scene, the determined indication of the reference coordinate frame, and the determined relation between the determined indication of the reference coordinate frame and the identified position of the feature in the scene space;

informing the position of the robot based on the identified position of the feature in the robot space; and

operating the robot based on the informed position of the robot.

Assignments (3)
MERGER Recorded Nov 15, 2016
From: ABB TECHNOLOGY LTD.
To: ABB SCHWEIZ AG
Reel/Frame 040621/0792 →
MERGER Recorded Oct 20, 2016
From: ABB TECHNOLOGY AG
To: ABB SCHWEIZ AG
Reel/Frame 040441/0836 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 8, 2015
From: WANG, JIANJUN; ZHANG, BIAO; MARTINEZ, CARLOS; MORATO, CARLOS; BOCA, REMUS
To: ABB TECHNOLOGY AG
Reel/Frame 037231/0921 →
Continuity (1)
Related Publication 20170136626A1 · May 18, 2017