In-hand pose refinement for pick and place automation

1. A method of tracking a pose of an object, the method comprising:

moving, by a robotic arm in a workcell, a physical object along a path;

determining, from one or more first images captured by one or more first cameras in the workcell, an initial pose of the object at a first position on the path;

receiving position data and velocity data corresponding to movement of the object along the path to a second position on the path by the robotic arm;

determining an expected pose of the object at the second position on the path based on the position and velocity data and the initial pose of the object;

receiving second image data captured by one or more second cameras in the workcell, the second image data being image data of the object at the second position; and

determining a refined pose of the object at the second position based on the second image data and the expected pose, including:

determining expected keypoints on the expected pose of the object and estimated keypoints on the object from the second image data,

providing the expected pose as an initial condition to a keypoint alignment process to compute an alignment of the expected keypoints and the estimated keypoints, and

computing the refined pose from the alignment of the expected keypoints and the estimated keypoints.

2. The method of claim 1 , wherein the robotic arm is configured to grasp the object and to move the object from the first position to the second position.

3. The method of claim 1 , wherein the one or more second cameras comprise a stereo-pair of camera modules, and

wherein the second image data comprises a depth map or a surface normals map of the object at the second position.

4. The method of claim 1 , wherein the initial pose, the expected pose, and the refined pose each correspond to a six-degrees-of-freedom (6DoF) pose of the object.

5. The method of claim 1 , wherein the determining the initial pose of the object at the first position comprises:

receiving first image data corresponding to the object at the first position from the one or more first cameras;

identifying a 3-D model corresponding to the object; and

aligning the 3-D model to be consistent with an appearance of the object associated with the first image data, and to generate the initial pose of the object.

6. The method of claim 5 , wherein the first image data comprises a first depth map or a first surface normals map of the object, and

wherein the aligning the 3-D model comprises:

extracting observed keypoints of the object from the first depth map or the first surface normals map of the object.

7. The method of claim 6 , wherein the 3-D model comprises a plurality of modeled keypoints of the object, and

wherein the aligning the 3-D model further comprises:

applying an iterative closest point (ICP) algorithm or a point pair feature matching algorithm to align the modeled keypoints with the observed keypoints.

8. The method of claim 1 , wherein determining the expected pose of the object at the second position comprises:

identifying initial keypoints of the object based on the initial pose;

performing transformations on the initial keypoints based on the position and velocity data to generate expected keypoints; and

determining the expected pose based on the expected keypoints.

9. The method of claim 8 , wherein the refined pose is a more accurate representation of an actual pose of the object than the expected pose.

10. The method of claim 1 , wherein performing the keypoint alignment process comprises:

identifying the estimated keypoints of the object based on the second image data; and

aligning the estimated keypoints with the expected keypoints by applying an iterative closest point (ICP) algorithm or a point pair feature matching algorithm to the estimated keypoints and the expected keypoints to generate aligned keypoints.

11. The method of claim 10 , wherein the second image data comprises a second depth map or a second surface normals map of the object, and

wherein identifying the estimated keypoints comprises:

extracting the estimated keypoints of the object from the second depth map or the second surface normals map of the object.

12. The method of claim 1 , further comprising:

calibrating a first camera to the robotic arm at the first position; and

recalibrating the first camera to the robotic arm at the second position based on the second image data,

wherein the second image data comprises calibration pattern data corresponding to a calibration pattern on the robotic arm.

13. The method of claim 12 , wherein the recalibrating the first camera to the robotic arm at the second position comprises:

computing expected calibration points based on the position and velocity data and an initial calibration of the first camera to the robotic arm;

identifying observed calibration points based on the second image data;

aligning the observed calibration points with the expected calibration points, by applying an iterative closest point (ICP) algorithm or a point pair feature matching algorithm to the observed and expected calibration points, to generate refined calibration points; and

calibrating the first camera to the robotic arm based on the refined calibration points.

14. A pose tracking system comprising:

a robotic arm; and

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

providing instructions to the robotic arm that cause the robotic arm to move a physical object along a path in a workcell;

determining, from one or more first images captured by one or more first cameras in the workcell, an initial pose of the object at a first position on the path;

receiving position data and velocity data corresponding to movement of the object along the path to a second position on the path by the robotic arm;

determining an expected pose of the object at the second position on the path based on the position and velocity data and the initial pose of the object;

receiving second image data captured by one or more second cameras in the workcell, the second image data being image data of the object at the second position; and

determining a refined pose of the object at the second position based on the second image data and the expected pose, including:

determining expected keypoints on the expected pose of the object and estimated keypoints on the object from the second image data,

providing the expected pose as an initial condition to a keypoint alignment process to compute an alignment of the expected keypoints and the estimated keypoints, and

computing the refined pose from the alignment of the expected keypoints and the estimated keypoints.

15. The pose tracking system of claim 14 ,

wherein the robotic arm is configured to grasp the object and to move the object from the first position to the second position.

16. The pose tracking system of claim 14 , further comprising a moveable camera platform configured to track movements of the robotic arm and to keep the object within a field of view of the one or more first cameras or the one or more second cameras, or both.

17. The pose tracking system of claim 14 , wherein the one or more first cameras or one or more second cameras comprise a plurality of stereo-pair cameras having partially overlapping fields of view that cover an entire range of motion of the moving device.

18. The pose tracking system of claim 14 , wherein determining the expected pose of the object at the second position comprises:

identifying initial keypoints of the object based on the initial pose;

performing transformations on the initial keypoints based on the position and velocity data to generate expected keypoints; and

determining the expected pose based on the expected keypoints.

19. The pose tracking system of claim 14 , wherein performing the keypoint alignment process comprises:

identifying the estimated keypoints of the object based on the second image data; and

aligning the estimated keypoints with the expected keypoints by applying an iterative closest point (ICP) algorithm or a point pair feature matching algorithm to the estimated keypoints and the expected keypoints to generate aligned keypoints.

20. The pose tracking system of claim 19 , wherein the second image data comprises a second depth map or a second surface normals map of the object, and

wherein the identifying the estimated keypoints comprises:

extracting the estimated keypoints of the object from the second depth map or the second surface normals map of the object.