Granted Patent B2

US 12,602,823 · App. 18/254,181 · Granted Apr 14, 2026

Inventors: Xuesong Shi (Beijing, CN); Yuxin Tian (Beijing, CN); Sangeeta Ghangam (Chandler, AZ); Dawei Wang (Beijing, CN)

Assignee: Intel Corporation

G06T7/74G06T1/0014G06T3/147G06T7/60G06T2207/10024G06T2207/10028

Abstract

Claims (82)

1 . A method for re-localization of a robot, comprising:

retrieving, for each keyframe of a set of keyframes in a keyframe database of the robot, image features and a pose of the keyframe, the image features of the keyframe comprising a global descriptor and local descriptors of the keyframe;

extracting image features of a current frame captured by the robot, the image features of the current frame comprising a global descriptor and local descriptors of the current frame;

determining one or more rough matching frames from the set of keyframes based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame;

determining a final matching frame from the one or more rough matching frames based on comparison between the local descriptors of each rough matching frame and the local descriptors of the current frame, wherein determining the final matching frame comprises:

for each rough matching frame,

determining, for each local descriptor of the current frame, a matching point in the rough matching frame corresponding to a keypoint in the current frame represented by the local descriptor of the current frame;

calculating a fundamental matrix or a homography matrix between the current frame and the rough matching frame based on the keypoint in the current frame and the matching point in the rough matching frame;

calculating a reprojection error between the keypoint in the current frame and the matching point in the rough matching frame based on the fundamental matrix or the homography matrix; and

determining the matching point as an inline point when the reprojection error is smaller than a predetermined threshold, and

selecting a rough matching frame with a largest number of inline points as the final matching frame; and

calculating a pose of the current frame based on a pose of the final matching frame.

2 . The method of claim 1 , wherein determining the one or more rough matching frames comprises:

calculating a Euclidean distance between the global descriptor of each keyframe and the global descriptor of the current frame; and

determining a predetermined number of keyframes with a smallest Euclidean distance as the one or more rough matching frames.

3 . The method of claim 1 , wherein determining the final matching frame comprises:

for each rough matching frame, calculating, for each local descriptor of the current frame, a Euclidean distance between each local descriptor of the rough matching frame and the local descriptor of the current frame,

wherein the matching point in the rough matching frame is represented by a local descriptor of the rough matching frame with a minimum Euclidean distance from the local descriptor of the current frame.

4 . The method of claim 3 , further comprising:

discarding, for each rough matching frame and each local descriptor of the current frame, the matching point when the minimum Euclidean distance is not less than a predetermined percentage of a second smallest Euclidean distance between the local descriptors of the rough matching frame and the local descriptor of the current frame.

5 . The method of claim 3 , wherein determining the final matching frame further comprises:

for each rough matching frame, when a number of inline points in the rough matching frame is greater than a predetermined threshold,

deriving a Rotation transform and a Translation transform between the rough matching frame and the current frame based on the fundamental matrix or the homography matrix; and

calculating the pose of the current frame based on the pose of the rough matching frame, the Rotation transform and the Translation transform between the rough matching frame and the current frame;

calculating a deviation between a predetermined percentage of poses of the current frame calculated based on corresponding rough matching frames; and

selecting the rough matching frame with the largest number of inline points as the final matching frame when the deviation does not exceed a predetermined threshold.

6 . The method of claim 3 , wherein calculating the pose of the current frame comprises:

calculating a Rotation transform and a Translation transform between the current frame and the final matching frame based on the fundamental matrix or the homography matrix between the current frame and the final matching frame; and

calculating the pose of the current frame based on the Rotation transform, the Translation transform and the pose of the final matching frame.

7 . The method of claim 3 , wherein the image features of the current frame comprise a depth value of the keypoint in the current frame, the image features of the final matching frame comprise a depth value of the inline point in the final matching frame, and calculating the pose of the current frame comprises:

calculating a Rotation transform and a Translation transform between the current frame and the final matching frame based on the fundamental matrix or the homography matrix between the current frame and the final matching frame;

calculating a scale S for correcting the Translation transform between the current frame and the final matching frame based on the depth values of the keypoints in the current frame and the depth values of the corresponding inline points in the final matching frame; and

calculating the pose of the current frame based on the Rotation transform, the Translation transform corrected by the scale S and the pose of the final matching frame.

8 . The method of claim 1 , further comprising:

receiving Red Green Blue Depth (RGBD) images from a visual system of the robot during a tracking stage of the robot;

acquiring the set of keyframes from the RGBD images;

extracting the image features of the set of keyframes via a Hierarchical Feature Network (HF-Net) model;

obtaining the poses of the set of keyframes from a Simultaneous Localization and Mapping (SLAM) system of the robot; and

storing the image features and the poses of the set of keyframes in the keyframe database.

9 . The method of claim 8 , wherein the SLAM system is a Laser SLAM system or a Visual SLAM system.

10 . The method of claim 1 , wherein extracting the image features of the current frame comprises extracting the image features of the current frame via a Hierarchical Feature Network (HF-Net) model.

11 . An apparatus for re-localization of a robot, comprising interface circuitry; processor circuitry coupled to the interface circuitry and configured to:

retrieve, for each keyframe of a set of keyframes in a keyframe database of the robot, image features and a pose of the keyframe from the keyframe database via the interface circuitry, the image features of the keyframe comprising a global descriptor and local descriptors of the keyframe;

extract image features of a current frame captured by the robot, the image features of the current frame comprising a global descriptor and local descriptors of the current frame;

determine one or more rough matching frames based on comparison between the global descriptor of each keyframe and the global descriptor of the current frame;

determine a final matching frame based on comparison between the local descriptors of each rough matching frame and the local descriptors of the current frame, wherein the processor circuitry configured to determine the final matching frame comprises the processor circuitry to:

for each rough matching frame,

determine, for each local descriptor of the current frame, a matching point in the rough matching frame corresponding to a keypoint in the current frame represented by the local descriptor of the current frame;

calculate a fundamental matrix or a homography matrix between the current frame and the rough matching frame based on the keypoint in the current frame and the matching point in the rough matching frame;

calculate a reprojection error between the keypoint in the current frame and the matching point in the rough matching frame based on the fundamental matrix or the homography matrix; and

determine the matching point as an inline point when the reprojection error is smaller than a predetermined threshold, and

select a rough matching frame with a largest number of inline points as the final matching frame; and

calculate a pose of the current frame based on a pose of the final matching frame.

12 . The apparatus of claim 11 , wherein the processor circuitry is configured to determine the one or more rough matching frames by:

calculating a Euclidean distance between the global descriptor of each keyframe and the global descriptor of the current frame; and

determining a predetermined number of keyframes with a smallest Euclidean distance as the one or more rough matching frames.

13 . The apparatus of claim 11 , wherein the processor circuitry is configured to determine the final matching frame by:

for each rough matching frame,

calculating, for each local descriptor of the current frame, a Euclidean distance between each local descriptor of the rough matching frame and the local descriptor of the current frame,

wherein the matching point in the rough matching frame is represented by a local descriptor of the rough matching frame with a minimum Euclidean distance from the local descriptor of the current frame.

14 . The apparatus of claim 13 , wherein the processor circuitry is further configured to:

discard, for each rough matching frame and each local descriptor of the current frame, the matching point when the minimum Euclidean distance is not less than a predetermined percentage of a second smallest Euclidean distance between the local descriptors of the rough matching frame and the local descriptor of the current frame.

15 . The apparatus of claim 13 , wherein the processor circuitry is configured to determine the final matching frame further by:

for each rough matching frame, when a number of inline points in the rough matching frame is greater than a predetermined threshold,

deriving a Rotation transform and a Translation transform between the rough matching frame and the current frame based on the fundamental matrix or the homography matrix; and

calculating the pose of the current frame based on the pose of the rough matching frame, the Rotation transform and the Translation transform between the rough matching frame and the current frame;

calculating a deviation between a predetermined percentage of poses of the current frame calculated based on corresponding rough matching frames; and

selecting the rough matching frame with the largest number of inline points as the final matching frame when the deviation does not exceed a predetermined threshold.

16 . The apparatus of claim 13 , wherein the processor circuitry is configured to calculate the pose of the current frame by:

calculating a Rotation transform and a Translation transform between the current frame and the final matching frame based on the fundamental matrix or the homography matrix between the current frame and the final matching frame; and

calculating the pose of the current frame based on the Rotation transform, the Translation transform and the pose of the final matching frame.

17 . The apparatus of claim 13 , wherein the image features of the current frame comprise a depth value of the keypoint in the current frame, the image features of the final matching frame comprise a depth value of the inline point in the final matching frame, and the processor circuitry is configured to calculate the pose of the current frame by:

calculating a Rotation transform and a Translation transform between the current frame and the final matching frame based on the fundamental matrix or the homography matrix between the current frame and the final matching frame;

calculating a scale S for correcting the Translation transform between the current frame and the final matching frame based on the depth values of the keypoints in the current frame and the depth values of the corresponding inline points in the final matching frame; and

calculating the pose of the current frame based on the Rotation transform, the Translation transform corrected by the scale S and the pose of the final matching frame.

18 . The apparatus of claim 11 , wherein the interface circuitry is coupled to a visual system and a Simultaneous Localization and Mapping (SLAM) system of the robot and the processor circuitry is further configured to:

acquire the set of keyframes from Red Green Blue Depth (RGBD) images received via the interface circuitry from the visual system of the robot during a tracking stage of the robot;

extract the image features of the set of keyframes via a Hierarchical Feature Network (HF-Net) model;

obtain the poses of the set of keyframes via the interface circuitry from the SLAM system of the robot; and

store the image features and the poses of the set of keyframes in the keyframe database via the interface circuitry.

19 . The apparatus of claim 18 , wherein the SLAM system is a Laser SLAM system or a Visual SLAM system.

20 . The apparatus of claim 11 , wherein the processor circuitry is configured to extract the image features of the current frame via a Hierarchical Feature Network (HF-Net) model.

Assignments (2)

Continuity (1)

References Cited (23)