Learning reliable keypoints in situ with introspective self-supervision
An apparatus to facilitate learning reliable keypoints in situ with introspective self-supervision is disclosed. The apparatus includes one or more processors to provide a view-overlapped keyframe pair from a pose graph that is generated by a visual simultaneous localization and mapping (VSLAM) process executed by the one or more processors; determine a keypoint match from the view-overlapped keyframe pair based on a keypoint detection and matching process, the keypoint match corresponding to a keypoint; calculate an inverse reliability score based on matched pixels corresponding to the keypoint match in the view-overlapped keyframe pair; identify a supervision signal associated with the keypoint match, the supervision signal comprising a keypoint reliability score of the keypoint based on a final pose output of the VSLAM process; and train a keypoint detection neural network using the keypoint match, the inverse reliability score, and the keypoint reliability score.
1 . An apparatus comprising:
one or more processors to:
provide a view-overlapped keyframe pair from a pose graph that is generated by a visual simultaneous localization and mapping (VSLAM) process executed by the one or more processors, wherein the view-overlapped keyframe pair comprises a matched pixel set (p, p′);
determine a keypoint match from the view-overlapped keyframe pair based on a keypoint detection and matching process, the keypoint match corresponding to a keypoint;
calculate an inverse reliability score based on matched pixels corresponding to the keypoint match in the view-overlapped keyframe pair, wherein the inverse reliability score comprises a pixel distance between p′ and an epipolar line for p;
identify a supervision signal associated with the keypoint match, the supervision signal comprising a keypoint reliability score of the keypoint based on a final pose output of the VSLAM process; and
train a keypoint detection neural network using the keypoint match, the inverse reliability score, and the keypoint reliability score.
2 . The apparatus of claim 1 , wherein the view-overlapped keyframe pair comprises a pair of image frames captured by a camera, and wherein the keypoint match corresponds to the keypoint that is present in each of the image frames in the view- overlapped keyframe pair.
3 . The apparatus of claim 2 , wherein the keypoint comprises a landmark in a scene of the view-overlapped keyframe pair.
4 . The apparatus of claim 1 , wherein the keypoint detection neural network comprises a convolutional neural network (CNN).
5 . The apparatus of claim 1 , wherein the keypoint reliability score is based on a comparison of coordinates of the keypoint in the final pose output generated by the VSLAM process to saved coordinates for a scene of the view-overlapped keyframe pair.
6 . The apparatus of claim 1 , wherein the one or more processors are further to:
regress the inverse reliability score into a regressed inverse reliability score;
train a separate head of the keypoint detection neural network with the regressed inverse reliability score; and
combine the regressed inverse reliability score with the keypoint reliability score to obtain a final keypoint reliability score.
7 . The apparatus of claim 1 , wherein the apparatus comprises a robot utilizing the VSLAM process for localization of the robot.
8 . The apparatus of claim 1 , wherein the one or more processors comprise one or more of a graphics processor, an application processor, and another processor, wherein the one or more processors are co-located on a common semiconductor package.
9 . A non-transitory computer-readable storage medium having stored thereon executable computer program instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
providing a view-overlapped keyframe pair from a pose graph that is generated by a visual simultaneous localization and mapping (VSLAM) process executed by the one or more processors, wherein the view-overlapped keyframe pair comprises a matched pixel set (p, p′);
determining a keypoint match from the view-overlapped keyframe pair based on a keypoint detection and matching process, the keypoint match corresponding to a keypoint;
calculating an inverse reliability score based on matched pixels corresponding to the keypoint match in the view-overlapped keyframe pair, wherein the inverse reliability score comprises a pixel distance between p′ and an epipolar line for p;
identifying a supervision signal associated with the keypoint match, the supervision signal comprising a keypoint reliability score of the keypoint based on a final pose output of the VSLAM process; and
training a keypoint detection neural network using the keypoint match, the inverse reliability score, and the keypoint reliability score.
10 . The non-transitory computer-readable storage medium of claim 9 , wherein the view-overlapped keyframe pair comprises a pair of image frames captured by a camera, and wherein the keypoint match corresponds to the keypoint that is present in each of the image frames in the view-overlapped keyframe pair.
11 . The non-transitory computer-readable storage medium of claim 9 , wherein the keypoint detection neural network comprises a convolutional neural network (CNN).
12 . The non-transitory computer-readable storage medium of claim 9 , wherein the keypoint reliability score is based on a comparison of coordinates of the keypoint in the final pose output generated by the VSLAM process to saved coordinates for a scene of the view-overlapped keyframe pair.
13 . The non-transitory computer-readable storage medium of claim 9 , wherein the operations further comprise:
regressing the inverse reliability score into a regressed inverse reliability score;
training a separate head of the keypoint detection neural network with the regressed inverse reliability score; and
combining the regressed inverse reliability score with the keypoint reliability score to obtain a final keypoint reliability score to utilize during an inference stage of the keypoint detection neural network.
14 . A method for facilitating learning reliable keypoints in situ with introspective self-supervision, the method comprising:
providing a view-overlapped keyframe pair from a pose graph that is generated by a visual simultaneous localization and mapping (VSLAM) process executed by one or more processors, wherein the view-overlapped keyframe pair comprises a matched pixel set (p, p′);
determining a keypoint match from the view-overlapped keyframe pair based on a keypoint detection and matching process, the keypoint match corresponding to a keypoint;
calculating an inverse reliability score based on matched pixels corresponding to the keypoint match in the view-overlapped keyframe pair, wherein the inverse reliability score comprises a pixel distance between p′ and an epipolar line for p;
identifying a supervision signal associated with the keypoint match, the supervision signal comprising a keypoint reliability score of the keypoint based on a final pose output of the VSLAM process; and
training a keypoint detection neural network using the keypoint match, the inverse reliability score, and the keypoint reliability score.
15 . The method of claim 14 , wherein the view-overlapped keyframe pair comprises a pair of image frames captured by a camera, and wherein the keypoint match corresponds to the keypoint that is present in each of the image frames in the view- overlapped keyframe pair.
16 . The method of claim 14 , wherein the keypoint detection neural network comprises a convolutional neural network (CNN).
17 . The method of claim 14 , wherein the keypoint reliability score is based on a comparison of coordinates of the keypoint in the final pose output generated by the VSLAM process to saved coordinates for a scene of the view-overlapped keyframe pair.