Keypoint detection with trackability measurements
Disclosed embodiments facilitate keypoint selection in part by assigning a similarity score to each candidate keypoint being considered for selection. The similarity score may be based on the maximum measured similarity of an image patch associated with a keypoint in relation to an image patch in a local image section in a region around the image patch. A subset of the candidate keypoints with the lowest similarity scores may be selected and used to detect and/or track objects in subsequent images and/or to determine camera pose.
1. A method comprising:
determining a set of candidate keypoints based on a first image captured with a camera;
determining, for each candidate keypoint in the set of candidate keypoints, a corresponding first similarity score, wherein the first similarity score corresponding to each candidate keypoint is determined, in part, by comparing an image patch associated with the corresponding candidate keypoint to a plurality of corresponding image sections in the first image in a region around the image patch; and
selecting a first subset of the candidate keypoints, wherein the first subset comprises a predefined number of candidate keypoints with lowest similarity scores.
2. The method of claim 1 , wherein determining the first similarity score corresponding to each candidate keypoint comprises:
determining a plurality of second similarity scores corresponding to each candidate keypoint, wherein each second similarity score is determined from a comparison of the image patch associated with the corresponding candidate keypoint with one of the plurality of corresponding image sections in the region around the image patch, and wherein each second similarity score is indicative of a similarity between the image patch associated with the corresponding candidate keypoint and the corresponding image section; and
selecting, as the first similarity score, a maximum of the second similarity scores for that candidate keypoint.
3. The method of claim 2 , wherein determining the plurality of second similarity scores corresponding to each candidate keypoint comprises:
computing each of the plurality of second similarity scores based, at least in part, on a measure of cross-correlation between the image patch associated with the candidate keypoint and the corresponding image section.
4. The method of claim 3 , wherein the measure of cross-correlation is determined based, in part, by performing cross-correlation based on a cross-correlation method selected from one of:
Normalized Cross-Correlation; or
Sum of Squared Differences; or
Sum of Absolute Differences.
5. The method of claim 4 , further comprising:
tracking, based on the selected cross-correlation, an object in at least one second image captured by the camera, based, in part, on the selected first subset of candidate keypoints.
6. The method of claim 1 , wherein the image patch associated with the candidate keypoint is represented using an image descriptor.
7. The method of claim 6 , wherein the image descriptor comprises one of:
Scale Invariant Feature Transform (SIFT) or variants thereof; or
Speeded-Up Robust Features (“SURF”), or variants thereof.
8. The method of claim 6 , wherein the first similarity score corresponding to the candidate keypoint is determined based, in part, on a similarity of the image descriptor associated with the image patch to a plurality of image descriptors in the region around the image patch, wherein each of the plurality of image descriptors corresponds to a distinct image section in the region around the image patch.
9. The method of claim 1 , further comprising:
tracking one or more objects in at least one second image captured by the camera, based, in part, on the first subset of candidate keypoints.
10. The method of claim 1 , wherein determining the set of candidate keypoints based on the first image comprises:
obtaining, from the first image, a hierarchy of images, wherein each image in the hierarchy of images has a corresponding image resolution; and
determining a plurality of second subsets of candidate keypoints, wherein each second subset of candidate keypoints is associated with a distinct image in the hierarchy of images, and wherein, the set of candidate keypoints comprises the plurality of second subsets of candidate keypoints.
11. The method of claim 10 , wherein obtaining, from the first image, a hierarchy of images comprises:
blurring one or more images in the hierarchy of images.
12. A Mobile Station (MS) comprising:
a camera to capture a plurality of images comprising a first image;
a memory to store the plurality of images; and
a processor coupled to the camera and the memory, wherein the processor is configured to
determine a set of candidate keypoints based on the first image captured with a camera;
determine, for each candidate keypoint in the set of candidate keypoints, a corresponding first similarity score, wherein the first similarity score corresponding to each candidate keypoint is determined, in part, by comparing an image patch associated with the corresponding candidate keypoint to a plurality of corresponding image sections in the first image in a region around the image patch; and
select a first subset of the candidate keypoints, wherein the first subset comprises a predefined number of candidate keypoints with lowest similarity scores.
13. The MS of claim 12 , wherein to determine the first similarity score corresponding to each candidate keypoint, the processor is configured to:
determine a plurality of second similarity scores corresponding to each candidate keypoint, wherein each second similarity score is determined from a comparison of the image patch associated with the corresponding candidate keypoint with one of the plurality of corresponding image sections in the region around the image patch, and wherein each second similarity score is indicative of a similarity between the image patch associated with the corresponding candidate keypoint and the corresponding image section; and
select, as the first similarity score, a maximum of the second similarity scores for that candidate keypoint.
14. The MS of claim 13 , wherein to determine the plurality of second similarity scores corresponding to each candidate keypoint, the processor is configured to:
compute each of the plurality of second similarity scores based, at least in part, on a measure of cross-correlation between the image patch associated with the candidate keypoint and the corresponding image section.
15. The MS of claim 14 , wherein to determine the measure of cross-correlation, the processor is configured to:
perform cross-correlation based on a cross-correlation method selected from one of:
Normalized Cross-Correlation; or
Sum of Squared Differences; or
Sum of Absolute Differences.
16. The MS of claim 15 , wherein the processor is further configured to:
track, based on the selected cross-correlation, an object in at least one second image captured by the camera, based, in part, on the selected first subset of candidate keypoints.
17. The MS of claim 12 , wherein the image patch associated with the candidate keypoint is represented using an image descriptor.
18. The MS of claim 17 , wherein the image descriptor comprises one of:
Scale Invariant Feature Transform (SIFT) or variants thereof; or
Speeded-Up Robust Features (“SURF”), or variants thereof.
19. The MS of claim 17 , wherein the first similarity score corresponding to the candidate keypoint is determined based, in part, on a similarity of the image descriptor associated with the image patch to a plurality of image descriptors in the region around the image patch, wherein each of the plurality of image descriptors corresponds to a distinct image section in the region around the image patch.
20. The MS of claim 12 , wherein to determine the set of candidate keypoints based on the first image, the processor is configured to:
obtain, from the first image, a hierarchy of images, wherein each image in the hierarchy of images has a corresponding image resolution; and
determine a plurality of second subsets of candidate keypoints, wherein each second subset of candidate keypoints is associated with a distinct image in the hierarchy of images, and wherein, the set of candidate keypoints comprises the plurality of second subsets of candidate keypoints.
21. The MS of claim 20 , wherein to obtain, from the first image, a hierarchy of images, the processor is configured to:
blur one or more images in the hierarchy of images.
22. The MS of claim 12 , wherein the processor is further configured to:
track an object in at least one second image captured by the camera, based, in part, on the first subset of candidate keypoints.
23. An apparatus comprising:
image sensing means to capture a plurality of images comprising a first image;
means for determining a set of candidate keypoints based on a first image captured by an image sensing means;
means for determining, for each candidate keypoint in the set of candidate keypoints, a corresponding first similarity score, wherein the first similarity score corresponding to each candidate keypoint is determined, in part, by comparing an image patch associated with the corresponding candidate keypoint to a plurality of corresponding image sections in the first image in a region around the image patch; and
means for selecting a first subset of the candidate keypoints, wherein the first subset comprises a predefined number of candidate keypoints with lowest similarity scores.
24. The apparatus of claim 23 , wherein means for determining the first similarity score corresponding to each candidate keypoint comprises:
means for determining a plurality of second similarity scores corresponding to each candidate keypoint, wherein each second similarity score is determined from a comparison of the image patch associated with the corresponding candidate keypoint with one of the plurality of corresponding image sections in the region around the image patch, and wherein each second similarity score is indicative of a similarity between the image patch associated with the corresponding candidate keypoint and the corresponding image section; and
means for selecting, as the first similarity score, a maximum of the second similarity scores for that candidate keypoint.
25. The apparatus of claim 23 , wherein means for determining the plurality of second similarity scores corresponding to each candidate keypoint comprises:
means for computing each of the plurality of second similarity scores based, at least in part, on a measure of cross-correlation between the image patch associated with the candidate keypoint and the corresponding image section, wherein the measure of cross-correlation is determined based, in part, by performing a cross-correlation selected from one of:
Normalized Cross-Correlation; or
Sum of Squared Differences; or
Sum of Absolute Differences.
26. A non-transitory computer-readable medium comprising instructions, which when executed by a processor, cause the processor to perform steps in a method comprising:
determining a set of candidate keypoints based on a first image captured with a camera;
determining, for each candidate keypoint in the set of candidate keypoints, a corresponding first similarity score, wherein the first similarity score corresponding to each candidate keypoint is determined, in part, by comparing an image patch associated with the corresponding candidate keypoint to a plurality of corresponding image sections in the first image in a region around the image patch; and
selecting a first subset of the candidate keypoints, wherein the first subset comprises a predefined number of candidate keypoints with lowest similarity scores.
27. The computer-readable medium of claim 26 , wherein determining the first similarity score corresponding to each candidate keypoint comprises:
determining a plurality of second similarity scores corresponding to each candidate keypoint, wherein each second similarity score is determined from a comparison of the image patch associated with the corresponding candidate keypoint with one of the plurality of corresponding image sections in the region around the image patch, and wherein each second similarity score is indicative of a similarity between the image patch associated with the corresponding candidate keypoint and the corresponding image section; and
selecting, as the first similarity score, a maximum of the second similarity scores for that candidate keypoint.
28. The computer-readable medium of claim 27 , wherein determining the plurality of second similarity scores corresponding to each candidate keypoint comprises:
computing each of the plurality of second similarity scores based, at least in part, on a measure of cross-correlation between the image patch associated with the candidate keypoint and the corresponding image section.
29. The computer-readable medium of claim 28 , wherein the measure of cross-correlation is determined based, in part, by performing a cross-correlation method selected from one of:
Normalized Cross-Correlation; or
Sum of Squared Differences; or
Sum of Absolute Differences.
30. The computer-readable medium of claim 26 , wherein determining the set of candidate keypoints based on the first image comprises:
obtaining, from the first image, a hierarchy of images, wherein each image in the hierarchy of images has a corresponding image resolution; and
determining a plurality of second subsets of candidate keypoints, wherein each second subset of candidate keypoints is associated with a distinct image in the hierarchy of images, and wherein, the set of candidate keypoints comprises the plurality of second subsets of candidate keypoints.