Methods and apparatuses for corner detection using neural network and corner detector
An apparatus configured to be head-worn by a user, includes: a screen configured to present graphics for the user; a camera system configured to view an environment in which the user is located; and a processing unit coupled to the camera system, the processing unit configured to: obtain locations of features for an image of the environment, wherein the locations of the features are identified by a neural network; determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and perform a corner detection using a corner detection algorithm to identify a corner in the region of interest.
1. An apparatus configured to be head-worn by a user, comprising:
a screen configured to present graphics for the user;
a camera system configured to view an environment in which the user is located; and
a processing unit coupled to the camera system, the processing unit configured to:
obtain locations of features for image data associated with the environment, wherein the locations of the features are identified by a neural network;
determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and
perform a corner detection using a corner detection algorithm to identify a corner in the region of interest,
wherein the image data includes at least one low resolution image that is reduced in resolution from at least one high resolution image generated by the camera system.
2. The apparatus of claim 1 , wherein the processing unit is configured to determine the region of interest as having a position that is based on at least one of the locations identified by the neural network, and wherein the position is with respect to the image.
3. The apparatus of claim 1 , wherein the image data is associated with at least one image that is generated by the camera system and transmitted to the neural network.
4. The apparatus of claim 1 , wherein the neural network is in a module of the apparatus.
5. The apparatus of claim 1 , wherein the neural network is implemented in one or more computing devices that are remote from the apparatus.
6. The apparatus of claim 1 , wherein the neural network has machine learning capability.
7. The apparatus of claim 1 , wherein the processing unit is configured to obtain the locations of the features by obtaining a heatmap generated by the neural network, the heatmap indicating the locations of the features.
8. The apparatus of claim 1 , wherein the region of interest comprises a N×N patch, and the processing unit is configured to perform the corner detection on the N×N patch, and wherein N is an integer larger than 1.
9. The apparatus of claim 1 , wherein the region of interest comprises a patch having 144 pixels or less, and the processing unit is configured to perform the corner detection on the patch.
10. The apparatus of claim 1 , wherein the region of interest comprises a 8×8 patch, and the processing unit is configured to perform the corner detection on the 8×8 patch.
11. The apparatus of claim 1 , wherein the processing unit is also configured to convert the image with the first resolution to the other image with the second resolution.
12. The apparatus of claim 1 , wherein the first resolution comprises a VGA resolution.
13. The apparatus of claim 1 , wherein the second resolution comprises a QVGA resolution.
14. The apparatus of claim 1 , further comprising the neural network.
15. The apparatus of claim 14 , wherein the neural network has been trained using a reference dataset.
16. The apparatus of claim 14 , wherein the neural network comprises a convolutional neural network.
17. The apparatus of claim 14 , wherein the neural network is configured to compute interest point locations and descriptors.
18. The apparatus of claim 14 , wherein the neural network comprises an encoder configured to down sample an input image spatially.
19. The apparatus of claim 18 , wherein the neural network also comprises:
an interest point decoder configured to operate on an encoder output from the encoder, and produce a score for each pixel in the input image; and
a descriptor decoder configured to operate on the encoder output, upsample the encoder output to a higher resolution, and produce a vector for each pixel in the input image.
20. The apparatus of claim 14 , wherein the neural network is configured to use homographic adaptation for improving geometric consistency of an interest point detector.
21. The apparatus of claim 20 , wherein the neural network comprises a convolutional neural network configured to train the interest point detector.
22. The apparatus of claim 20 , wherein the neural network is configured to perform image warping to create one or more warped images in the homographic adaptation.
23. The apparatus of claim 1 , wherein the neural network is a part of the processing unit.
24. The apparatus of claim 1 , further comprising the neural network, the neural network being communicatively coupled to the processing unit.
25. An apparatus configured to be head-worn by a user, comprising:
a screen configured to present graphics for the user;
a camera system configured to view an environment in which the user is located; and
a processing unit coupled to the camera system, the processing unit configured to:
obtain locations of features for image data associated with the environment, wherein the locations of the features are identified by a neural network;
determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and
perform a corner detection using a corner detection algorithm to identify a corner in the region of interest,
wherein the processing unit is configured to determine a position of the corner in the image based at least in part on a position of the corner in the region of interest.
26. An apparatus configured to be head-worn by a user, comprising:
a screen configured to present graphics for the user;
a camera system configured to view an environment in which the user is located; and
a processing unit coupled to the camera system, the processing unit configured to:
obtain locations of features for image data associated with the environment, wherein the locations of the features are identified by a neural network;
determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and
perform a corner detection using a corner detection algorithm to identify a corner in the region of interest,
wherein the processing unit is configured to determine a score for each pixel in the region of interest.
27. A method performed by a head-worn image display device, comprising:
obtaining locations of features in an image, wherein the locations of the features are identified by a neural network;
determining a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and
performing a corner detection using a corner detection algorithm to identify a corner in the region of interest,
wherein the image has a first resolution, and wherein the locations of the features are identified by the neural network based on another image having a second resolution that is less than the first resolution.
28. The method of claim 27 , wherein the region of interest is determined as having a position that is based on at least one of the locations identified by the neural network, and wherein the position is with respect to the image.
29. The method of claim 27 , further comprising
generating the image; and
transmitting the image to the neural network.
30. The method of claim 27 , wherein the neural network is in a module of the head-worn image display device.
31. The method of claim 27 , wherein the neural network is implemented in one or more computing devices that are remote from the head-worn image display device.
32. The method of claim 27 , wherein the neural network has machine learning capability.
33. The method of claim 27 , wherein the locations of the features are obtained by receiving a heatmap from the neural network, wherein the heatmap indicates the locations of the features.
34. The method of claim 27 , wherein the region of interest comprises a N×N patch, and the corner detection is performed on the N×N patch, and wherein N is an integer larger than 1.
35. The method of claim 27 , wherein the region of interest comprises a patch having 144 pixels or less, and the corner detection is performed on the patch.
36. The method of claim 27 , wherein the region of interest comprises a 8×8 patch, and the corner detection is performed on the 8×8 patch.
37. The method of claim 27 , further comprising converting the image with the first resolution to the other image with the second resolution.
38. The method of claim 27 , wherein the first resolution comprises a VGA resolution.
39. The method of claim 29 , wherein the second resolution comprises a QVGA resolution.
40. The method of claim 27 , further comprising the neural network.
41. The method of claim 40 , wherein the neural network has been trained using a reference dataset.
42. The method of claim 40 , wherein the neural network comprises a convolutional neural network.
43. The method of claim 40 , wherein the neural network is configured to compute interest point locations and descriptors.
44. The method of claim 40 , wherein the neural network comprises an encoder configured to down sample an input image spatially.
45. The method of claim 44 , wherein the neural network also comprises:
an interest point decoder configured to operate on an encoder output from the encoder, and produce a score for each pixel in the input image; and
a descriptor decoder configured to operate on the encoder output, upsample the encoder output to a higher resolution, and produce a vector for each pixel in the input image.
46. The method of claim 40 , wherein the neural network is configured to use homographic adaptation for improving geometric consistency of an interest point detector.
47. The method of claim 46 , wherein the neural network comprises a convolutional neural network, and wherein the interest point detector is trained with the convolutional neural network.
48. The method of claim 46 , wherein the neural network is configured to perform image warping to create one or more warped images in the homographic adaptation.
49. A method performed by a head-worn image display device, comprising:
obtaining locations of features in an image, wherein the locations of the features are identified by a neural network;
determining a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image;
performing a corner detection using a corner detection algorithm to identify a corner in the region of interest; and
determining a position of the corner in the image based at least in part on a position of the corner in the region of interest.
50. A method performed by a head-worn image display device, comprising:
obtaining locations of features in an image, wherein the locations of the features are identified by a neural network;
determining a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image;
performing a corner detection using a corner detection algorithm to identify a corner in the region of interest; and
determining a score for each pixel in the region of interest.