Fully convolutional interest point detection and description via homographic adaptation
Systems, devices, and methods for training a neural network and performing image interest point detection and description using the neural network. The neural network may include an interest point detector subnetwork and a descriptor subnetwork. An optical device may include at least one camera for capturing a first image and a second image. A first set of interest points and a first descriptor may be calculated using the neural network based on the first image, and a second set of interest points and a second descriptor may be calculated using the neural network based on the second image. A homography between the first image and the second image may be determined based on the first and second sets of interest points and the first and second descriptors. The optical device may adjust virtual image light being projected onto an eyepiece based on the homography.
1. A computer-implemented method comprising:
obtaining a first image;
obtaining a second image;
calculating, by a neural network receiving the first image as input, a first set of calculated interest points and a first calculated descriptor;
calculating, by the neural network receiving the second image as input, a second set of calculated interest points and a second calculated descriptor; and
determining a homography between the first image and the second image based on the first and second sets of calculated interest points and the first and second calculated descriptors, wherein the homography represents a rotation and/or a translation between the first image and the second image;
wherein the neural network includes:
an interest point detector subnetwork configured to calculate the first set of calculated interest points and the second set of calculated interest points; and
a descriptor subnetwork configured to calculate the first calculated descriptor and the second calculated descriptor, wherein the descriptor subnetwork shares one or more layers with the interest point detector subnetwork.
2. The computer-implemented method of claim 1 , further comprising:
determining point correspondences between the first and second sets of calculated interest points, wherein the homography is determined based on the point correspondences.
3. The computer-implemented method of claim 1 , wherein the first image has dimensions of H×W, and wherein the second image has dimensions of H×W.
4. The computer-implemented method of claim 3 , wherein the first set of calculated interest points is represented by a first two-dimensional (2D) map having dimensions of H×W, and wherein the second set of calculated interest points is represented by a second 2D map having dimensions of H×W.
5. The computer-implemented method of claim 4 , wherein the first calculated descriptor has dimensions of H×W×D, and wherein the second calculated descriptor has dimensions of H×W×D.
6. The computer-implemented method of claim 1 , wherein the interest point detector subnetwork is configured to calculate the first set of calculated interest points concurrently with the descriptor subnetwork calculating the first calculated descriptor.
7. The computer-implemented method of claim 6 , wherein the interest point detector subnetwork is configured to calculate the second set of calculated interest points concurrently with the descriptor subnetwork calculating the second calculated descriptor.
8. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
obtaining a first image;
obtaining a second image;
calculating, by a neural network receiving the first image as input, a first set of calculated interest points and a first calculated descriptor;
calculating, by the neural network receiving the second image as input, a second set of calculated interest points and a second calculated descriptor; and
determining a homography between the first image and the second image based on the first and second sets of calculated interest points and the first and second calculated descriptors, wherein the homography represents a rotation and/or a translation between the first image and the second image;
wherein the neural network includes:
an interest point detector subnetwork configured to calculate the first set of calculated interest points and the second set of calculated interest points; and
a descriptor subnetwork configured to calculate the first calculated descriptor and the second calculated descriptor, wherein the descriptor subnetwork shares one or more layers with the interest point detector subnetwork.
9. The non-transitory computer-readable medium of claim 8 , wherein the operations further comprise:
determining point correspondences between the first and second sets of calculated interest points, wherein the homography is determined based on the point correspondences.
10. The non-transitory computer-readable medium of claim 8 , wherein the first image has dimensions of H×W, and wherein the second image has dimensions of H×W.
11. The non-transitory computer-readable medium of claim 10 , wherein the first set of calculated interest points is represented by a first two-dimensional (2D) map having dimensions of H×W, and wherein the second set of calculated interest points is represented by a second 2D map having dimensions of H×W.
12. The non-transitory computer-readable medium of claim 11 , wherein the first calculated descriptor has dimensions of H×W×D, and wherein the second calculated descriptor has dimensions of H×W×D.
13. The non-transitory computer-readable medium of claim 8 , wherein the interest point detector subnetwork is configured to calculate the first set of calculated interest points concurrently with the descriptor subnetwork calculating the first calculated descriptor.
14. The non-transitory computer-readable medium of claim 13 , wherein the interest point detector subnetwork is configured to calculate the second set of calculated interest points concurrently with the descriptor subnetwork calculating the second calculated descriptor.
15. A system comprising:
one or more processors; and
a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
obtaining a first image;
obtaining a second image;
calculating, by a neural network receiving the first image as input, a first set of calculated interest points and a first calculated descriptor;
calculating, by the neural network receiving the second image as input, a second set of calculated interest points and a second calculated descriptor; and
determining a homography between the first image and the second image based on the first and second sets of calculated interest points and the first and second calculated descriptors, wherein the homography represents a rotation and/or a translation between the first image and the second image;
wherein the neural network includes:
an interest point detector subnetwork configured to calculate the first set of calculated interest points and the second set of calculated interest points; and
a descriptor subnetwork configured to calculate the first calculated descriptor and the second calculated descriptor, wherein the descriptor subnetwork shares one or more layers with the interest point detector subnetwork.
16. The system of claim 15 , wherein the operations further comprise:
determining point correspondences between the first and second sets of calculated interest points, wherein the homography is determined based on the point correspondences.
17. The system of claim 15 , wherein the first image has dimensions of H×W, and wherein the second image has dimensions of H×W.
18. The system of claim 17 , wherein the first set of calculated interest points is represented by a first two-dimensional (2D) map having dimensions of H×W, and wherein the second set of calculated interest points is represented by a second 2D map having dimensions of H×W.
19. The system of claim 18 , wherein the first calculated descriptor has dimensions of H×W×D, and wherein the second calculated descriptor has dimensions of H×W×D.
20. The system of claim 15 , wherein the interest point detector subnetwork is configured to calculate the first set of calculated interest points concurrently with the descriptor subnetwork calculating the first calculated descriptor.