Methods and apparatuses for corner detection using neural network and corner detector

1. A method of training and using a neural network for image interest point detection, comprises:

generating a reference dataset comprising a plurality of reference sets, wherein each of the plurality of reference sets comprises:

an image, and

a set of reference interest points corresponding to the image; and

for each reference set of the plurality of reference sets:

generating a warped image by applying a homograph to the image,

generating a warped set of reference interest points by applying the homograph to the set of reference interest points,

calculating, by the neural network receiving the image as input, a set of calculated interest points and a calculated descriptor,

calculating, by the neural network receiving the warped image as input, a set of calculated warped interest points and a calculated warped descriptor,

calculating a loss based on the set of calculated interest points, the calculated descriptor, the set of calculated warped interest points, the calculated warped descriptor, the set of reference interest points, the warped set of reference interest points, and the homograph, and

modifying the neural network based on the loss; wherein the method further comprises: obtaining locations of features in an input image, wherein the locations of the features are identified by the neural network; determining a region of interest for one of the features in the input image, the region of interest having a size that is less than a size of the input image; and performing a corner detection using a corner detection algorithm to identify a corner in the region of interest.

2. The method of claim 1 , wherein the neural network comprises an interest point detector subnetwork and a descriptor subnetwork,

wherein the interest point detector subnetwork is configured to receive the image as input and calculate the set of calculated interest points based on the image, and

wherein the descriptor subnetwork is configured to receive the image as input and calculate the calculated descriptor based on the image.

3. The method of claim 1 , wherein modifying the neural network based on the loss comprises modifying one or both of the interest point detector subnetwork and the descriptor subnetwork based on the loss.

4. The method of claim 1 , further comprising prior to generating the reference dataset, training the interest point detector subnetwork using a synthetic dataset comprising a plurality of synthetic images and a plurality of sets of synthetic interest points,

wherein generating the reference dataset comprises generating the reference dataset using the interest point detector subnetwork.

5. The method of claim 1 , wherein generating the reference dataset comprises:

for each reference set of the plurality of reference sets:

obtaining the image from an unlabeled dataset comprising a plurality of unlabeled images,

generating a plurality of warped images by applying a plurality of homographs to the image,

calculating, by the neural network receiving the plurality of warped images as input, a plurality of sets of calculated warped interest points,

generating a plurality of sets of calculated interest points by applying a plurality of inverse homographs to the plurality of sets of calculated warped interest points, and

aggregating the plurality of sets of calculated interest points to obtain the set of reference interest points.

6. The method of claim 1 , wherein each of the plurality of reference sets further includes a reference descriptor corresponding to the image, and wherein generating the reference dataset includes:

for each reference set of the plurality of reference sets:

obtaining the image from an unlabeled dataset comprising a plurality of unlabeled images,

generating a plurality of warped images by applying a plurality of homographs to the image,

calculating, by the neural network receiving the plurality of warped images as input, a plurality of calculated warped descriptors,

generating a plurality of calculated descriptors by applying a plurality of inverse homographs to the plurality of calculated warped descriptors, and

aggregating the plurality of calculated descriptors to obtain the reference descriptor.

7. The method of claim 1 , wherein the set of reference interest points is a two-dimensional map having values corresponding to a probability that a particular pixel of the image has an interest point is located at the particular pixel.

8. A method comprising:

capturing a first image;

capturing a second image;

calculating, by a neural network receiving the first image as input, a first set of calculated interest points and a first calculated descriptor;

calculating, by the neural network receiving the second image as input, a second set of calculated interest points and a second calculated descriptor; and

determining a homograph between the first image and the second image based on the first and second sets of calculated interest points and the first and second calculated descriptors,

wherein the neural network includes: an interest point detector subnetwork configured to calculate the first set of calculated interest points and the second set of calculated interest points, and a descriptor subnetwork configured to calculate the first calculated descriptor and the second calculated descriptor, and

wherein the method further comprises:

obtaining locations of features in an input image, wherein the locations of the features are identified by the neural network;

determining a region of interest for one of the features in the input image, the region of interest having a size that is less than a size of the input image; and

performing a corner detection using a corner detection algorithm to identify a corner in the region of interest.

9. The method of claim 8 , wherein:

the interest point detector subnetwork is configured to calculate the first set of calculated interest points concurrently with the descriptor subnetwork calculating the first calculated descriptor; and

the interest point detector subnetwork is configured to calculate the second set of calculated interest points concurrently with the descriptor subnetwork calculating the second calculated descriptor.

10. The method of claim 8 , further comprising training the neural network by generating a reference dataset comprising a plurality of reference sets, wherein each of the plurality of reference sets includes an image and a set of reference interest points corresponding to the image; and

for each reference set of the plurality of reference sets:

generating a warped image by applying a homograph to the image,

generating a warped set of reference interest points by applying the homograph to the set of reference interest points,

calculating, by the neural network receiving the image as input, a set of calculated interest points and a calculated descriptor,

calculating, by the neural network receiving the warped image as input, a set of calculated warped interest points and a calculated warped descriptor,

calculating a loss based on the set of calculated interest points, the calculated descriptor, the set of calculated warped interest points, the calculated warped descriptor, the set of reference interest points, the warped set of reference interest points, and the homograph, and

modifying the neural network based on the loss.

11. The method of claim 10 , wherein modifying the neural network based on the loss includes modifying one or both of the interest point detector subnetwork and the descriptor subnetwork based on the loss.

12. The method of claim 10 , further comprising prior to generating the reference dataset, training the interest point detector subnetwork using a synthetic dataset including a plurality of synthetic images and a plurality of sets of synthetic interest points,

wherein generating the reference dataset includes generating the reference dataset using the interest point detector subnetwork.

13. The method of claim 10 , wherein generating the reference dataset includes:

for each reference set of the plurality of reference sets:

obtaining the image from an unlabeled dataset comprising a plurality of unlabeled images,

generating a plurality of warped images by applying a plurality of homographs to the image,

calculating, by the neural network receiving the plurality of warped images as input, a plurality of sets of calculated warped interest points,

generating a plurality of sets of calculated interest points by applying a plurality of inverse homographs to the plurality of sets of calculated warped interest points, and

aggregating the plurality of sets of calculated interest points to obtain the set of reference interest points.

14. The method of claim 10 , wherein each of the plurality of reference sets further includes a reference descriptor corresponding to the image, and wherein generating the reference dataset includes:

for each reference set of the plurality of reference sets:

obtaining the image from an unlabeled dataset comprising a plurality of unlabeled images,

generating a plurality of warped images by applying a plurality of homographs to the image,

calculating, by the neural network receiving the plurality of warped images as input, a plurality of calculated warped descriptors,

generating a plurality of calculated descriptors by applying a plurality of inverse homographs to the plurality of calculated warped descriptors, and

aggregating the plurality of calculated descriptors to obtain the reference descriptor.

15. An optical device comprising:

at least one camera configured to capture a first image and a second image; and

one or more processors coupled to the camera and configured to perform operations comprising:

receiving the first image and the second image from the at least one camera,

calculating, by a neural network using the first image as an input, a first set of calculated interest points and a first calculated descriptor,

calculating, by the neural network using the second image as an input, a second set of calculated interest points and a second calculated descriptor, and

determining a homograph between the first image and the second image based on the first and second sets of calculated interest points and the first and second calculated descriptors;

wherein the neural network includes:

an interest point detector subnetwork configured to calculate the first set of calculated interest points and the second set of calculated interest points\, and

a descriptor subnetwork configured to calculate the first calculated descriptor and the second calculated descriptor; and

wherein the one or more processors are configured to:

obtain locations of features in an input image, wherein the locations of the features are identified by the neural network,

determine a region of interest for one of the features in the input image, the region of interest having a size that is less than a size of the image, and

perform a corner detection using a corner detection algorithm to identify a corner in the region of interest.

16. The optical device of claim 15 , wherein:

the interest point detector subnetwork is configured to calculate the first set of calculated interest points concurrently with the descriptor subnetwork calculating the first calculated descriptor; and

the interest point detector subnetwork is configured to calculate the second set of calculated interest points concurrently with the descriptor subnetwork calculating the second calculated descriptor.

17. The optical device of claim 15 , wherein the neural network was previously trained by generating a reference dataset comprising a plurality of reference sets, wherein each of the plurality of reference sets includes an image and a set of reference interest points corresponding to the image; and

for each reference set of the plurality of reference sets:

generating a warped image by applying a homograph to the image,

generating a warped set of reference interest points by applying the homograph to the set of reference interest points,

calculating, by the neural network receiving the image as input, a set of calculated interest points and a calculated descriptor,

calculating, by the neural network receiving the warped image as input, a set of calculated warped interest points and a calculated warped descriptor,

calculating a loss based on the set of calculated interest points, the calculated descriptor, the set of calculated warped interest points, the calculated warped descriptor, the set of reference interest points, the warped set of reference interest points, and the homograph, and

modifying the neural network based on the loss.

18. The optical device of claim 17 , wherein modifying the neural network based on the loss includes modifying one or both of the interest point detector subnetwork and the descriptor subnetwork based on the loss.

19. The optical device of claim 17 , wherein generating the reference dataset includes:

for each reference set of the plurality of reference sets:

obtaining the image from an unlabeled dataset comprising a plurality of unlabeled images,

generating a plurality of warped images by applying a plurality of homographs to the image,

calculating, by the neural network receiving the plurality of warped images as input, a plurality of sets of calculated warped interest points,

generating a plurality of sets of calculated interest points by applying a plurality of inverse homographs to the plurality of sets of calculated warped interest points, and

aggregating the plurality of sets of calculated interest points to obtain the set of reference interest points.