Method and system for performing simultaneous localization and mapping using convolutional image transformation
View Patent ↗Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.
1. A method for computing a homography based on two images, the method comprising:
receiving a first image based on a first camera pose and a second image based on a second camera pose;
generating a first 2D point cloud based on the first image and a second 2D point cloud based on the second image;
providing the first 2D point cloud and the second 2D point cloud to a neural network; and
generating, by the neural network, the homography based on the first 2D point cloud and the second 2D point cloud, the neural network having been previously trained by:
generating a 3D point cloud containing a plurality of points;
obtaining two camera poses from which the plurality of points are at least partially viewable, the two camera poses being different from the first camera pose and the second camera pose;
projecting the plurality of points onto two 2D planes using the two camera poses to generate two 2D point clouds;
generating, by the neural network, a particular homography based on the two 2D point clouds;
determining a ground-truth homography based on the two camera poses; and
modifying the neural network using the particular homography and the ground-truth homography.
2. The method of claim 1 , wherein the first image was captured by a first camera at a first instant in time, and wherein the second image was captured by the first camera at a second instant in time after the first instant in time.
3. The method of claim 1 , wherein the first 2D point cloud and the second 2D point cloud are generated using a first neural network, and wherein the neural network is a second neural network.
4. The method of claim 1 , the neural network having been previously trained by:
determining a 3D trajectory within a threshold distance of the plurality of points; and
sampling the 3D trajectory to obtain the two camera poses.
5. The method of claim 1 , wherein the 3D point cloud is generated by sampling one or more geometries.
6. The method of claim 1 , wherein the two camera poses have at least 30% overlap.
7. An augmented reality (AR) device comprising:
a camera; and
a processor communicatively coupled to the camera and configured to perform operations including:
receiving, from the camera, a first image based on a first camera pose and a second image based on a second camera pose;
generating a first 2D point cloud based on the first image and a second 2D point cloud based on the second image;
providing the first 2D point cloud and the second 2D point cloud to a neural network; and
generating, by the neural network, a homography based on the first 2D point cloud and the second 2D point cloud, the neural network having been previously trained by:
generating a 3D point cloud containing a plurality of points;
obtaining two camera poses from which the plurality of points are at least partially viewable, the two camera poses being different from the first camera pose and the second camera pose;
projecting the plurality of points onto two 2D planes using the two camera poses to generate two 2D point clouds;
generating, by the neural network, a particular homography based on the two 2D point clouds;
determining a ground-truth homography based on the two camera poses; and
modifying the neural network using the particular homography and the ground-truth homography.
8. The AR device of claim 7 , wherein the first 2D point cloud and the second 2D point cloud are generated using a first neural network, and wherein the neural network is a second neural network.
9. The AR device of claim 7 , the neural network having been previously trained by:
determining a 3D trajectory within a threshold distance of the plurality of points; and
sampling the 3D trajectory to obtain the two camera poses.
10. The AR device of claim 7 , wherein the 3D point cloud is generated by sampling one or more geometries.
11. The AR device of claim 7 , wherein the two camera poses have at least 30% overlap.
12. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving a first image based on a first camera pose and a second image based on a second camera pose;
generating a first 2D point cloud based on the first image and a second 2D point cloud based on the second image;
providing the first 2D point cloud and the second 2D point cloud to a neural network; and
generating, by the neural network, a homography based on the first 2D point cloud and the second 2D point cloud, the neural network having been previously trained by:
generating a 3D point cloud containing a plurality of points;
obtaining two camera poses from which the plurality of points are at least partially viewable, the two camera poses being different from the first camera pose and the second camera pose;
projecting the plurality of points onto two 2D planes using the two camera poses to generate two 2D point clouds;
generating, by the neural network, a particular homography based on the two 2D point clouds;
determining a ground-truth homography based on the two camera poses; and
modifying the neural network using the particular homography and the ground-truth homography.
13. The non-transitory computer-readable medium of claim 12 , wherein the first image was captured by a first camera at a first instant in time, and wherein the second image was captured by the first camera at a second instant in time after the first instant in time.
14. The non-transitory computer-readable medium of claim 12 , wherein the first 2D point cloud and the second 2D point cloud are generated using a first neural network, and wherein the neural network is a second neural network.
15. The non-transitory computer-readable medium of claim 12 , the neural network having been previously trained by:
determining a 3D trajectory within a threshold distance of the plurality of points; and
sampling the 3D trajectory to obtain the two camera poses.
16. The non-transitory computer-readable medium of claim 12 , wherein the 3D point cloud is generated by sampling one or more geometries.
17. The non-transitory computer-readable medium of claim 12 , wherein the two camera poses have at least 30% overlap.