Method and system for performing simultaneous localization and mapping using convolutional image transformation
Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.
1. A method for training a neural network to compute a homography, the method comprising:
generating a 3D point cloud containing a plurality of points;
obtaining two camera poses from which the plurality of points are at least partially viewable;
projecting the plurality of points onto two 2D planes using the two camera poses to generate two 2D point clouds;
generating, by the neural network, a particular homography based on the two 2D point clouds;
determining a ground-truth homography based on the two camera poses;
comparing the particular homography to the ground-truth homography to generate an error; and
modifying the neural network based on the error.
2. The method of claim 1 , wherein the 3D point cloud is generated by sampling one or more geometries.
3. The method of claim 2 , wherein the one or more geometries include a plane, a sphere, or a cube.
4. The method of claim 1 , further comprising:
determining a 3D trajectory within a threshold distance of the plurality of points; and
sampling the 3D trajectory to obtain the two camera poses.
5. The method of claim 4 , wherein the 3D trajectory is sampled such that at least 75% of the plurality of points are viewable by each of the two camera poses.
6. The method of claim 1 , wherein determining the ground-truth homography based on the two camera poses includes:
determining a relative rotation and a relative translation between the two camera poses.
7. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
generating a 3D point cloud containing a plurality of points;
obtaining two camera poses from which the plurality of points are at least partially viewable;
projecting the plurality of points onto two 2D planes using the two camera poses to generate two 2D point clouds;
generating, by a neural network, a particular homography based on the two 2D point clouds;
determining a ground-truth homography based on the two camera poses;
comparing the particular homography to the ground-truth homography to generate an error; and
modifying the neural network based on the error.
8. The non-transitory computer-readable medium of claim 7 , wherein the 3D point cloud is generated by sampling one or more geometries.
9. The non-transitory computer-readable medium of claim 8 , wherein the one or more geometries include a plane, a sphere, or a cube.
10. The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise:
determining a 3D trajectory within a threshold distance of the plurality of points; and
sampling the 3D trajectory to obtain the two camera poses.
11. The non-transitory computer-readable medium of claim 10 , wherein the 3D trajectory is sampled such that at least 75% of the plurality of points are viewable by each of the two camera poses.
12. The non-transitory computer-readable medium of claim 7 , wherein determining the ground-truth homography based on the two camera poses includes:
determining a relative rotation and a relative translation between the two camera poses.
13. A system comprising:
one or more processors; and
a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
generating a 3D point cloud containing a plurality of points;
obtaining two camera poses from which the plurality of points are at least partially viewable;
projecting the plurality of points onto two 2D planes using the two camera poses to generate two 2D point clouds;
generating, by a neural network, a particular homography based on the two 2D point clouds;
determining a ground-truth homography based on the two camera poses;
comparing the particular homography to the ground-truth homography to generate an error; and
modifying the neural network based on the error.
14. The system of claim 13 , wherein the 3D point cloud is generated by sampling one or more geometries.
15. The system of claim 13 , wherein the operations further comprise:
determining a 3D trajectory within a threshold distance of the plurality of points; and
sampling the 3D trajectory to obtain the two camera poses.
16. The system of claim 15 , wherein the 3D trajectory is sampled such that at least 75% of the plurality of points are viewable by each of the two camera poses.
17. The system of claim 13 , wherein determining the ground-truth homography based on the two camera poses includes:
determining a relative rotation and a relative translation between the two camera poses.