Learning based camera pose estimation from images of an environment
A deep neural network (DNN) system learns a map representation for estimating a camera position and orientation (pose). The DNN is trained to learn a map representation corresponding to the environment, defining positions and attributes of structures, trees, walls, vehicles, etc. The DNN system learns a map representation that is versatile and performs well for many different environments (indoor, outdoor, natural, synthetic, etc.). The DNN system receives images of an environment captured by a camera (observations) and outputs an estimated camera pose within the environment. The estimated camera pose is used to perform camera localization, i.e., recover the three-dimensional (3D) position and orientation of a moving camera, which is a fundamental task in computer vision with a wide variety of applications in robot navigation, car localization for autonomous driving, device localization for mobile navigation, and augmented/virtual reality.
1. A computer-implemented method, comprising:
receiving an input image at a deep neural network DNN, wherein weights of the DNN define a map representation of an environment and the weights are determined during training using a labeled training dataset including images and corresponding absolute camera poses and relative camera poses; and
applying, by the DNN, the weights to the input image to generate an estimated camera pose for capturing the environment to produce the input image.
2. The computer-implemented method of claim 1 , wherein, during the training, image pairs are input to the DNN and corresponding estimated camera pose pairs are generated by the DNN.
3. The computer-implemented method of claim 2 , wherein an image pair includes a first image and an additional image in an image sequence, and one or more intervening images may occur between the first image and the additional image.
4. The computer-implemented method of claim 2 , wherein, during the training, relative estimated camera poses are computed for each estimated camera pose pair.
5. The computer-implemented method of claim 4 , wherein, during the training, the weights are modified to simultaneously reduce differences between the relative estimated camera poses and the relative camera poses included in the training dataset and differences between estimated camera poses generated by the DNN and the absolute camera poses included in the training dataset.
6. The computer-implemented method of claim 1 , wherein a rotation portion of the estimated camera pose is parameterized as a three-dimensional logarithm of a unit quaternion.
7. The computer-implemented method of claim 1 , further comprising:
receiving visual odometry data corresponding to the input image; and
modifying the weights of the DNN to minimize differences between the visual odometry data and a relative camera pose computed using the estimated camera pose and an additional estimated camera pose generated by the DNN.
8. The computer-implemented method of claim 1 , further comprising:
receiving global position sensor data corresponding to the input image; and
modifying the weights to minimize differences between the global position sensor data and the estimated camera pose.
9. The computer-implemented method of claim 1 , further comprising:
receiving inertial measurement data corresponding to the input image; and
modifying the weights to minimize differences between the inertial measurement data and the estimated camera pose.
10. The computer-implemented method of claim 1 , further comprising post-processing the estimated camera pose using pose graph optimization PGO, to produce a refined camera pose.
11. The computer-implemented method of claim 1 , wherein the DNN comprises at least a convolutional neural network layer, followed by a global average pooling layer, followed by a fully-connected layer to output the estimated camera pose.
12. A system, comprising:
a deep neural network DNN, configured to:
receive an input image, wherein weights of the DNN define a map representation of an environment and the weights are determined during training using a labeled training dataset including images and corresponding absolute camera poses and relative camera poses; and
apply the weights to the input image to generate an estimated camera pose for capturing the environment to produce the input image.
13. The system of claim 12 , wherein, during the training, image pairs are input to the DNN and corresponding estimated camera pose pairs are generated by the DNN.
14. The system of claim 13 , wherein an image pair includes a first image and an additional image in an image sequence, and one or more intervening images may occur between the first image and the additional image.
15. The system of claim 13 , further comprising a relative pose computation unit configured to compute relative estimated camera poses for each estimated camera pose pair during the training.
16. The system of claim 15 , further comprising a training loss unit configured to modify the weights during the training to simultaneously reduce differences between the relative estimated camera poses and the relative camera poses included in the training dataset and differences between estimated camera poses generated by the DNN and the absolute camera poses included in the training dataset.
17. The system of claim 12 , wherein a rotation portion of the estimated camera pose is parameterized as a three-dimensional logarithm of a unit quaternion.
18. The system of claim 12 , further comprising a training loss unit configured to:
receive visual odometry data corresponding to the input image; and
modify the weights of the DNN to minimize differences between the visual odometry data and a relative camera pose computed using the estimated camera pose and an additional estimated camera pose generated by the DNN.
19. The system of claim 12 , further comprising a pose graph optimization unit configured to post-process the estimated camera pose using pose graph optimization PGO, to produce a refined camera pose.
20. A non-transitory computer-readable media storing computer instructions for estimating camera poses that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving an input image at a deep neural network DNN, wherein weights of the DNN define a map representation of an environment and the weights are determined during training using a labeled training dataset including images and corresponding absolute camera poses and relative camera poses; and
applying, by the DNN, the weights to the input image to generate an estimated camera pose for capturing the environment to produce the input image.