Learning to fuse geometrical and CNN relative camera pose via uncertainty
A computer-implemented method for fusing geometrical and Convolutional Neural Network (CNN) relative camera pose is provided. The method includes receiving two images having different camera poses. The method further includes inputting the two images into a geometric solver branch to return, as a first solution, an estimated camera pose and an associated pose uncertainty value determined from a Jacobian of a reproduction error function. The method also includes inputting the two images into a CNN branch to return, as a second solution, a predicted camera pose and an associated pose uncertainty value. The method additionally includes fusing, by a processor device, the first solution and the second solution in a probabilistic manner using Bayes' rule to obtain a fused pose.
1. A computer-implemented method for fusing geometrical and Convolutional Neural Network (CNN) relative camera pose, comprising:
receiving two images having different camera poses;
inputting the two images into a geometric solver branch to return, as a first solution, an estimated camera pose and an associated pose uncertainty value determined from a Jacobian of a reproduction error function;
inputting the two images into a CNN branch, having a pose branch multi-layer perceptron (MLP) and an uncertainty branch MLP, to return, as a second solution, a predicted camera pose by the pose branch MLP and an associated pose uncertainty value predicted by the uncertainty branch MLP by extracting appearance features with a feature extractor and geometric features with an attentional graph neural network and a geometric feature MLP; and
fusing, by a processor device, the first solution and the second solution in a probabilistic manner based on the uncertainty predictions using Bayes' rule to obtain a fused pose.
2. The computer-implemented method of claim 1 , wherein the geometric solver branch comprises a correspondence point minimal solver.
3. The computer-implemented method of claim 1 , wherein the geometric solver branch performs a Bundle Adjustment process to nonlinearly minimize a reprojection error.
4. The computer-implemented method of claim 1 , wherein the CNN branch parameterizes a translation direction of the predicted camera pose by an azimuth angle and an elevation angle.
5. The computer-implemented method of claim 1 , wherein the predicted camera pose and the pose uncertainty value are interpreted as a mean and a variance, respectively, of an underlying Gaussian distribution.
6. The computer-implemented method of claim 5 , further comprising concatenating the appearance and the geometric features extracted from the two images prior to predicting the mean and inverse variance of the underlying Gaussian distribution.
7. The computer-implemented method of claim 1 , wherein said fusing step uses circular fusion based on parameters of a circle.
8. The computer-implemented method of claim 1 , wherein said fusing step comprising performing a weighted averaging of the first solution and the second solution based on parameters of a circle.
9. The computer-implemented method of claim 1 , further comprising performing robotic movement control using the fused pose.
10. A computer program product for fusing geometrical and Convolutional Neural Network (CNN) relative camera pose, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
receiving, by a processor device of the computer, two images having different camera poses;
inputting, by the processor device, the two images into a geometric solver branch to return, as a first solution, an estimated camera pose and an associated pose uncertainty value determined from a Jacobian of a reproduction error function;
inputting, by the processor device, the two images into a CNN branch, having a pose branch multi-layer perceptron (MLP) and an uncertainty branch MLP, to return, as a second solution, a predicted camera pose by the pose branch MLP and an associated pose uncertainty value predicted by the uncertainty branch MLP by extracting appearance features with a feature extractor and geometric features with an attentional graph neural network and a geometric feature MLP; and
fusing, by the processor device, the first solution and the second solution in a probabilistic manner based on the uncertainty predictions using Bayes' rule to obtain a fused pose.
11. The computer program product of claim 10 , wherein the geometric solver branch comprises a correspondence point minimal solver.
12. The computer program product of claim 10 , wherein the geometric solver branch performs a Bundle Adjustment process to nonlinearly minimize a reprojection error.
13. The computer program product of claim 10 , wherein the CNN branch parameterizes a translation direction of the predicted camera pose by an azimuth angle and an elevation angle.
14. The computer program product of claim 13 , further comprising concatenating the appearance and the geometric features extracted from the two images prior to predicting the mean and inverse variance of the underlying Gaussian distribution.
15. The computer program product of claim 10 , wherein the predicted camera pose and the pose uncertainty value are interpreted as a mean and a variance, respectively, of an underlying Gaussian distribution.
16. The computer program product of claim 10 , wherein said fusing step uses circular fusion based on parameters of a circle.
17. The computer program product of claim 10 , wherein said fusing step comprising performing a weighted averaging of the first solution and the second solution based on parameters of a circle.
18. The computer program product of claim 10 , wherein the method further comprises performing robotic movement control using the fused pose.
19. A computer processing system for fusing geometrical and Convolutional Neural Network (CNN) relative camera pose, comprising:
a memory device for storing program code; and
a processor device operatively coupled to the memory device for running the program code to:
receive two images having different camera poses;
input the two images into a geometric solver branch to return, as a first solution, an estimated camera pose and an associated pose uncertainty value determined from a Jacobian of a reproduction error function;
input the two images into a CNN branch, having a pose branch multi-layer perceptron (MLP) and an uncertainty branch MLP, to return, as a second solution, a predicted camera pose by the pose branch MLP and an associated pose uncertainty value predicted by the uncertainty branch MLP by extracting appearance features with a feature extractor and geometric features with an attentional graph neural network and a geometric feature MLP; and
fuse the first solution and the second solution in a probabilistic manner based on the uncertainty predictions using Bayes' rule to obtain a fused pose.
20. The computer processing system of claim 19 , wherein the geometric solver branch comprises a correspondence point minimal solver.