Visual odometry using object priors
View Patent ↗Disclosed are techniques for more accurately estimating the pose of a camera used to capture a three-dimensional scene. Accuracy is enhanced by leveraging three-dimensional object priors extracted from a large-scale three-dimensional shape database. This allows existing feature matching techniques to be augmented by generic three-dimensional object priors, thereby providing robust information about object orientations across multiple images or frames. More specifically, the three-dimensional object priors provide a unit that is easier and more reliably tracked between images than a single feature point. By adding object pose estimates across images, drift is reduced and the resulting visual odometry techniques are more robust and accurate. This eliminates the need for three-dimensional object templates that are specifically generated for the imaged object, training data obtained for a specific environment, and other tedious preprocessing steps. Entire object classes identified in a three-dimensional shape database can be used to train an object detector.
1. A visual odometry method comprising:
receiving source imagery that comprises a plurality of two-dimensional images of a three-dimensional scene, wherein each of the two-dimensional images was captured from a corresponding camera pose; and
estimating a camera trajectory in the three-dimensional scene, the camera trajectory representing the plurality of camera poses from which the corresponding plurality of two-dimensional images were captured, wherein the camera trajectory is estimated based on both (a) feature point correspondences detected in a pair of the two-dimensional images, and (b) an object prior derived from a three-dimensional shape database and the source imagery;
wherein the object prior represents a three-dimensional shape of an object detected in at least some of the source imagery, and
wherein estimating the camera trajectory further comprises:
estimating first rotation and first translation components of camera motion between a pair of the camera poses based on the feature point correspondences;
estimating second rotation and second translation components of the camera motion between the pair of camera poses using the object prior; and
minimizing a sum of the first and second rotation components by calculating
min
(
t
i
,
t
α
)
[
∑
(
i
,
j
)
d
2
(
R
i
T
t
_
ij
i
,
t
j
-
t
i
t
j
-
t
i
2
)
+
σ
T
∑
(
α
,
i
)
d
2
(
R
i
T
t
_
i
α
i
,
t
α
-
t
i
t
α
-
t
i
2
)
]
;
wherein i and j represent ith and jth camera poses, respectively, used to acquire at least some of the source imagery;
wherein α represents an αth object detected in at least some of the source imagery;
wherein R i and R j represent rotation of the ith and jth camera poses, respectively, with respect to a three-dimensional world coordinate system w∈ 3 ;
wherein R ij represents relative camera rotation between the ith and jth camera poses based on the feature point correspondences;
wherein R α represents rotation of the αth object with respect to the world coordinate system w;
wherein R αi represents relative rotation of the αth object with respect to the Ali camera pose;
wherein d 1 is a distance metric SO3×SO3→ ; and
wherein σ R is a rotation energy term weight.
2. The visual odometry method of claim 1 , wherein the source imagery comprises a monocular video recording having a plurality of frames, each of which corresponds to one of the two-dimensional images.
3. The visual odometry method of claim 1 , wherein estimating the camera trajectory further comprises:
minimizing a sum of the first and second translation components by calculating
min
(
t
i
,
t
α
)
[
∑
(
i
,
j
)
d
2
(
R
i
T
t
_
ij
i
,
t
j
-
t
i
t
j
-
t
i
2
)
+
σ
T
∑
(
α
,
i
)
d
2
(
R
i
T
t
_
i
α
i
,
t
α
-
t
i
t
α
-
t
i
2
)
]
;
wherein i and j represent ith and jth camera poses, respectively, used to acquire at least some embodiments of the source imagery;
wherein α represents an αth object detected in at least some of the source imagery;
wherein t i and t j represent translation of the ith and jth camera poses, respectively, with respect to a three-dimensional world coordinate system w∈ 3 ;
wherein t ij represents relative camera translation between the ith and jth camera poses based on the feature point correspondences;
wherein t α represents translation of the αth object with respect to the world coordinate system w;
wherein t αi represents relative translation of the αth object with respect to the ith camera pose;
wherein d 2 is a distance metric S 2 ×S 2 → ; and
wherein σ T is a translation energy term weight.
4. The visual odometry method of claim 1 , the method further comprising causing a visual representation of the estimated camera trajectory to be displayed.
5. The visual odometry method of claim 1 , further comprising:
defining a two-dimensional bounding box around the object detected in at least some of the source imagery; and
estimating a three-dimensional camera direction for a particular two-dimensional image containing the detected object, the camera direction being estimated by projecting a center of the bounding box and calculating a translation direction defined by a ray extending from a camera location associated with the particular two-dimensional image to the bounding box center.
6. The visual odometry method of claim 1 , further comprising:
defining a two-dimensional bounding box around the object detected in at least some of the source imagery; and
using the object prior to estimate a three-dimensional camera direction for a particular two-dimensional image containing the detected object.
7. The visual odometry method of claim 1 , further comprising:
defining a two-dimensional bounding box around the object detected in at least some of the source imagery; and
tracking the detected object across at least a portion of the two-dimensional images using a Kalman filter applied to the two-dimensional bounding box.
8. A visual odometry system comprising:
a processor;
a display device coupled to the processor;
a memory coupled to the processor, the memory having stored therein source imagery that comprises a plurality of two-dimensional images of a three-dimensional scene, wherein each of the two-dimensional images was captured from a corresponding camera pose;
a stereo matching sub-module that is stored in the memory, the stereo matching sub-module comprising means for estimating rotation and translation components of camera motion between a pair of the camera poses based on feature point correspondences detected in a pair of the two-dimensional images corresponding to the pair of camera poses;
an object prior acquisition sub-module that is stored in the memory, the object prior acquisition sub-module comprising means for estimating rotation and translation components of the camera motion between the pair of camera poses based on an object prior derived from a three-dimensional shape database, wherein the derived object prior represents a three-dimensional shape of an object detected in at least some of the source imagery;
a global optimization sub-module that is stored in the memory, the global optimization sub-module comprising means for estimating a camera trajectory in the three-dimensional scene based on (a) the rotation and translation components estimated by the stereo matching sub-module and (b) the rotation and translation components estimated by the object prior acquisition sub-module and a trajectory smoothening sub-module comprising means for calculating an accumulated score a(f, s) for an angle sample s at a particular two-dimensional source image f;
wherein a(f, s)=p(f, s)+min s′∈N(s) {a(f−1,s′)+w∥s−s′∥ 2 2 };
wherein N(s) is a range of angles neighboring the angle sample s; and
wherein w is a weight that penalizes large-angle movements within the range N(s).
9. The visual odometry system of claim 8 , wherein the three-dimensional shape database includes a plurality of annotated three-dimensional models.
10. The visual odometry system of claim 8 , wherein the object prior acquisition sub-module further comprises means for tracking the detected object across at least some of the two-dimensional images using a Kalman filter.
11. The visual odometry system of claim 8 , further comprising a user interface sub-module that is stored in the memory, the user interface sub-module comprising means for causing the estimated camera trajectory to be displayed on the display device.
12. A non-transitory computer readable medium encoded with instructions that, when executed by one or more processors, causes a visual odometry process to be invoked, the visual odometry process comprising:
receiving source imagery that comprises a plurality of two-dimensional images of a three-dimensional scene, wherein each of the two-dimensional images was captured from a corresponding camera pose;
estimating first rotation and first translation components of camera motion between a pair of the camera poses by identifying feature point correspondences in a pair of the two-dimensional images corresponding to the pair of camera poses;
estimating second rotation and second translation components of the camera motion between the pair of camera poses using a three-dimensional shape that is derived from a shape database, and that represents an object detected in at least some of the source imagery;
minimizing a rotation sum of the first and second rotation components over a plurality of pairwise transformations between camera poses corresponding to at least some of the two-dimensional images;
minimizing a translation sum of the first and second translation components over the plurality of pairwise transformations; and
estimating a camera trajectory that represents the plurality of camera poses from which the corresponding plurality of two-dimensional images were captured, wherein the camera trajectory is defined by a sequence of transformations that establish the minimized rotation and translation sums;
wherein minimizing the translation sum is given by
min
(
R
i
,
R
α
)
[
∑
(
i
,
j
)
d
1
(
R
ij
,
R
j
R
i
T
)
+
σ
R
∑
(
α
,
i
)
d
1
(
R
α
i
,
R
i
R
α
T
)
]
;
wherein i and j represent ith and jth camera poses, respectively, used to acquire at least some of the source imagery;
wherein α represents an αth object detected in at least some of the source imagery;
wherein t i and t j represent translation of the ith and jth camera poses, respectively, with respect to a three-dimensional world coordinate system w∈ 3 ;
wherein t ij represents relative camera translation between the ith and jth camera poses based on the identified feature point correspondences;
wherein t α represents translation of the αth object with respect to the world coordinate system w;
wherein t αi represents relative translation of the αth object with respect to the ith camera pose;
wherein d 2 is a distance metric S 2 ×S 2 → ; and
wherein σ T is a translation energy term weight.
13. The non-transitory computer readable medium of claim 12 , wherein minimizing the rotation sum is given by
min
(
R
i
,
R
α
)
[
∑
(
i
,
j
)
d
1
(
R
ij
,
R
j
R
i
T
)
+
σ
R
∑
(
α
,
i
)
d
1
(
R
α
i
,
R
i
R
α
T
)
]
;
wherein i and j represent ith and jth camera poses, respectively, used to acquire at least some of the source imagery;
wherein α represents an αth object detected in at least some of the source imagery;
wherein R i and R j represent rotation of the ith and jth camera poses, respectively, with respect to a three-dimensional world coordinate system w∈ 3 ;
wherein R ij represents relative camera rotation between the ith and jth camera poses based on the identified feature point correspondences;
wherein R α represents rotation of the αth object with respect to the world coordinate system w;
wherein R αi represents relative rotation of the αth object with respect to the ith camera pose;
wherein d 1 is a distance metric SO3×SO3→ ; and
wherein σ R is a rotation energy term weight.
14. The non-transitory computer readable medium of claim 12 , wherein the visual odometry process further comprises causing a visual representation of the estimated camera trajectory to be displayed on a display device coupled to the one or more processors.
15. The non-transitory computer readable medium of claim 12 , wherein the source imagery comprises a video recording that includes a plurality of sequential video frames.
16. The non-transitory computer readable medium of claim 12 , wherein the source imagery comprises a plurality of unorganized still images.