Image sequence trajectories for visual odometry
Images captured by a camera moving in an environment are received, and for each of a plurality of points in one of the images, outputs are computed using a neural network. The outputs comprise: a trajectory depicting the point in each of the plurality of images, as well as, for each trajectory, a prediction of visibility of the trajectory in each of the images and a prediction of whether the trajectory depicts a static or moving surface in the environment. The neural network receives the images and points as input and computes the outputs, wherein the outputs comprise for each of the trajectories, confidence data. The outputs are sent to a downstream process selected from any of: visual odometry, structure from motion, human body tracking, video editing, vehicle tracking.
1 . A computer-implemented method comprising:
receiving a sequence of a plurality of images captured by a camera moving in an environment;
for each of a plurality of points in one of the plurality of images, computing outputs comprising: a trajectory depicting the plurality of points in each of the plurality of images, as well as, for each trajectory, a prediction of visibility of the trajectory in each of the plurality of images and a prediction of whether the trajectory depicts a static or moving surface in the environment;
wherein computing the trajectories comprises using a neural network which receives the plurality of images and the plurality of points as input and computes the outputs, wherein the outputs comprise for each of the trajectories, confidence data;
selecting individual ones of the trajectories with the confidence data above a threshold, the prediction of visibility of the trajectory above a threshold, and the prediction of whether the trajectory depicts the static surface above a threshold;
computing a position and orientation of the camera from the individual ones of the trajectories; and
sending the individual ones of the trajectories to a downstream process.
2 . The method of claim 1 , wherein the confidence data comprises probability distribution parameters for each point in each trajectory.
3 . The method of claim 2 , wherein the probability distribution parameters comprise a location parameter indicating estimated point position and a scale parameter representing data deviation.
4 . The method of claim 2 , wherein the probability distribution parameters are from a Cauchy distribution.
5 . The method of claim 2 , wherein the probability distribution parameters are updated in an iterative refinement module of the neural network.
6 . The method of claim 1 , wherein the neural network comprises a feature extraction neural network that adaptively aggregates spatial information for each point.
7 . The method of claim 1 , wherein the neural network comprises a feature extraction neural network that performs feature enhancement for each image independently.
8 . The method of claim 1 , wherein the neural network comprises a feature extraction neural network that performs feature enhancement using correlation between the images.
9 . The method of claim 1 , wherein the neural network comprises a feature neural network having: a deformable convolution layer that employs adaptive kernel size and positioning, or a dilated convolution which introduces gaps into kernel filters.
10 . The method of claim 1 , wherein the downstream process is selected from any of: visual odometry, structure from motion, human body tracking, video editing, and vehicle tracking.
11 . The method of claim 9 , wherein dilution rate of the dilution convolution doubles for every consecutive layer of the feature neural network.
12 . The method of claim 1 , wherein the neural network comprises a spatial mixer component that facilitates information exchange across all points within the same image.
13 . The method of claim 12 , wherein the spatial mixer component comprises a channel mixer which combines information channel-wise for each point, an inter-trajectory mixer which combines information between different trajectories, and a temporal mixer which combines information within the same trajectory.
14 . The method of claim 1 wherein the plurality of points comprises a plurality of anchor points, one anchor point per grid cell of one of the images, and where each anchor point depicts an edge in the one of the images.
15 . A computer-implemented method of computing position and orientation of a camera moving in an environment comprising:
receiving a sequence of two or more images captured by the camera moving in the environment;
for each of a plurality of points in one of the images, computing outputs comprising: a trajectory depicting the point in each of the images, as well as, for each trajectory, a prediction of visibility of the trajectory in each of the images and a prediction of whether the trajectory depicts a static or moving surface in the environment;
wherein computing the trajectories comprises using a neural network which receives the images and the plurality of points as input and computes the outputs, wherein the outputs comprise for each of the trajectories, confidence data;
selecting individual ones of the trajectories with confidence data above a threshold, the prediction of visibility of the trajectory above a threshold, the prediction of whether the trajectory depicts the static surface above a threshold; and
computing a position and orientation of the camera from the selected individual ones of the trajectories.
16 . The method of claim 15 , wherein computing the position and orientation of the camera from the selected individual ones of the trajectories comprises using bundle adjustment with a bundle adjustment window greater than the two or more images.
17 . The method of claim 15 , wherein the neural network has been trained using semi-supervised learning that tracks a first plurality of the points referred to as queries and a second plurality of the points referred to as anchor points together but only calculates losses using query predictions.
18 . The method of claim 15 , wherein the neural network comprises; a feature neural network having a deformable convolution layer that employs adaptive kernel size and positioning, or an inter-trajectory attention mechanism.
19 . The method of claim 15 , further comprising sending the individual ones of the trajectories to a downstream process selected from visual odometry, structure from motion, human body tracking, video editing, and vehicle tracking.
20 . An apparatus comprising:
a processor;
a memory storing instructions that, when executed by the processor, perform a method comprising:
receiving two or more images captured by a camera moving in an environment;
for each of a plurality of points in one of the images, computing outputs comprising: a trajectory depicting the point in each of the images, as well as, for each trajectory, a prediction of visibility of the trajectory in each of the images and a prediction of whether the trajectory depicts a static or moving surface in the environment;
wherein computing the trajectories comprises using a neural network which receives the images and the plurality of points as input and computes the outputs, and wherein the outputs comprise for each of the trajectories, confidence data; and
computing a position and orientation of the camera from the outputs comprising the confidence data.