Self-supervised training from a teacher network for cost volume based depth estimates
View Patent ↗A method for controlling a vehicle in an environment includes generating, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images. The method also includes generating combined features by combining cost volume features of the cross-attention cost volume with single-frame features associated with the current image. The single-frame features may be generated via a single-frame encoding model. The method further includes generating a depth estimate of the current image based on the combined features. The method still further includes controlling an action of the vehicle based on the depth estimate.
1. A method for controlling a vehicle in an environment, comprising:
generating, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images;
generating combined features by combining cost volume features of the cross-attention cost volume with single-frame features associated with the current image, the single-frame features being generated via a single-frame encoding model;
generating a depth estimate of the current image based on the combined features; and
controlling an action of the vehicle based on the depth estimate.
2. The method of claim 1 , further comprising:
generating current image features from the current image via a feature extraction network, each one of the current image features corresponds to a current image pixel in the current image; and
generating previous image features from the previous image via the feature extraction network, each one of the previous image features corresponds to a previous image pixel in the previous image, wherein the cross-attention cost volume is generated by cross-attention matching each feature from the current image features with one or more features of the previous image features.
3. The method of claim 2 , further comprising cross-attention matching each feature from the current image features with the one or more features of the previous image features by:
sampling, for each current image pixel, one or more candidate pixels from the previous image corresponding to the current image pixel along an epipolar line; and
matching, for each current image pixel, a current image feature associated with the current image pixel with each previous image feature associated with the one or more sampled candidate pixels corresponding to the current image pixel.
4. The method of claim 1 , further comprising removing each cost volume feature of the cost volume features that satisfies a removal condition, wherein the cost volume features and the single-frame features are combined after removing each cost volume that satisfies the removal condition.
5. The method of claim 1 , further comprising obtaining the current image and the previous image from a monocular camera associated with the vehicle, wherein the current image and the previous image are two-dimensional (2D) images.
6. The method of claim 1 , further comprising generating a three-dimensional (3D) reconstruction of the environment via the depth estimate.
7. The method of claim 1 , further comprising training the cross-attention model and the single-frame encoding model end-to-end in a self-supervised manner.
8. An apparatus for controlling a vehicle in an environment, comprising:
a processor; and
a memory coupled with the processor and storing instructions operable, when executed by the processor, to cause the apparatus to:
generate, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images;
generate combined features by combining cost volume features of the cross-attention cost volume with single-frame features associated with the current image, the single-frame features being generated via a single-frame encoding model;
generate a depth estimate of the current image based on the combined features; and
control an action of the vehicle based on the depth estimate.
9. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to:
generate current image features from the current image via a feature extraction network, each one of the current image features corresponds to a current image pixel in the current image;
generate previous image features from the previous image via the feature extraction network, each one of the previous image features corresponds to a previous image pixel in the previous image, wherein the cross-attention cost volume is generated by cross-attention matching each feature from the current image features with one or more features of the previous image features.
10. The apparatus of claim 9 , wherein execution of the instructions further cause the apparatus to cross-attention match each feature from the current image features with the one or more features of the previous image features by:
sampling, for each current image pixel, one or more candidate pixels from the previous image corresponding to the current image pixel along an epipolar line; and
matching, for each current image pixel, a current image feature associated with the current image pixel with each previous image feature associated with the one or more sampled candidate pixels corresponding to the current image pixel.
11. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to remove each cost volume feature of the cost volume features that satisfies a removal condition, wherein the cost volume features and the single-frame features are combined after removing each cost volume that satisfies the removal condition.
12. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to obtain the current image and the previous image from a monocular camera associated with the vehicle, wherein the current image and the previous image are two-dimensional (2D) images.
13. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to generate a three-dimensional (3D) reconstruction of the environment via the depth estimate.
14. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to train the cross-attention model and the single-frame encoding model end-to-end in a self-supervised manner.
15. A non-transitory computer-readable medium having program code recorded thereon for controlling a vehicle in an environment, the program code executed by a processor and comprising:
program code to generate, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images;
program code to generate combined features by combining cost volume features of the cross-attention cost volume with single-frame features associated with the current image, the single-frame features being generated via a single-frame encoding model;
program code to generate a depth estimate of the current image based on the combined features; and
program code to control an action of the vehicle based on the depth estimate.
16. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises:
program code to generate current image features from the current image via a feature extraction network, each one of the current image features corresponds to a current image pixel in the current image;
program code to generate previous image features from the previous image via the feature extraction network, each one of the previous image features corresponds to a previous image pixel in the previous image, wherein the cross-attention cost volume is generated by cross-attention matching each feature from the current image features with one or more features of the previous image features.
17. The non-transitory computer-readable medium of claim 16 , wherein the program code further comprises program code to cross-attention match each feature from the current image features with the one or more features of the previous image features by:
sampling, for each current image pixel, one or more candidate pixels from the previous image corresponding to the current image pixel along an epipolar line; and
matching, for each current image pixel, a current image feature associated with the current image pixel with each previous image feature associated with the one or more sampled candidate pixels corresponding to the current image pixel.
18. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises program code to remove each cost volume feature of the cost volume features that satisfies a removal condition, wherein the cost volume features and the single-frame features are combined after removing each cost volume that satisfies the removal condition.
19. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises program code to obtain the current image and the previous image from a monocular camera associated with the vehicle, wherein the current image and the previous image are two-dimensional (2D) images.
20. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises program code to generate a three-dimensional (3D) reconstruction of the environment via the depth estimate.