Methods and apparatus for metric depth estimation using a monocular visual-inertial system
Methods, apparatus, systems, and articles of manufacture are disclosed for metric depth estimation using a monocular visual-inertial system. An example apparatus for metric depth estimation includes at least one memory, instructions in the apparatus, and processor circuitry to execute the instructions to access a globally-aligned depth prediction, the globally-aligned depth prediction generated based on a monocular depth estimator, access a dense scale map scaffolding, the dense scale map scaffolding generated based on visual-inertial odometry, regress a dense scale residual map determined using the globally-aligned depth prediction and the dense scale map scaffolding, and apply the dense scale residual map to the globally-aligned depth prediction.
1. An apparatus for metric depth estimation, the apparatus comprising:
at least one memory;
instructions; and
processor circuitry to execute the instructions to:
access a globally-aligned depth prediction, the globally-aligned depth prediction generated based on a monocular depth estimator;
access a dense scale map scaffolding, the dense scale map scaffolding generated based on visual-inertial odometry;
regress a dense scale residual map determined using the globally-aligned depth prediction and the dense scale map scaffolding; and
apply the dense scale residual map to the globally-aligned depth prediction.
2. The apparatus of claim 1 , wherein the visual-inertial odometry determines the dense scale map scaffolding based on inertial measurement unit (IMU) data and visual data.
3. The apparatus of claim 2 , wherein the visual-inertial odometry generates a sequence of sparse maps based on the IMU data and the visual data, the sequence of sparse maps including metric depth values, the dense scale map scaffolding based on the metric depth values.
4. The apparatus of claim 3 , wherein the globally-aligned depth prediction is based on an alignment of monocular depth estimates to the metric depth values.
5. The apparatus of claim 1 , wherein the globally-aligned depth prediction is based on a least-squares estimation for global scale and global shift.
6. The apparatus of claim 1 , wherein the processor circuitry is to train a scale map learner (SML) neural network to resolve the scale ambiguity in monocular depth estimates.
7. The apparatus of claim 6 , wherein the SML neural network is to fill a region within a convex hull via linear interpolation of anchor values and fill a region outside the convex hull with an identify scale value of one.
8. A method for metric depth estimation, the method comprising:
accessing a globally-aligned depth prediction, the globally-aligned depth prediction generated based on a monocular depth estimator;
accessing a dense scale map scaffolding, the dense scale map scaffolding generated based on visual-inertial odometry;
regressing, by executing an instruction with at least one processor circuit, a dense scale residual map determined using the globally-aligned depth prediction and the dense scale map scaffolding; and
applying, by executing an instruction with one or more of the at least one processor circuit, the dense scale residual map to the globally-aligned depth prediction.
9. The method of claim 8 , wherein the visual-inertial odometry determines the dense scale map scaffolding based on inertial measurement unit (IMU) data and visual data.
10. The method of claim 9 , wherein the visual-inertial odometry generates a sequence of sparse maps based on the IMU data and the visual data, the sequence of sparse maps including metric depth values, the dense scale map scaffolding based on the metric depth values.
11. The method of claim 10 , wherein the globally-aligned depth prediction is based on an alignment of monocular depth estimates to the metric depth values.
12. The method of claim 8 , wherein the globally-aligned depth prediction is determined based on a least-squares estimation for global scale and global shift.
13. The method of claim 8 , further including training a scale map learner (SML) neural network to resolve the scale ambiguity in monocular depth estimates.
14. The method of claim 13 , wherein the SML neural network is to fill a region within a convex hull via linear interpolation of anchor values and fill a region outside the convex hull with an identify scale value of one.
15. A non-transitory computer readable storage medium comprising instructions that, when executed, cause at least one processor circuit to at least:
access a globally-aligned depth prediction, the globally-aligned depth prediction generated based on a monocular depth estimator;
access a dense scale map scaffolding, the dense scale map scaffolding generated based on visual-inertial odometry;
regresses a dense scale residual map determined using the globally-aligned depth prediction and the dense scale map scaffolding; and
apply the dense scale residual map to the globally-aligned depth prediction.
16. The non-transitory computer readable storage medium of claim 15 , wherein the visual-inertial odometry determines the dense scale map scaffolding based on inertial measurement unit (IMU) data and visual data.
17. The non-transitory computer readable storage medium of claim 16 , wherein the visual-inertial odometry generates a sequence of sparse maps based on the IMU data and the visual data, the sequence of sparse maps including metric depth values, the dense scale map scaffolding based on the metric depth values.
18. The non-transitory computer readable storage medium of claim 17 , wherein globally-aligned depth prediction is based on an alignment of monocular depth estimates to the metric depth values.
19. The non-transitory computer readable storage medium of claim 15 , wherein the instructions, when executed, cause one or more of the at least one processor circuit to train a scale map learner (SML) neural network to resolve the scale ambiguity in monocular depth estimates.
20. The non-transitory computer readable storage medium of claim 19 , wherein the SML neural network is to fill a region within a convex hull via linear interpolation of anchor values and fill a region outside the convex hull with an identify scale value of one.
21. An apparatus for metric depth estimation, the apparatus comprising:
means for accessing a globally-aligned depth prediction, the globally-aligned depth prediction generated based on means for estimating using monocular depth estimation;
means for accessing a dense scale map scaffolding, the dense scale map scaffolding generated based on means for estimating using visual-inertial odometry;
means for regressing a dense scale residual map determined using the globally-aligned depth prediction and the dense scale map scaffolding; and
means for applying the dense scale residual map to the globally-aligned depth prediction.
22. The apparatus of claim 21 , wherein the means for estimating using visual-inertial odometry is to determine the dense scale map scaffolding based on inertial measurement unit (IMU) data and visual data.
23. The apparatus of claim 22 , wherein the means for estimating using visual-inertial odometry is to generate a sequence of sparse maps based on the IMU data and the visual data, the sequence of sparse maps including metric depth values, the dense scale map scaffolding based on the metric depth values.
24. The apparatus of claim 23 , wherein the globally-aligned depth prediction is based on an alignment of monocular depth estimates to the metric depth values.
25. The apparatus of claim 21 , wherein the globally-aligned depth prediction is based on a least-squares estimation for global scale and global shift.