Camera-invariant 3D property formulation
Determining an absolute depth estimate of a vehicle object in a 2D image captured by a camera includes receiving the 2D image captured by the camera. It further includes determining, at least in part by using a segmentation model, a plurality of pixels corresponding to the vehicle object in the 2D image. It furhter includes using a prediction model to determine, for each pixel in the plurality of pixels determined at least in part by using the segmentation model, a correspondng camera invariant distance value comprising a predicted height of a cross-section of the vehicle object that is visible within a given pixel. It further includes determining the absolute depth estimate of the vehicle object in the 2D image based on: a focal length of the camera that captured the 2D image; and an aggregation of the predicted heights determined for each of the pixels in the plurality of pixels.
1 . A system, comprising:
a processor configured to:
receive a 2D (two-dimensional) image captured by a camera;
determine, at least in part by using a segmentation model, a plurality of pixels corresponding to a vehicle object in the 2D image;
use a prediction model to determine, for each pixel in the plurality of pixels determined at least in part by using the segmentation model, a corresponding camera invariant distance value comprising a predicted height of a cross-section of the vehicle object that is visible within a given pixel, wherein training the prediction model to produce predictions of camera invariant distance values for input 2D images comprises:
generating, for training images containing vehicles in various distances in relation to an ego vehicle, numerical arrays comprising target camera invariant distance values determined for pixels corresponding to vehicles whose distances are known in the training images, wherein for each pixel corresponding to a unique vehicle, a same target camera invariant distance value is assigned as a label, the same target camera invariant distance value comprising a smallest depth value of the unique vehicle normalized by pixel focal length; and
sampling from a plurality of different distances at least in part by separating the numerical arrays generated for the training images containing vehicles in various distances into distinct partitions corresponding to different ranges of distances; and
determine an absolute depth estimate of the vehicle object in the 2D image based on:
a focal length of the camera that captured the 2D image; and
an aggregation of the predicted heights determined for each of the pixels in the plurality of pixels; and
a memory coupled to the processor and configured to provide the processor with instructions.
2 . The system of claim 1 , wherein the 2D image comprises an RGB image.
3 . A method, comprising:
receiving a 2D (two-dimensional) image captured by a camera;
determining, at least in part by using a segmentation model, a plurality of pixels corresponding to a vehicle object in the 2D image;
using a prediction model to determine, for each pixel in the plurality of pixels determined at least in part by using the segmentation model, a corresponding camera invariant distance value comprising a predicted height of a cross-section of the vehicle object that is visible within a given pixel, wherein training the prediction model to produce predictions of camera invariant distance values for input 2D images comprises:
generating, for training images containing vehicles in various distances in relation to an ego vehicle, numerical arrays comprising target camera invariant distance values determined for pixels corresponding to vehicles whose distances are known in the training images, wherein for each pixel corresponding to a unique vehicle, a same target camera invariant distance value is assigned as a label, the same target camera invariant distance value comprising a smallest depth value of the unique vehicle normalized by pixel focal length; and
sampling from a plurality of different distances at least in part by separating the numerical arrays generated for the training images containing vehicles in various distances into distinct partitions corresponding to different ranges of distances; and
determining an absolute depth estimate of the vehicle object in the 2D image based on:
a focal length of the camera that captured the 2D image; and
an aggregation of the predicted heights determined for each of the pixels in the plurality of pixels.
4 . The method of claim 3 , wherein the 2D image comprises an RGB image.