Depth estimation using a single near-infrared camera and dot illuminator
Provided are computing systems, methods, and platforms for using machine-learned models to generate a depth map. The operations can include projecting, using a dot illuminator, near-infrared (NIR) dots on the scene. The NIR dots can have a uniform pattern. Additionally, the operations can include capturing, using a single NIR camera, the projected NIR dots on the scene. Moreover, the operations can include generating a dot image based on the captured NIR dots on the scene. Furthermore, the operations can include processing the dot image with a machine-learned model to generate a depth map of the scene. Subsequently, the operations can further include evaluating the generated depth map of the scene and a ground truth depth map, and performing an action based on the evaluation.
1 . A computer-implemented method for generating a depth map of a scene, the method comprising:
projecting, using a dot illuminator, near-infrared (NIR) dots on the scene, the NIR dots having a uniform pattern;
capturing, using a single NIR camera, the projected NIR dots on the scene;
generating a dot image based on the captured NIR dots on the scene;
processing the dot image with a machine-learned model to generate a depth map of the scene;
wherein the machine-learned model comprises an autoencoder model comprising one or more neural networks; and
wherein the autoencoder model is configured to generate and output the depth map of the scene by:
determining a depth of the scene based on the dot image; and
selecting a depth contour plot from a plurality of depth contour plots based on the determined depth of the scene.
2 . The method of claim 1 , wherein the dot image is a single dot image, and wherein the depth map is generated based on the single dot image.
3 . The method of claim 1 , wherein machine-learned model is trained on a training dataset of a plurality of training examples, each training example in the plurality of training examples having a dot image and a ground truth depth map.
4 . The method of claim 1 , further comprising:
comparing the generated depth map of the scene with a ground truth depth map to generate a comparison score;
classifying the scene as a spoof representation classification when the comparison score exceeds a threshold value; and
rejecting an authentication request based on the spoof representation classification.
5 . The method of claim 1 , further comprising:
evaluating the generated depth map of the scene and a ground truth depth map; and
performing an action based on the evaluation.
6 . The method of claim 5 , wherein the action performed is to unlock a device.
7 . The method of claim 5 , wherein the action performed is authenticating a user.
8 . The method of claim 1 , wherein the uniform pattern is a uniform dot matrix pattern.
9 . The method of claim 1 , wherein the generation of the depth map of the scene occurs without performing a calibration between the dot illuminator and the single NIR camera.
10 . The method of claim 9 , wherein the calibration is performed during a training stage of the machine-learned model by capturing data from a plurality of devices with different calibration.
11 . The method of claim 10 , wherein each device in the plurality of devices has a similar model type.
12 . The method of claim 1 , further comprising:
evaluating the selected depth contour plot and the determined depth of the scene; and
performing an action based on the evaluation.
13 . The method of claim 1 , further comprising:
accessing the plurality of depth contour plots, wherein the plurality of depth contour plots is associated with a plurality of dot images.
14 . The method of claim 1 , wherein each plot in the plurality of depth contour plots is labeled offline and be associated with a specific dot image, and wherein the specific dot image for each depth contour plot is the ground truth.
15 . A computing system for performing inference, comprising:
one or more processors; and
memory that stores a machine-learned model and instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
projecting, using a dot illuminator, near-infrared (NIR) dots on a scene, the NIR dots having a uniform pattern;
capturing, using a single NIR camera, the projected NIR dots on the scene;
generating a dot image based on the captured NIR dots on the scene; and
processing the dot image with a machine-learned model to generate a depth map of the scene;
wherein the machine-learned model comprises an autoencoder model comprising one or more neural networks; and
wherein the autoencoder model is configured to generate and output the depth map of the scene by:
determining a depth of the scene based on the dot image; and
selecting a depth contour plot from a plurality of depth contour plots based on the determined depth of the scene.
16 . The system of claim 15 , the operations further comprising:
comparing the generated depth map with a ground truth depth map to generate a comparison score;
classifying the scene as a spoof representation classification when the comparison score exceeds a threshold value; and
rejecting an authentication request based on the spoof representation classification.
17 . A computer-implemented method for training a machine-learning model, the method comprising:
obtaining training input comprising a dot image and a ground truth depth map, the dot image being captured using a single camera;
processing the dot image with the machine-learning model to generate a predicted depth map;
wherein the machine-learned model comprises an autoencoder model comprising one or more neural networks; and
wherein the autoencoder model is configured to generate and output the depth map of the scene by:
determining a depth of the scene based on the dot image; and
selecting a depth contour plot from a plurality of depth contour plots based on the determined depth of the scene;
evaluating a loss function based on the predicted depth map and the ground truth depth map; and
modifying one or more parameters of the machine-learning model based on the evaluation of the loss function.
18 . The method of claim 17 , wherein the ground truth depth map is generated using at least two cameras that are calibrated to each other.
19 . The method of claim 17 , wherein the machine-learning model is trained offline, and wherein the ground truth depth map is generated by solving a series of partial differential equations (PDEs).
20 . The method of claim 17 , wherein the ground truth depth map is generated using time-of-flight (ToF) depth estimation.