VOLUMETRIC SCENE RECONSTRUCTION WITH VARIABLE VOXEL RESOLUTION IN TRUNCATED SIGNED DISTANCE FUNCTION (TSDF) FUSION

1 . A computer-implemented method comprising:

receiving image data capturing a real-world environment and captured by a camera assembly of a client device, the image data comprising a plurality of frames;

applying a depth estimation model to each frame to output a depth map corresponding to the frame, wherein the depth map comprises depth predictions for pixels in the frame;

applying a semantic segmentation model to each frame to output a segmentation mask corresponding to the frame, wherein the segmentation mask classifies pixels in the frame into one of a plurality of semantic classes;

determining level hints for each frame based on the segmentation mask and the depth map corresponding to the frame, wherein the level hints indicate a voxel resolution level for each pixel;

generating a variable-resolution truncated signed distance function (TSDF) grid by fusing depth predictions from the depth maps corresponding to the plurality of frames, the variable-resolution TSDF grid comprising TSDF values indicating distance to a surface in the real-world environment, wherein the variable-resolution TSDF grid includes at least one portion at a first voxel resolution level and another portion at a second voxel resolution level of finer resolution than the first voxel resolution level, and wherein the voxel resolution at each portion of the variable-resolution TSDF grid is based on the level hints;

generating a polygon mesh from the variable-resolution TSDF grid digitally representing surfaces in the real-world environment captured by the image data; and

storing the polygon mesh in a map database.

2 . The computer-implemented method of claim 1 , further comprising:

applying an object detection model to each frame to identify one or more objects in the frame; and

tracking one or more of the objects across frames;

wherein generating the variable-resolution TSDF grid is further based on the tracked one or more objects in the real-world environment.

3 . The computer-implemented method of claim 2 , wherein the object detection model is trained as a machine-learning model in a supervised manner with training image data labeled with identified objects.

4 . The computer-implemented method of claim 1 ,

wherein applying the depth estimation model to each frame further comprises applying the depth estimation model to identify surface orientation of one or more surfaces present in the frame; and

wherein generating the variable-resolution TSDF grid is further based on the one or more surface orientations of the one or more surfaces.

5 . The computer-implemented method of claim 1 , wherein the depth estimation model is trained as a machine-learning model in a self-supervised manner by projecting frames from training image data onto other frames of the training image data based on depth predictions by the depth estimation model.

6 . The computer-implemented method of claim 1 , wherein generating the variable-resolution TSDF grid is constrained by limiting neighboring voxel cells to be at most one voxel resolution level difference.

7 . The computer-implemented method of claim 1 , wherein generating the variable-resolution TSDF grid comprises implementing a hyperparameter that sets a quantity of depth predictions fused into the TSDF value per voxel, wherein the hyperparameter is fit to an error curve for depth predictions by the depth estimation model.

8 . The computer-implemented method of claim 1 , wherein generating the polygon mesh from the variable-resolution TSDF grid comprises interpolating between neighboring voxels of different voxel resolution levels.

9 . The computer-implemented method of claim 1 , further comprising:

augmenting the polygon mesh with patterns from the image data corresponding to one or more surfaces represented by the polygon mesh.

10 . The computer-implemented method of claim 1 , further comprising:

receiving a request from a second client device to view the polygon mesh;

retrieving the polygon mesh from the map database; and

transmitting the polygon mesh to the second client device for presentation on the second client device.

11 . A non-transitory computer-readable storage medium storing instructions that, when executed by a computer processor, cause the computer processor to perform operations comprising:

receiving image data capturing a real-world environment and captured by a camera assembly of a client device, the image data comprising a plurality of frames;

applying a depth estimation model to each frame to output a depth map corresponding to the frame, wherein the depth map comprises depth predictions for pixels in the frame;

applying a semantic segmentation model to each frame to output a segmentation mask corresponding to the frame, wherein the segmentation mask classifies pixels in the frame into one of a plurality of semantic classes;

determining level hints for each frame based on the segmentation mask and the depth map corresponding to the frame, wherein the level hints indicate a voxel resolution level for each pixel;

generating a variable-resolution truncated signed distance function (TSDF) grid by fusing depth predictions from the depth maps corresponding to the plurality of frames, the variable-resolution TSDF grid comprising TSDF values indicating distance to a surface in the real-world environment, wherein the variable-resolution TSDF grid includes at least one portion at a first voxel resolution level and another portion at a second voxel resolution level of finer resolution than the first voxel resolution level, and wherein the voxel resolution at each portion of the variable-resolution TSDF grid is based on the level hints;

generating a polygon mesh from the variable-resolution TSDF grid digitally representing surfaces in the real-world environment captured by the image data; and

storing the polygon mesh in a map database.

12 . The non-transitory computer-readable storage medium of claim 11 , the operations further comprising:

applying an object detection model to each frame to identify one or more objects in the frame; and

tracking one or more of the objects across frames;

wherein generating the variable-resolution TSDF grid is further based on the tracked one or more objects in the real-world environment.

13 . The non-transitory computer-readable storage medium of claim 12 , wherein the object detection model is trained as a machine-learning model in a supervised manner with training image data labeled with identified objects.

14 . The non-transitory computer-readable storage medium of claim 1 ,

wherein applying the depth estimation model to each frame further comprises applying the depth estimation model to identify surface orientation of one or more surfaces present in the frame; and

wherein generating the variable-resolution TSDF grid is further based on the one or more surface orientations of the one or more surfaces.

15 . The non-transitory computer-readable storage medium of claim 1 , wherein the depth estimation model is trained as a machine-learning model in a self-supervised manner by projecting frames from training image data onto other frames of the training image data based on depth predictions by the depth estimation model.

16 . The non-transitory computer-readable storage medium of claim 11 , wherein generating the variable-resolution TSDF grid is constrained by limiting neighboring voxel cells to be at most one voxel resolution level difference.

17 . The non-transitory computer-readable storage medium of claim 11 , wherein generating the variable-resolution TSDF grid comprises implementing a hyperparameter that sets a quantity of depth predictions fused into the TSDF value per voxel, wherein the hyperparameter is fit to an error curve for depth predictions by the depth estimation model.

18 . The non-transitory computer-readable storage medium of claim 11 , wherein generating the polygon mesh from the variable-resolution TSDF grid comprises interpolating between neighboring voxels of different voxel resolution levels.

19 . The non-transitory computer-readable storage medium of claim 1 , the operations further comprising:

augmenting the polygon mesh with patterns from the image data corresponding to one or more surfaces represented by the polygon mesh.

20 . The non-transitory computer-readable storage medium of claim 11 , the operations further comprising:

receiving a request from a second client device to view the polygon mesh;

retrieving the polygon mesh from the map database; and

transmitting the polygon mesh to the second client device for presentation on the second client device.