Multi-frame three-dimensional (3D) reconstruction

1 . A method for three-dimensional reconstruction (3DR) of a scene, the method comprising:

projecting a sample location, of a plurality of sample locations of a block of a scene, onto a plurality of depth frames to determine pixel values for the sample location, wherein each depth frame of the plurality of depth frames corresponds to a pose and comprises a respective depth prediction value corresponding to the sample location, such that the sample location is evaluated across the plurality of depth frames, and wherein the pixel values for the sample location include a respective pixel value corresponding to each depth frame of the plurality of depth frames for the sample location;

reading the depth prediction values of the plurality of depth frames at the pixel values for the sample location on each depth frame of the plurality of depth frames;

obtaining 3D points of the depth prediction values in a three-dimensional (3D) space, wherein the 3D points provide depth estimates for the sample location, each 3D point corresponding to one of the depth prediction values for the sample location;

generating weighted depth prediction values by assigning a weight to each depth prediction value of the depth prediction values for the sample location; and

updating the sample location based on the weighted depth prediction values for the sample location.

2 . The method of claim 1 , further comprising removing outlier depth prediction values from the depth prediction values based on a variance of the depth prediction values.

3 . The method of claim 1 , wherein the pixel values comprise two-dimensional (2D) coordinates.

4 . The method of claim 1 , wherein the 3D space is a camera coordinate space.

5 . The method of claim 1 , wherein the block is a voxel.

6 . The method of claim 1 , further comprising unprojecting the depth prediction values at the pixel values onto the 3D space to obtain the 3D points of the depth prediction values in the 3D space.

7 . The method of claim 6 , wherein the unprojecting of the depth prediction values at the pixel values is performed using camera intrinsic parameters.

8 . The method of claim 1 , further comprising assigning the weight to each depth prediction value of the depth prediction values based on a confidence level for each depth prediction value of the depth prediction values.

9 . The method of claim 8 , further comprising determining the confidence level for each depth prediction value of the depth prediction values using a variance of the depth prediction values.

10 . The method of claim 1 , wherein updating the sample location comprises updating a measure of distance for the sample location from a surface in the scene based on the weighted depth prediction values.

11 . The method of claim 10 , wherein the measure of distance is determined by calculating a truncated signed distance function (TSDF) of the sample location based on the weighted depth prediction values.

12 . An apparatus for three-dimensional reconstruction (3DR) of a scene, the apparatus comprising:

at least one memory; and

at least one processor coupled to the at least one memory and configured to:

project a sample location, of a plurality of sample locations of a block of a scene, onto a plurality of depth frames to determine pixel values for the sample location, wherein each depth frame of the plurality of depth frames corresponds to a pose and comprises a respective depth prediction value corresponding to the sample location, such that the sample location is evaluated across the plurality of depth frames, and wherein the pixel values for the sample location include a respective pixel value corresponding to each depth frame of the plurality of depth frames for the sample location;

read the depth prediction values of the plurality of depth frames at the pixel values for the sample location on each depth frame of the plurality of depth frames;

obtain 3D points of the depth prediction values in a three-dimensional (3D) space, wherein the 3D points provide depth estimates for the sample location, each 3D point corresponding to one of the depth prediction values for the sample location;

generate weighted depth prediction values by assigning a weight to each depth prediction value of the depth prediction values for the sample location; and

update the sample location based on the weighted depth prediction values for the sample location.

13 . The apparatus of claim 12 , wherein the at least one processor is configured to remove outlier depth prediction values from the depth prediction values based on a variance of the depth prediction values.

14 . The apparatus of claim 12 , wherein the pixel values comprise two-dimensional (2D) coordinates.

15 . The apparatus of claim 12 , wherein the 3D space is a camera coordinate space.

16 . The apparatus of claim 12 , wherein the block is a voxel.

17 . The apparatus of claim 12 , wherein the at least one processor is configured to unproject the depth prediction values at the pixel values onto the 3D space to obtain the 3D points of the depth prediction values in the 3D space.

18 . The apparatus of claim 17 , wherein the at least one processor is configured to unproject the depth prediction values at the pixel values using camera intrinsic parameters.

19 . The apparatus of claim 12 , wherein the at least one processor is configured to assign the weight to each depth prediction value of the depth prediction values based on a confidence level for each depth prediction value of the depth prediction values.

20 . The apparatus of claim 19 , wherein the at least one processor is configured to determine the confidence level for each depth prediction value of the depth prediction values using a variance of the depth prediction values.

21 . The apparatus of claim 12 , wherein, to update the sample location, the at least one processor is configured to update a measure of distance for the sample location from a surface in the scene based on the weighted depth prediction values.

22 . The apparatus of claim 21 , wherein, to determine the measure of distance, the at least one processor is configured to calculate a truncated signed distance function (TSDF) of the sample location based on the weighted depth prediction values.

23 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:

project a sample location, of a plurality of sample locations of a block of a scene, onto a plurality of depth frames to determine pixel values for the sample location, wherein each depth frame of the plurality of depth frames corresponds to a pose and comprises a respective depth prediction value corresponding to the sample location, such that the sample location is evaluated across the plurality of depth frames, and wherein the pixel values for the sample location include a respective pixel value corresponding to each depth frame of the plurality of depth frames for the sample location;

read the depth prediction values of the plurality of depth frames at the pixel values for the sample location on each depth frame of the plurality of depth frames;

obtain 3D points of the depth prediction values in a three-dimensional (3D) space, wherein the 3D points provide depth estimates for the sample location, each 3D point corresponding to one of the depth prediction values for the sample location;

generate weighted depth prediction values by assigning a weight to each depth prediction value of the depth prediction values for the sample location; and

update the sample location based on the weighted depth prediction values for the sample location.