Method and an electronic device for 3D scene reconstruction and visualization

1 . A method for 3D scene reconstruction and visualization, the method comprising, using at least one processor:

obtaining a trained base neural network by training the base neural network for obtaining distance information for voxels of a real scene;

operating the trained base neural network for obtaining the distance information of an input sequence of frames of the real scene;

inputting the distance information to an algorithm that outputs a 3D reconstruction of the real scene;

obtaining a 3D visualization of the real scene by rendering the 3D reconstruction of the real scene; and

instructing at least one display to display the 3D visualization of the real scene,

wherein the training the base neural network includes:

inputting training data of a training scene, including a training sequence of RGB (Red, Green, Blue) frames with corresponding camera pose data, into a backbone of the base neural network;

computing a common loss function as a sum of Truncated Signed Distance Function (TSDF) losses, segmentation losses, and normal losses; and

minimizing the common loss function to obtain a minimized common loss function.

2 . The method of claim 1 , wherein the training the base neural network includes:

computing the TSDF losses between a TSDF prediction, obtained by a TSDF head of the base neural network from data output from the backbone, and a ground truth scan;

deriving, by a segmentation head of the base neural network from the data output from the backbone, a segmentation prediction defining floor areas, wall areas, and other areas for each training voxel in the training scene;

computing the segmentation losses between the segmentation prediction and a segmentation ground truth;

computing normal coordinates as gradients of the TSDF prediction of TSDF values in each training voxel;

computing the normal losses for normals for the wall areas and the floor areas based on gradients of TSDF for the ground truth scan and gradients of the TSDF prediction.

3 . The method of claim 2 , wherein the segmentation head and TSDF head are used in parallel while training.

4 . The method of claim 2 , wherein the minimizing the common loss function comprises:

minimizing the common loss function by computing a gradient of the common loss function over parameters of the base neural network.

5 . The method of claim 2 , further comprising updating parameters of the base neural network by providing backpropagation of the minimized common loss function and updating the parameters of the base neural network according to the minimized common loss function.

6 . The method of claim 2 , wherein the minimizing the common loss function comprises:

minimizing the common loss function repeatedly until the common loss function stops decreasing.

7 . The method of claim 2 , wherein the minimizing the common loss function comprises:

minimizing the common loss function repeatedly until the common loss function reaches a preset threshold value.

8 . The method of claim 2 , further comprising, after the computing normal coordinates:

selecting wall normal vectors in the wall areas and computing vertical components of the wall normal vectors; and

selecting floor normal vectors in the floor areas and computing horizontal components of the floor normal vectors.

9 . The method of claim 8 , wherein the computing the normal losses comprises:

obtaining, at each point in the wall areas, the vertical components of wall normal vectors,

obtaining, at each point in the floor areas, the horizontal components of the floor normal vectors and x- and y-components of the floor normal vectors, and

computing the normal losses by using lengths of z-components of the wall normal vectors, normals of two-dimensional vectors composed of the x- and y-components.

10 . The method of claim 1 , wherein the 3D reconstruction of the real scene comprises a point cloud or triangle mesh.

11 . A non-transitory computer-readable medium storing program instructions that when executed by at least one processor, cause the at least one processor of an electronic device to implement the method of claim 1 .

12 . An electronic device for 3D scene reconstruction and visualization, the electronic device comprising:

at least one display;

at least one memory configured to store instructions; and

at least one processor configured to execute the instructions to:

obtain a trained base neural network by training the base neural network for obtaining distance information for voxels of a real scene;

operate the trained base neural network for obtaining the distance information of an input sequence of frames of the real scene;

input the distance information to an algorithm that outputs a 3D reconstruction of the real scene;

obtain a 3D visualization of the real scene by rendering the 3D reconstruction of the real scene; and

instruct the at least one display to display the 3D visualization of the real scene,

wherein the at least one processor is further configured to execute the instructions to:

input training data of a training scene, including a training sequence of RGB (Red, Green, Blue) frames with corresponding camera pose data, into a backbone of the base neural network;

compute a common loss function as a sum of Truncated Signed Distance Function (TSDF) losses, segmentation losses, and normal losses; and

minimize the common loss function to obtain a minimized common loss function.

13 . The electronic device of claim 12 , wherein the at least one processor is further configured to:

compute the TSDF losses between a TSDF prediction, obtained by a TSDF head of the base neural network from data output from the backbone, and a ground truth scan,

derive, by a segmentation head of the base neural network from the data output from the backbone, a segmentation prediction defining floor areas, wall areas, and other areas for each training voxel in the training scene,

compute the segmentation losses between the segmentation prediction and a segmentation ground truth,

compute normal coordinates as gradients of the TSDF prediction of TSDF values in each training voxel,

compute the normal losses for normals for the wall areas and floor areas based on gradients of TSDF for the ground truth scan and gradients of the TSDF prediction.

14 . The electronic device of claim 13 , wherein the segmentation head and TSDF head are used in parallel while training.

15 . The electronic device of claim 13 , wherein the at least one processor is further configured to:

minimize the common loss function by computing a gradient of the common loss function over parameters of the base neural network.

16 . The electronic device of claim 13 , wherein the at least one processor is further configured to:

update parameters of the base neural network by providing backpropagation of the minimized common loss function, and

update the parameters of the base neural network according to the minimized common loss function.

17 . The electronic device of claim 13 , wherein the at least one processor is further configured to:

minimize the common loss function repeatedly until the common loss function stops decreasing.

18 . The electronic device of claim 12 , wherein the at least one processor is further configured to:

select the wall normal vectors in the wall areas and compute vertical components of the wall normal vectors, and

select floor normal vectors in the floor areas and compute horizontal components of the floor normal vectors.

19 . The electronic device of claim 18 , wherein the at least one processor is further configured to:

obtain, at each point in the wall areas, the vertical components of wall normal vectors,

obtain, at each point in the floor areas, the horizontal components of the floor normal vectors and x- and y-components of the floor normal vectors, and

compute the normal losses by using lengths of z-components of the wall normal vectors, normals of two-dimensional vectors composed of the x- and y-components.

20 . The electronic device of claim 12 , wherein the 3D reconstruction of the real scene comprises a point cloud or triangle mesh.