3D reconstruction using 3D triplane representation
An apparatus for generation of a 3D representation of a scene includes a memory for storing a plurality of images depicting a scene; and processing circuitry in communication with the memory. The processing circuitry is configured to generate a depth map for the plurality of input images depicting a scene and unproject the depth map to a three-dimensional (3D) point cloud. The processing circuitry is also configured to project a plurality of 3D points from the 3D point cloud to a plurality of two dimensional (2D) planes and extract a plurality of triplane features from the plurality of 2D planes. The processing circuitry is further configured to generate a 3D representation of the scene based on combining the plurality of triplane features and a plurality of image features extracted from the plurality of input images.
1 . An apparatus for generation of a three-dimensional (3D) representation of a scene, the apparatus comprising:
a memory for storing a plurality of input images depicting a scene; and
processing circuitry in communication with the memory, wherein the processing circuitry is configured to:
generate a depth map for the plurality of input images depicting a scene;
unproject the depth map to a 3D point cloud;
project a plurality of 3D points from the 3D point cloud to a plurality of two dimensional (2D) planes;
extract a plurality of triplane features from the plurality of 2D planes; and
generate a 3D representation of the scene based on combining the plurality of triplane features and a plurality of image features extracted from the plurality of input images.
2 . The apparatus of claim 1 , wherein the processing circuitry is further configured to:
generate a plurality of truncated signed distance function (TSDF) values, prior to generating the 3D representation of the scene; and
convert the plurality of TSDF values into a 3D mesh.
3 . The apparatus of claim 1 , wherein the processing circuitry is further configured to:
sample a bounding box of a predetermined size from the scene using random translation function and rotation function.
4 . The apparatus of claim 1 , wherein the processing circuitry configured to project the plurality of 3D points from the 3D point cloud is further configured to project the plurality of 3D points using a point-based learning network model.
5 . The apparatus of claim 4 , wherein the point-based learning network model is configured to generate a plurality of 2D feature maps.
6 . The apparatus of claim 5 , wherein the processing circuitry configured to extract the plurality of triplane features is further configured to input the plurality of 2D feature maps into a 2D U-Net model having U-Net architecture.
7 . The apparatus of claim 1 , wherein the processing circuitry configured to extract the plurality of triplane features is further configured to:
determine if a test position does not project to an integer coordinate in a 2D plane of the plurality of 2D planes; and
generate, for the test position, a feature based on features from one or more neighboring coordinates of the test position using bilinear interpolation, responsive to determining that the test position does not project to the integer coordinate in the 2D plane.
8 . The apparatus of claim 1 , wherein the 3D representation of the scene comprises a 3D mesh.
9 . The apparatus of claim 1 , wherein the plurality of input images comprises a plurality of images of the scene taken from different viewpoints.
10 . The apparatus of claim 1 , wherein the processing circuitry configured to generate the 3D representation of the scene is further configured to generate a single feature vector comprising the 3D representation of the scene.
11 . The apparatus of claim 1 , wherein the plurality of 2D planes comprises six planes.
12 . The apparatus of claim 1 , wherein the processing circuitry configured to generate the depth map is further configured to generate the depth map using a deep neural network.
13 . A method comprising:
generating a depth map for the plurality of input images depicting a scene;
unprojecting the depth map to a 3D point cloud;
projecting a plurality of 3D points from the 3D point cloud to a plurality of two dimensional (2D) planes;
extracting a plurality of triplane features from the plurality of 2D planes; and
generating a 3D representation of the scene based on combining the plurality of triplane features and a plurality of image features extracted from the plurality of input images.
14 . The method of claim 13 , further comprising:
generating a plurality of truncated signed distance function (TSDF) values, prior to generating the 3D representation of the scene; and
converting the plurality of TSDF values into a 3D mesh.
15 . The method of claim 13 , further comprising:
sampling a bounding box of a predetermined size from the scene using random translation function and rotation function.
16 . The method of claim 13 , wherein projecting the plurality of 3D points from the 3D point cloud further comprises projecting the plurality of 3D points using a point-based learning network model.
17 . The method of claim 16 , wherein the point-based learning network model is configured to generate a plurality of 2D feature maps.
18 . The method of claim 17 , wherein extracting the plurality of triplane features further comprises inputting the plurality of 2D feature maps into a 2D U-Net model having U-Net architecture.
19 . The method of claim 13 , wherein extracting the plurality of triplane features further comprises:
determining if a test position does not project to an integer coordinate in a 2D plane of the plurality of 2D planes; and
generating, for the test position, a feature based on features from one or more neighboring coordinates of the test position using bilinear interpolation, responsive to determining that the test position does not project to the integer coordinate in the 2D plane.
20 . The method of claim 13 , wherein the 3D representation of the scene comprises a 3D mesh.
21 . The method of claim 13 , wherein the plurality of input images comprises a plurality of images of the scene taken from different viewpoints.
22 . The method of claim 13 , wherein generating the 3D representation of the scene further comprises generating a single feature vector comprising the 3D representation of the scene.
23 . The method of claim 13 , wherein the plurality of 2D planes comprises six planes.
24 . The method of any of claim 13 , wherein generating the depth map further comprises generating the depth map using a deep neural network.
25 . A non-transitory computer-readable medium storing instructions that, when applied by processing circuitry, causes the processing circuitry to:
generate a depth map for the plurality of input images depicting a scene;
unproject the depth map to a 3D point cloud;
project a plurality of 3D points from the 3D point cloud to a plurality of two dimensional (2D) planes;
extract a plurality of triplane features from the plurality of 2D planes; and
generate a 3D representation of the scene based on combining the plurality of triplane features and a plurality of image features extracted from the plurality of input images.
26 . A device for generation of a three-dimensional (3D) representation of a scene, the device comprising one or more means for performing steps of:
generating a depth map for the plurality of input images depicting a scene;
unprojecting the depth map to a 3D point cloud;
projecting a plurality of 3D points from the 3D point cloud to a plurality of two dimensional (2D) planes;
extracting a plurality of triplane features from the plurality of 2D planes; and
generating a 3D representation of the scene based on combining the plurality of triplane features and a plurality of image features extracted from the plurality of input images.
27 . The device of claim 26 , wherein the one or more means comprise one or more processors implemented in circuitry.
28 . The device of claim 27 , further comprising a memory to store the plurality of input images.
29 . The device of claim 26 , further comprising the one or more means for performing steps of:
generating a plurality of truncated signed distance function (TSDF) values, prior to generating the 3D representation of the scene; and
converting the plurality of TSDF values into a 3D mesh.
30 . The device of claim 26 , further comprising the one or more means for performing steps of:
sampling a bounding box of a predetermined size from the scene using random translation function and rotation function.