High-Speed Real-Time Scene Reconstruction from Input Image Data
A computer-implemented method is disclosed for generating scene reconstructions from image data. The method includes: receiving image data of a scene captured by a camera; inputting the image data of the scene into a scene reconstruction model; receiving, from the scene reconstruction model, a final spatial model of the scene, wherein the scene reconstruction model generates the final spatial model by: predicting a depth map for each image of the image data, extracting a feature map for each image of the image data, generating a first spatial model based on the predicted depth maps of the images, generating a second spatial model based on the extracted feature maps of the images, and determining the final spatial model by combining the first spatial model and the second spatial model; and providing functionality on a computing device related to the scene and based on the final spatial model.
1 . A computer-implemented method comprising:
receiving image data of a scene captured by a camera;
inputting the image data of the scene into a scene reconstruction model;
receiving, from the scene reconstruction model, a final spatial model of the scene, wherein the scene reconstruction model generates the final spatial model by:
predicting a depth map for each image of the image data,
extracting a feature map for each image of the image data,
generating a first spatial model based on the predicted depth maps of the images,
generating a second spatial model based on the extracted feature maps of the images, and
determining the final spatial model by combining the first spatial model and the second spatial model; and
providing functionality on a computing device related to the scene and based on the final spatial model.
2 . The computer-implemented method of claim 1 , wherein the image data comprises a plurality of images and a camera pose for each image.
3 . The computer-implemented method of claim 2 , wherein the camera pose for each image is captured by a position sensor coupled to the camera.
4 . The computer-implemented method of claim 2 , wherein the camera pose for each image is estimated by a pose estimation model based on the images.
5 . The computer-implemented method of claim 1 , wherein predicting the depth map comprises applying a depth estimation model to the image to determine the depth map.
6 . The computer-implemented method of claim 1 , wherein extracting the feature map comprises applying a convolutional network to the image to determine the feature map.
7 . The computer-implemented method of claim 1 , wherein the feature map comprises a first tensor for a first feature type and a second tensor for a second feature type.
8 . The computer-implemented method of claim 1 , wherein the first spatial model is a heightfield, and wherein generating the first spatial model comprises:
generating a 3D model using truncated signed distance field with the predicted depth maps; and
ray casting the 3D model to generate the heightfield.
9 . The computer-implemented method of claim 8 , wherein ray casting the 3D model to generate the heightfield comprises, for each position of the heightfield, casting a ray downward to a surface of the 3D model to determine a height of the surface at that position.
10 . The computer-implemented method of claim 1 , wherein the scene reconstruction model generates the final spatial model by:
generating an aggregate feature map from the feature maps of the images,
wherein generating the second spatial model comprises generating the second spatial model further based on the aggregate feature map.
11 . The computer-implemented method of claim 1 , wherein providing functionality on the computing device related to the scene and based on the final spatial model comprises:
receiving a request from the computing device to view the final spatial model of the scene;
responsive to the request, obtaining the final spatial model of the scene from a map database; and
transmitting the final spatial model to the computing device for viewing the final spatial model of the scene, wherein the computing device is configured to render a perspective of the final spatial model based on a pose of the computing device.
12 . The computer-implemented method of claim 1 , wherein providing functionality on the computing device related to the scene and based on the final spatial model comprises generating, using the final spatial model, virtual content for display in conjunction with the image data of the scene.
13 . A computer-implemented method comprising:
receiving a request from a computing device to view a spatial model of a real-world scene;
responsive to the request, obtaining the spatial model of the real-world scene from a map database, wherein the spatial model of the real-world scene is generated by:
receiving image data of the real-world scene captured by a camera, and
applying a scene reconstruction model to the image data to generate the spatial model of the scene, wherein applying the scene reconstruction model comprises:
predicting a depth map for each image of the image data,
extracting a feature map for each image of the image data,
generating a first spatial representation based on the predicted depth maps of the images,
generating a second spatial representation based on the extracted feature maps of the images, and
determining the spatial model by combining the first spatial representation and the second spatial representation; and
transmitting the spatial model to the computing device for rendering the spatial model of the scene, wherein the computing device is configured to render a perspective of the spatial model based on a pose of the computing device.
14 . The computer-implemented method of claim 13 , wherein the image data comprises a plurality of images and a camera pose for each image.
15 . The computer-implemented method of claim 13 , wherein the feature map comprises a first tensor for a first feature type and a second tensor for a second feature type.
16 . The computer-implemented method of claim 13 , wherein the first spatial representation is a heightfield, and wherein generating the first spatial representation comprises:
generating a 3D model using truncated signed distance field with the predicted depth maps; and
ray casting the 3D model to generate the heightfield.
17 . The computer-implemented method of claim 16 , wherein ray casting the 3D model to generate the heightfield comprises, for each position of the heightfield, casting a ray downward to a surface of the 3D model to determine a height of the surface at that position.
18 . The computer-implemented method of claim 13 , wherein the scene reconstruction model generates the spatial model by:
generating an aggregate feature map from the feature maps of the images,
wherein generating the second spatial representation comprises generating the second spatial representation further based on the aggregate feature map.
19 . The computer-implemented method of claim 13 , further comprising:
augmenting the spatial model with one or more virtual elements,
wherein transmitting the spatial model comprises transmitting the spatial model with the augmented one or more virtual elements to the computing device.
20 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving a request from a computing device to view a spatial model of a real-world scene;
responsive to the request, obtaining the spatial model of the real-world scene from a map database, wherein the spatial model of the real-world scene is generated by:
receiving image data of the real-world scene captured by a camera, and
applying a scene reconstruction model to the image data to generate the spatial model of the scene, wherein applying the scene reconstruction model comprises:
predicting a depth map for each image of the image data,
extracting a feature map for each image of the image data,
generating a first spatial representation based on the predicted depth maps of the images,
generating a second spatial representation based on the extracted feature maps of the images, and
determining the spatial model by combining the first spatial representation and the second spatial representation; and
transmitting the spatial model to the computing device for rendering the spatial model of the scene, wherein the computing device is configured to render a perspective of the spatial model based on a pose of the computing device.