Systems and methods for neural implicit scene representation with dense, uncertainty-aware monocular depth constraints
In accordance with one embodiment of the present disclosure, a method includes receiving a set of images, each image depicting a view of a scene, generating sparse depth data from each image of the set of images, training a monocular depth estimation model with the sparse depth data, generating, with the trained monocular depth estimation model, depth data and uncertainty data for each image, training a NeRF model with the set of images, wherein the training is constrained by the depth data and uncertainty data, and rendering, with the trained NeRF model, a new image having a new view of the scene.
1. A method comprising:
receiving a set of images, each image depicting a view of a scene;
generating sparse depth data from each image of the set of images;
training a monocular depth estimation model with the sparse depth data;
generating, with the trained monocular depth estimation model, depth data and uncertainty data for each image;
training a NeRF model with the set of images, wherein the training is constrained by the depth data and uncertainty data; and
rendering, with the trained NeRF model, a new image having a new view of the scene;
wherein generating sparse depth data comprises:
extracting geometric features and color features from each image of the set of images;
identifying matching geometric features and color features between images of the set of images;
determining a depth measurement based on the identified matching geometric features and color features; and
compiling the depth measurements into the sparse depth data.
2. The method of claim 1 , wherein the set of images comprises monocular RGB images.
3. The method of claim 1 , wherein the NeRF model comprises:
an input layer that receives geometrics features;
one or more hidden layers, each hidden layer having a weight; and
an output layer that generates color features.
4. The method of claim 3 , wherein training the NeRF model comprises:
extracting geometric features and color features from each image of the set of images; and
adjusting the weights of the NeRF model to map the geometric features of an image to its corresponding color features.
5. The method of claim 1 , further comprising receiving geometric features from a user.
6. The method of claim 5 , wherein rendering the new view comprises:
inputting geometric features of the new view into the trained NeRF model;
generating color features of the new view with the trained NeRF model; and
compositing the geometric features and the color features into the new image with a differentiable volume rendering function.
7. A system comprising:
a processor configured to perform operations comprising:
receiving a set of images, each image depicting a view of a scene;
generating sparse depth data from each image of the set of images;
training a monocular depth estimation model with the sparse depth data;
generating, with the trained monocular depth estimation model, depth data and uncertainty data for each image;
training a NeRF model with the set of images, wherein the training is constrained by the depth data and uncertainty data; and
rendering, with the trained NeRF model, a new image having a new view of the scene;
wherein generating sparse depth data comprises:
extracting geometric features and color features from each image of the set of images;
identifying matching geometric features and color features between images of the set of images;
determining a depth measurement based on the identified matching geometric features and color features; and
compiling the depth measurements into the sparse depth data.
8. The system of claim 7 , wherein the set of images comprises monocular RGB images.
9. The system of claim 8 , wherein the NeRF model comprises:
an input layer that receives geometrics features;
one or more hidden layers, each hidden layer having a weight; and
an output layer that generates color features.
10. The system of claim 9 , wherein training the NeRF model comprises:
extracting geometric features and color features from each image of the set of images; and
adjusting the weights of the NeRF model to map the geometric features of an image to its corresponding color features.
11. The system of claim 7 , wherein the processor is configured to perform operations further comprising receiving geometric features from a user.
12. The system of claim 11 , wherein rendering the new view comprises:
inputting geometric features of the new view into the trained NeRF model;
generating color features of the new view with the trained NeRF model; and
compositing the geometric features and the color features into the new image with a differentiable volume rendering function.
13. A non-transitory medium having machine-readable instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving a set of images, each image depicting a view of a scene;
generating sparse depth data from each image of the set of images;
training a monocular depth estimation model with the sparse depth data;
generating, with the trained monocular depth estimation model, depth data and uncertainty data for each image;
training a NeRF model with the set of images, wherein the training is constrained by the depth data and uncertainty data; and
rendering, with the trained NeRF model, a new image having a new view of the scene;
wherein generating sparse depth data comprises:
extracting geometric features and color features from each image of the set of images;
identifying matching geometric features and color features between images of the set of images;
determining a depth measurement based on the identified matching geometric features and color features; and
compiling the depth measurements into the sparse depth data.
14. The non-transitory medium of claim 13 , wherein the set of images comprises monocular RGB images.
15. The non-transitory medium of claim 13 , wherein the NeRF model comprises:
an input layer that receives geometrics features;
one or more hidden layers, each hidden layer having a weight; and
an output layer that generates color features.
16. The non-transitory medium of claim 15 , wherein training the NeRF model comprises:
extracting geometric features and color features from each image of the set of images; and
adjusting the weights of the NeRF model to map the geometric features of an image to its corresponding color features.
17. The non-transitory medium of claim 13 , wherein rendering the new view comprises:
receiving geometric features from a user;
inputting geometric features of the new view into the trained NeRF model;
generating color features of the new view with the trained NeRF model; and
compositing the geometric features and the color features into the new image with a differentiable volume rendering function.