Methods and systems for rendering view-dependent images using 2D images
Disclosed herein is methods and systems for providing different views to a viewer. One particular embodiment includes a method including providing, to a neural network, a plurality of 2D images of a 3D object. The neural network may include a signed distance function based sinusoidal representation network. The method may further include obtaining a neural model of a shape of the object by obtaining a zero-level set of the signed distance function; and modeling an appearance of the object using a spatially varying emission function. In some embodiments, the neural model may be converted into a triangular mesh representing the object which may be used to render multiple view-dependent images representative of the 3D object.
1. A method, comprising:
modeling, using image data, a three-dimensional (3D) shape of an object by obtaining a zero-level set of a signed distance function using a shape renderer of a rendering engine;
modeling an appearance of the object by minimizing an image reconstruction error based upon the image data capturing the 3D shape of the object using an appearance renderer of the rendering engine;
combining the 3D shape and the appearance of the object to generate a neural model of the object;
converting the neural model into a triangular mesh representing the object using the rendering engine; and
rendering at least one image using the triangular mesh and using the rendering engine.
2. The method of claim 1 , wherein the signed distance function is based at least in part on a location in 3D space and a first learnable parameter of a sinusoidal representation network.
3. The method of claim 1 , wherein obtaining the zero-level set of the signed distance function includes sphere tracing the signed distance function.
4. The method of claim 3 , wherein sphere tracing the signed distance function includes:
defining a view;
defining a projection matrix;
solving for a ray origin;
solving for a ray direction with vector components and a vector normalization;
minimizing along each ray using iterative updates; and
solving for a zero-set of rays converged to a foreground object for a set count.
5. The method of claim 1 , wherein modeling the appearance of the object includes using a spatially varying emission function.
6. The method of claim 5 , wherein modeling the appearance further comprises defining the spatially varying emission function for directions in a global coordinate system.
7. The method of claim 6 , further comprising conditioning the spatially varying emission function by a local normal direction as computed by automatic differentiation.
8. The method of claim 7 , wherein the spatially varying emission function includes a second learnable parameter of a sinusoidal representation network.
9. The method of claim 8 , further comprising optimizing parameters based on weights for at least one loss function.
10. The method of claim 5 , further comprising minimizing the image reconstruction error for the object in foreground pixels of a display.
11. The method of claim 10 , wherein the image reconstruction error is based on a red-green-blue (RGB) value of a foreground pixel of the display and a portion of pixels with RGB values and object masks.
12. The method of claim 11 , further comprising regularizing the signed distance function by an eikonal constraint.
13. The method of claim 12 , wherein the eikonal constraint is based on a portion of pixels with the RGB values and the object masks.
14. The method of claim 13 , further comprising enforcing a projected pattern to fall within at least one boundary of the object masks.
15. The method of claim 14 , wherein enforcing the projected pattern comprises using a soft mask loss defined for pixels other than the foreground pixels of the display.
16. The method of claim 15 , wherein the soft mask loss includes a binary cross entropy and a minimum value along an entire ray approximated by dense sampling.
17. The method of claim 16 , further comprising regularizing the spatially varying emission function to avoid overfitting to training views.
18. The method of claim 17 , wherein regulating the spatially varying emission function comprises linearizing an angular behavior using a smoothness term.
19. The method of claim 1 , further comprising:
rasterizing the triangular mesh;
projecting vertex positions to each one of a plurality of pixels; and
computing angles between a ray towards a rendering camera and rays towards each of a plurality of projective texture map viewpoints.
20. The method of claim 1 , further comprising applying unstructured lumigraph rendering to blend contributions from first textures, sorted in ascending order to create a rendered image based on computed weights.
21. The method of claim 1 , wherein the at least one image is a view-dependent image.
22. A method, comprising:
modeling, using image data, a 3D shape of an object by obtaining a zero-level set of a signed distance function using a shape renderer of a rendering engine;
modeling an appearance of the object based on a spatially varying emission function using an appearance renderer of the rendering engine;
combining the 3D shape and the appearance of the object to generate a neural model of the object;
converting the neural model into a triangular mesh representing the object using the rendering engine; and
rendering at least one image using the triangular mesh and using the rendering engine.
23. The method of claim 22 , wherein modeling the appearance of the object is performed after the modeling the 3D shape of the object within a processing pipeline.
24. The method of claim 22 , wherein the image data is two-dimensional (2D) image data.