Layered scene decomposition CODEC with higher order lighting
A system and methods for a CODEC driving a real-time light field display for multi-dimensional video streaming, interactive gaming and other light field display applications is provided applying a layered scene decomposition strategy. Multi-dimensional scene data including information on directions of normals is divided into a plurality of data layers of increasing depths as the distance between a given layer and the display surface increases. Data layers which are sampled using a plenoptic sampling scheme and rendered using hybrid rendering, such as perspective and oblique rendering, to encode light fields corresponding to each data layer. The resulting compressed, (layered) core representation of the multi-dimensional scene data is produced at predictable rates, reconstructed and merged at the light field display in real-time by applying view synthesis protocols, including edge adaptive interpolation, to reconstruct pixel arrays in stages (e.g. columns then rows) from reference elemental images.
1. A computer-implemented method comprising:
receiving a first data set comprising a three-dimensional description of a scene, the first data set comprising information on directions of normals on surfaces in the scene, the directions of the normals represented with respect to a reference direction, wherein at least some of the surfaces have non-Lambertian reflection properties;
partitioning the first data set into a plurality of layers, each layer representing a portion of the scene at a location with respect to a reference location; and
encoding multiple layers to generate a second data set, wherein a size of the second data set is smaller than a size of the first data set,
wherein layers located closer to a display surface achieve a lower compression ratio than layers of the same width located further away from the display surface.
2. The method of claim 1 , further comprising transmitting the second data set to a remote device comprising a display for presenting the scene.
3. The method of claim 1 , further comprising:
transmitting the second data set to a remote device comprising a display; and
presenting the scene on the display.
4. The method of claim 1 , wherein encoding multiple layers comprises performing a sampling operation on at least a portion of the first data set to generate the second data set.
5. The method of claim 4 , wherein performing the sampling operation is based on a target compression rate associated with the second data set.
6. The method of claim 1 , wherein encoding multiple layers comprises:
rendering using ray tracing, a set of pixels to be encoded;
selecting multiple elemental images from a plurality of elemental images such that the set of pixels are rendered using the selected multiple elemental images; and
sampling the set of pixels using a sampling operation.
7. The method of claim 4 , wherein performing the sampling operation comprises selecting multiple elemental images from a plurality of elemental images in accordance with a plenoptic sampling scheme.
8. The method of claim 4 , wherein performing the sampling operation comprises:
determining an effective spatial resolution associated with each layer; and
selecting multiple elemental images, from a plurality of elemental images in accordance with a determined angular resolution.
9. The method of claim 8 , wherein the angular resolution is determined as a function of a directional resolution associated with the portion of the scene associated with each layer.
10. The method of claim 8 , wherein the angular resolution is determined as a field of view associated with a display device.
11. The method of claim 1 , wherein the three-dimensional description comprises light field data representing a plurality of elemental images.
12. The method of claim 11 , wherein each of the plurality of elemental images is captured by one or more image acquisition devices.
13. The method of claim 1 , wherein the light field data includes a depth map corresponding to the elemental images.
14. The method of claim 1 , further comprising:
receiving the second data set;
reconstructing the portions associated with a layer using the directions of normals on surfaces included in the scene for calculation of a specular component;
combining the reconstructed portions into a light field; and
presenting the light field image on a display device.
15. The method of claim 14 , further comprising:
receiving user-input indicative of a location of a user with respect to the light field image; and
updating the light field image in accordance with the user-input prior to presentation on the display device.
16. The method of claim 1 , wherein the first data set includes extra-pixel information comprising information on the directions of normals stored in a geometry buffer.
17. The method of claim 15 , wherein the geometry buffer further stores color and depth information.
18. The method of claim 1 wherein the second data set comprise multiple layers of light fields.
19. The method of claim 18 wherein the light fields are merged to create a final light field.
20. The method of claim 1 wherein partitioning the first data set into a plurality of layers comprises restricting a depth range of each layer.
21. The method of claim 1 wherein layers located closer to the display surface are narrower in width than layers located farther away from the display surface.
22. The method of claim 1 wherein partitioning the first data set into a plurality of layers maintains a uniform compression rate across the scene.
23. The method of claim 1 wherein partitioning the first data set into a plurality of layers comprises partitioning the light field display into inner and outer frustum volume layer sets.
24. The method of claim 1 wherein the method is used to generate a synthetic light field for multi-dimensional video streaming, multi-dimensional interactive gaming, real-time interactive content, or other light field display scenarios.
25. The method of claim 24 wherein the synthetic light field is generated only in a valid viewing zone.
26. A light field image rendering method comprising of the steps of:
partitioning a three-dimensional surface description of a scene into layers, each layer having an associated light field and sampling scheme;
further partitioning at least one layer into a plurality of subsections, each subsection having an associated light field and sampling, wherein a location of a particular subsection is determined in accordance with geometry of at least a portion of an object represented within the scene;
rendering a first set of pixels, comprising extra-pixel information, for each layer and each subsection in accordance with the sampling scheme and corresponds to a sampled light field;
reconstructing the sampled light field for each layer and subsection using the first set of pixels; and
merging the reconstructed light fields into a single output light field image.
27. The method of claim 26 , wherein the first set of pixels and associated extra-pixel information is partitioned into subsets, whereby reconstructing sampled light fields for each layer and subsection and merging are performed using pixels from a single subset in a cache to create a subset of the output light field image.
28. The method of claim 27 , wherein reconstructing the sampled light field for each layer and subsection is performed by re-projecting pixels in the first set from the cache to create the subset of the output light field image.
29. The method of claim 28 , wherein re-projecting pixels is performed using a warping process along a single dimension in the first set of pixels followed by a second warping process in a second dimension in the first set of pixels.
30. The method of claim 26 , wherein the three-dimensional surface description comprises information on directions of normals on surfaces in the scene.
31. The method of claim 30 , wherein the directions of normals are represented with respect to a reference direction.
32. The method of claim 30 , wherein at least some of the surfaces have non-Lambertian reflection properties.
33. The method of claim 26 , wherein the first set of pixels when rendered comprises extra-pixel information comprising of normals.
34. The method of claim 26 , further comprising:
reconstructing the sampled light field for each layer and subsection using the first set of pixels, including normals on surfaces included in the scene for calculation of a specular component.