Layered scene decomposition CODEC with layered depth imaging
A system and methods for a CODEC driving a real-time light field display for multi-dimensional video streaming, interactive gaming and other light field display applications is provided applying a layered scene decomposition strategy. Multi-dimensional scene data is divided into a plurality of data layers of increasing depths as the distance between a given layer and the display surface increases. Data layers are sampled using a plenoptic sampling scheme and rendered using hybrid rendering, such as perspective and oblique rendering, to encode light fields corresponding to each data layer. The resulting compressed, (layered) core representation of the multi-dimensional scene data is produced at predictable rates, reconstructed and merged at the light field display in real-time by applying view synthesis protocols, including edge adaptive interpolation, to reconstruct pixel arrays in stages (e.g. columns then rows) from reference elemental images.
1. A computer-implemented method comprising:
receiving a data set comprising a three-dimensional description of a scene, the data set representing intersecting objects in the scene;
partitioning the three-dimensional description of a scene into layers, each layer having an associated light field and sampling scheme;
further partitioning at least one layer into a plurality of layer subsections, each subsection comprising at least one elemental image with an associated light field and resulting in a surface buffer that contains samples of surfaces in the scene;
partitioning the surface buffer; and
rendering the surface buffer partitions for each subsection, the surface buffer partitions comprising pixels selected by the sampling scheme and data enabling reconstruction of each light field associated with each subsection to generate data for an output light field.
2. The method of claim 1 , wherein the surface buffer comprises fewer pixels than the output light field.
3. The method of claim 1 , wherein the pixels in the surface buffer comprise extra-pixel information.
4. The method of claim 3 , wherein the extra-pixel information comprises one or more of color, depth, surface coordinates, surface normals, material values, and transparency value.
5. The method of claim 1 , wherein objects in the scene are represented schematically as prismatic shapes or circular shapes.
6. The method of claim 1 , wherein objects in the scene are represented as polygons.
7. The method of claim 1 , wherein objects in the scene are located at varying depths from a display surface.
8. The method of claim 1 , wherein each subsection location is determined in accordance with geometry of at least a portion of an object represented within the scene.
9. The method of claim 1 , wherein rendering the surface buffer comprises using a two-dimensional rendering ray-tracing pipeline.
10. The method of claim 1 , further comprising rendering a deep geometry buffer for each elemental image.
11. The method of claim 1 , wherein the sampling scheme specifies pixels which comprise the surface buffer.
12. The method of claim 1 , wherein the output light field is displayed on a display device.
13. The method of claim 1 , wherein the surface buffer partitions are loaded into a faster cache memory from which reconstruction and merging calculations can be performed.
14. The method of claim 1 , further comprising rendering an output frame by reconstructing the layers and subsections to create corresponding portions of the output light field.