Systems and methods for reconstructing scenes to disentangle light and matter fields
A stored volumetric scene model of a real scene is generated from data defining digital images of light that flows multi-directionally in the real scene forming a light field, the light interacting with different types of media that form a matter field. The digital images have been formed by a camera from one or more poses, and each digital image contains image data elements defined by stored data representing light field flux received by light sensing detectors in the camera. The digital images are processed by a scene reconstruction engine to form a digital volumetric scene model representing the real scene. The volumetric scene model (i) contains volumetric data elements defined by stored data representing one or more media characteristics and (ii) contains data elements defined by stored data representing the flux of the light field. Adjacent volumetric data elements form corridors, at least one of the volumetric data elements in at least one corridor represents media that is partially light transmissive. The constructed digital volumetric scene model data is stored in a digital data memory for subsequent uses and applications.
1 . A scene reconstruction system for creating a digital model of a scene, the scene comprising light and different types of media that interact and about which the system lacks a priori knowledge, comprising:
a storage medium configured to store image data from a single type of sensor, and the scene model, wherein:
the image data represents a light flow in one or more regions in the scene; and
the scene model represents the scene as matter field data and multi-directional light field data;
a processor configured to:
access the image data,
update the scene model using the image data to calculate:
updated matter field data comprising a volumetric representation of the media in the one or more regions including light interaction data representing one or more interactions of the light with the media, wherein the media in each of the one or more regions comprises some matter or no matter, and
updated light field data representing the light in the scene based at least in part on the light flow; and
store the updated scene model in the storage medium, wherein each of the updated matter field data and the updated light field data are independently accessible and outputtable; and
an output circuit configured to output at least a portion of the updated scene model.
2 . The system of claim 1 wherein the image data is one or more of sensed image data or synthetic image data, each of which comprises one or more of unidirectional image data, multidirectional image data, or omnidirectional image data.
3 . The system of claim 1 wherein the storage medium comprises a hierarchical, multi-resolution, spatially-sorted data structure and the processor is configured to store the scene model in the data structure.
4 . The system of claim 1 wherein the light interaction data represents one or more of a refractive index, an absorption, a transmission, a reflection, a refraction, a scattering, an opacity, a texture, a polarization characteristic, a polarized characteristic, an unpolarized characteristic, and an extinction coefficient.
5 . The system of claim 1 wherein the processor is configured to perform the update to the scene model associated with the one or more regions along a corridor in the scene.
6 . The system of claim 5 wherein the corridor is represented by a ray and the system further determines a characteristic of the one or more regions through which the ray passes.
7 . The system of claim 1 wherein the processor is configured to perform the update to the scene model using one or more of spherical harmonics, interpolation, one or more basis functions, machine learning, or machine intelligence.
8 . The system of claim 1 wherein the processor is configured to perform the update to the scene model by iteratively performing the updating until the updated scene model exceeds a threshold level of accuracy, certainty, or confidence, or a user-defined factor.
9 . The system of claim 1 wherein at least part of the scene model represents one or more of a dynamic light field or a dynamic matter field.
10 . The system of claim 1 wherein the processor is further configured to select at least a portion of the updated scene model representing one or more objects in the scene, wherein the selected portion of the updated scene model is configured to be relighted with a simulated source of illumination.
11 . The system of claim 1 wherein the processor is further configured to select at least a portion of the updated light field data, wherein the selected portion of the updated light field data is configured to relight a simulated scene.
12 . The system of claim 1 , wherein the system lacks a priori knowledge of all light flows and all media in the scene.
13 . The system of claim 1 , wherein the updated light field data comprises a representation of light exitant from the one or more regions as a responsive light field and an optional emissive light field.
14 . The system of claim 1 wherein the storage medium is further configured to store a second image data from a second type of sensor, and wherein the processor is further configured to further update the scene model using the second image data to calculate the updated matter field data and the updated light field data.
15 . A method for reconstructing a scene comprising light and different types of media that interact with no or limited a priori knowledge thereof, the method comprising:
accessing image data from a single type of sensor representing a light flow in one or more regions in the scene;
accessing a scene model representing the scene as matter field data and multi-directional light field data;
updating the scene model using the image data, wherein the updating comprises calculating:
updated matter field data comprising a volumetric representation of the media in the one or more regions including light interaction data representing one or more interactions of the light with the media, wherein the media in each of the one or more regions comprises some matter or no matter; and
updated light field data representing the light in the scene based at least in part on the light flow;
storing the updated scene model in a storage medium, wherein each of the updated matter field data and the updated light field data are independently accessible and outputtable; and
outputting at least a portion of the updated scene model.
16 . The method of claim 15 wherein the image data is one or more of sensed image data or synthetic image data, each of which comprises one or more of unidirectional image data, multidirectional image data, or omnidirectional image data.
17 . The method of claim 15 wherein the storage medium is a hierarchical, multi-resolution, spatially-sorted data structure.
18 . The method of claim 15 wherein the light interaction data represents one or more of a refractive index, an absorption, a transmission, a reflection, a refraction, a scattering, an opacity, a texture, a polarization characteristic, a polarized characteristic, an unpolarized characteristic, and an extinction coefficient.
19 . The method of claim 15 wherein the updating the scene model comprises calculating the scene model associated with the one or more regions along a corridor in the scene.
20 . The method of claim 19 wherein the corridor is represented by a ray and the method further comprises determining a characteristic of the one or more regions through which the ray passes.
21 . The method of claim 15 wherein the updating the scene model uses one or more of spherical harmonics, interpolation, one or more basis functions, machine learning, or machine intelligence.
22 . The method of claim 15 wherein the updating the scene model comprises iteratively performing the updating until the updated scene model exceeds a threshold level of accuracy, certainty, or confidence, or a user-defined factor.
23 . The method of claim 15 wherein the scene model represents one or more of a dynamic light field or a dynamic matter field.
24 . The method of claim 15 further comprising selecting at least a portion of the updated scene model representing one or more objects in the scene, wherein the selected portion of the updated scene model is configured to be relighted with a simulated source of illumination.
25 . The method of claim 15 further comprising selecting at least a portion of the updated light field data, wherein the selected portion of the updated light field is configured to relight a simulated scene.
26 . The method of claim 15 , wherein there is no a priori knowledge on the light flow and media in the scene.
27 . The method of claim 15 , wherein the updated light field data comprises a representation of light exitant from the one or more regions as a responsive light field and an optional emissive light field.
28 . The method of claim 15 further comprising accessing a second image data from a second type of sensor, and further updating the scene model using the second image data to calculate the updated matter field data and the updated light field data.
29 . A non-transitory computer readable medium storing instructions that, when executed by at least one processor of a computer, cause the computer to perform a method for reconstructing a scene comprising light and different types of media that interact with no or limited a priori knowledge thereof, wherein the method comprises:
accessing image data from a single type of sensor representing a light flow in one or more regions in the scene;
accessing a scene model representing the scene as matter field data and multi-directional light field data;
updating the scene model using the image data, wherein the updating comprises calculating:
updated matter field data comprising a volumetric representation of the media in the one or more regions including light interaction data representing one or more interactions of the light with the media, wherein the media in each of the one or more regions comprises some matter or no matter; and
updated light field data representing the light in the scene based at least in part on the light flow;
storing the updated scene model in a storage medium, wherein each of the updated matter field data and the updated light field data are independently accessible and outputtable; and
outputting at least a portion of the updated scene model.