Quotidian scene reconstruction engine
A stored volumetric scene model of a real scene is generated from data defining digital images of a light field in a real scene containing different types of media. The digital images have been formed by a camera from opposingly directed poses and each digital image contains image data elements defined by stored data representing light field flux received by light sensing detectors in the camera. The digital images are processed by a scene reconstruction engine to form a digital volumetric scene model representing the real scene. The volumetric scene model (i) contains volumetric data elements defined by stored data representing one or more media characteristics and (ii) contains solid angle data elements defined by stored data representing the flux of the light field. Adjacent volumetric data elements form corridors, at least one of the volumetric data elements in at least one corridor represents media that is partially light transmissive. The constructed digital volumetric scene model data is stored in a digital data memory for subsequent uses and applications.
1. A scene processing method comprising:
acquiring one or more sensed digital images of scene light flowing in a scene comprising scene media, wherein (A) said scene light flows in opposing directions in said scene media, (B) said sensed digital images are sensed by at least one camera located at one or more camera poses, (C) said sensed digital images comprise sensed pixel data elements representing characteristics of said scene light, and (D) scene entities formed by at least part of said scene media and/or said scene light comprise at least one of a scene characteristic, a scene surface, a scene feature and a scene object; and
determining one or more updated scene reconstruction data elements using (a) one or more of the sensed pixel data elements, (b) one or more initial scene reconstruction data elements, and (c) a light transport equation representing the scene light flowing in equilibrium into, out of and within the scene media, wherein the scene light flowing within is represented in the light transport equation which transforms incident scene light to exitant scene light by way of a light interaction function,
wherein
scene reconstruction data elements comprise i) scene media data elements representing a matter field of the scene comprising geometric and material properties of said scene media; ii) light data elements representing a light field of the scene comprising geometric and radiometric characteristics of said scene light; and iii) camera data elements representing said camera poses,
initial scene reconstruction data elements are scene reconstruction data elements that serve as input to the determining process if they exist, and
updated scene reconstruction data elements are scene reconstruction data elements output from the determining process.
2. The scene processing method of claim 1 , wherein:
acquiring one or more sensed digital images comprises capturing and/or accessing one or more sensed digital images,
determining one or more updated scene reconstruction data elements comprises using non-linear optimization and/or using machine learning, and
at least one of said scene entities is represented by at least one of said scene reconstruction data elements using at least one of said sensed pixel data elements.
3. The scene processing method of claim 2 , further comprising:
accessing at least one user-identified scene reconstruction goal;
repeating the process of claim 2 until the scene reconstruction goal is satisfied, and
generating output data representing at least one of (aa) at least one scene entity associated at least in part with the user-identified goal, (bb) at least one camera data element representing at least part of said camera poses and at least part of a camera model, and (cc) at least one camera data element representing at least part of said camera model, wherein the output data is determined at least in part using at least one scene reconstruction data element.
4. The scene processing method of claim 2 , wherein:
at least one of the sensed digital images is a substantially omnidirectional image sensed by at least one camera.
5. The scene processing method of claim 2 , wherein at least one of the sensed pixel data elements represents a plurality of characteristics of polarized light, and at least one of the scene reconstruction data elements is determined at least in part using characteristics of polarized light.
6. The scene processing method of claim 5 , wherein:
the at least one scene reconstruction data element is determined at least in part using characteristics of polarized light representing at least one spatially localized scene feature sensitive to polarized light.
7. The scene processing method of claim 6 , wherein the at least one scene reconstruction data element represents the at least one spatially localized scene feature sensitive to polarized light and is used at least in part to perform any one or any combination of:
labeling scene features across a plurality of images,
image registration,
scene initialization, and
scene reconstruction.
8. The scene processing method of claim 5 , wherein:
the scene surface is a vehicle surface,
the scene feature is damage to the vehicle surface comprising at least one of a dent, an anomaly, and debris, and
one or more characteristics of the damage are determined using any one or any combination of i)
at least one of the scene reconstruction data elements,
a 3D model of the associated undamaged vehicle surface,
non-linear optimization, and
a machine learning model.
9. The scene processing method of claim 2 , wherein:
the scene media comprises one or more volumetric scene media subregions comprising homogeneous or heterogeneous media,
the scene light interacts with the scene media subregions by phenomena comprising any one of or any combination of absorption, emission, reflection, transmission, and scattering,
the scene media data elements representing geometric properties of the scene media subregions comprise at least one of a value, a distance, an angle, a point, a vector, a plane, a surfel, a voxel, and a reference, and
the scene media data elements representing material properties of the scene media subregions comprise at least one of a value, a gradient, a bi-directional reflectance distribution function (BRDF), a bi-directional light interaction function (BLIF), and a reference.
10. The scene processing method of claim 2 , wherein:
the scene light flows in at least one frusta,
the scene light interacts with the scene media by phenomena comprising any one or any combination of absorption, emission, reflection, transmission, and scattering,
the scene light data elements representing geometric properties of the frusta comprise at least one of a value, a point, a voxel, a direction, an angle, a vector, a solid angle, and a reference, and
the scene light data elements representing radiometric properties of the frusta comprising at least one of a value, a gradient, a radiant flux, a radiant power, and a reference.
11. The scene processing method of claim 2 , wherein:
at least one of the geometric characteristics of the scene media form an octree subdivided into octree cells organized in a spatially sorted and hierarchical manner.
12. The scene processing method of claim 11 , wherein:
the geometric characteristics of the scene light form a solid-angle octree,
directions at a point are subdivided into solid-angle elements, and
the solid-angle elements are organized in a spatially sorted and hierarchical manner.
13. The scene processing method of claim 12 , further comprising computing synthetic images with perspective projection using at least in part the solid-angle octree, wherein the computations use shift operations rather than general purpose divide operations.
14. The scene processing method of claim 13 , further comprising using spans to perform perspective projection.
15. The scene processing method of claim 14 , wherein:
at least one excludable octree cell is excluded from processing during computation of the synthetic images,
determination of the at least one excludable octree cell comprises any one or any combination of: i) determining occluded octree cells using a quadtree or equivalent data structure that enables a ‘front-to-back’ traversal of the scene media for accessing occlusion, and ii) determining lesser-sized octree cells using a comparison of a window size with respect to a size that may vary within a synthetic image computation that is determined at least in part with respect to a pixel size, wherein the window size is determined at least in part with respect to a perspective projection onto a display plane.
16. A three dimensional (3D) imaging system comprising at least one scene reconstruction engine having at least one digital signal processor connected for digital communication with at least one camera and a digital signal input/output communication interface, said 3D imaging system being configured to execute a scene processing method comprising:
acquiring one or more sensed digital images of scene light flowing in a scene comprising scene media, wherein (A) said scene light flows in opposing directions in said scene media, (B) said sensed digital images are sensed by at least one camera located at one or more camera poses, (C) said sensed digital images comprise sensed pixel data elements representing characteristics of said scene light, and (D) scene entities formed by at least part of said scene media and/or said scene light comprise at least one of a scene characteristic, a scene surface, a scene feature and a scene object; and
determining one or more updated scene reconstruction data elements using (a) one or more of the sensed pixel data elements, (b) one or more initial scene reconstruction data elements, and (c) a light transport equation representing the scene light flowing in equilibrium into, out of and within the scene media wherein the scene light flowing within is represented in the light transport equation which transforms incident scene light to exitant scene light by way of a light interaction function,
wherein
scene reconstruction data elements comprise i) scene media data elements representing a matter field of the scene comprising geometric and material properties of said scene media; ii) light data elements representing a light field of the scene comprising geometric and radiometric characteristics of said scene light; and iii) camera data elements representing said camera poses,
initial scene reconstruction data elements are scene reconstruction data elements that serve as input to the determining process if they exist, and
updated scene reconstruction data elements are scene reconstruction data elements output from the determining process.
17. The 3D digital imaging system of claim 16 , wherein:
acquiring one or more sensed digital images comprises capturing and/or accessing one or more sensed digital images,
determining one or more updated scene reconstruction data elements comprises using non-linear optimization and/or using machine learning, and
at least one of said scene entities is represented by at least one of said scene reconstruction data elements using at least one of said sensed pixel data elements.
18. The 3D digital imaging system of claim 17 , further comprising:
accessing at least one user-identified scene reconstruction goal;
repeating the process of claim 17 until the scene reconstruction goal is satisfied, and
generating output data representing at least one of (aa) at least one scene entity associated at least in part with the user-identified goal, (bb) at least one camera data element representing at least part of said camera poses and at least part of a camera model, and (cc) at least one camera data element representing at least part of said camera model, wherein the output data is determined at least in part using at least one scene reconstruction data element.
19. The 3D digital imaging system of claim 17 , wherein:
at least one of the sensed digital images is a substantially omnidirectional image sensed by at least one camera.
20. The 3D digital imaging system of claim 17 , wherein at least one of the sensed pixel data elements represents a plurality of characteristics of polarized light, and at least one of the scene reconstruction data elements is determined at least in part using characteristics of polarized light.
21. The 3D digital imaging system of claim 20 , wherein:
the at least one scene reconstruction data element is determined at least in part using characteristics of polarized light representing at least one spatially localized scene feature sensitive to polarized light.
22. The 3D digital imaging system of claim 21 , wherein the at least one scene reconstruction data element represents the at least one spatially localized scene feature sensitive to polarized light and is used at least in part to perform any one or any combination of:
labeling scene features across a plurality of images,
image registration,
scene initialization, and
scene reconstruction.
23. The 3D digital imaging system of claim 20 , wherein:
the scene surface is a vehicle surface,
the scene feature is damage to the vehicle surface comprising at least one of a dent, an anomaly, and debris, and
one or more characteristics of the damage are determined using any one or any combination of
at least one of the scene reconstruction data elements,
a 3D model of the associated undamaged vehicle surface,
non-linear optimization, and
a machine learning model.
24. The 3D digital imaging system of claim 17 , wherein:
the scene media comprises one or more volumetric scene media subregions comprising homogeneous or heterogeneous media,
the scene light interacts with the scene media subregions by phenomena comprising any one of or any combination of absorption, emission, reflection, transmission, and scattering,
the scene media data elements representing geometric properties of the scene media subregions comprise at least one of a value, a distance, an angle, a point, a vector, a plane, a surfel, a voxel, and a reference, and
the scene media data elements representing material properties of the scene media subregions comprise at least one of a value, a gradient, a bi-directional reflectance distribution function (BRDF), a bi-directional light interaction function (BLIF), and a reference.
25. The 3D digital imaging system of claim 17 , wherein:
the scene light flows in at least one frusta,
the scene light interacts with the scene media by phenomena comprising any one or any combination of absorption, emission, reflection, transmission, and scattering,
the scene light data elements representing geometric properties of the frusta comprise at least one of a value, a point, a voxel, a direction, an angle, a vector, a solid angle, and a reference, and
the scene light data elements representing radiometric properties of the frusta comprising at least one of a value, a gradient, a radiant flux, a radiant power, and a reference.
26. The 3D digital imaging system of claim 17 , wherein:
at least one of the geometric characteristics of the scene media form an octree subdivided into octree cells organized in a spatially sorted and hierarchical manner.
27. The 3D digital imaging system of claim 26 , wherein:
the geometric characteristics of the scene light form a solid-angle octree,
directions at a point are subdivided into solid-angle elements, and
the solid-angle elements are organized in a spatially sorted and hierarchical manner.
28. The 3D digital imaging system of claim 27 , further comprising computing synthetic images with perspective projection using at least in part the solid-angle octree, wherein the computations use shift operations rather than general purpose divide operations.
29. The 3D digital imaging system of claim 28 , further comprising using spans to perform perspective projection.
30. The 3D digital imaging system of claim 29 , wherein:
at least one excludable octree cell is excluded from processing during computation of the synthetic images,
determination of the at least one excludable octree cell comprises any one or any combination of: i) determining occluded octree cells using a quadtree or equivalent data structure that enables a ‘front-to-back’ traversal of the scene media for accessing occlusion, and ii) determining lesser-sized octree cells using a comparison of a window size with respect to a size that may vary within a synthetic image computation that is determined at least in part with respect to a pixel size, wherein the window size is determined at least in part with respect to a perspective projection onto a display plane.
31. A non-transitory computer program storage media containing computer program instructions configured, when executed in a 3D imaging system, to effect a scene processing method comprising:
acquiring one or more sensed digital images of scene light flowing in a scene comprising scene media, wherein (A) said scene light flows in opposing directions in said scene media, (B) said sensed digital images are sensed by at least one camera located at one or more camera poses, (C) said sensed digital images comprise sensed pixel data elements representing characteristics of said scene light, and (D) scene entities formed by at least part of said scene media and/or said scene light comprise at least one of a scene characteristic, a scene surface, a scene feature and a scene object; and
determining one or more updated scene reconstruction data elements using (a) one or more of the sensed pixel data elements, (b) one or more initial scene reconstruction data elements, and (c) a light transport equation representing the scene light flowing in equilibrium into, out of and within the scene, wherein the scene light flowing within is represented in the light transport equation which transforms incident scene light to exitant scene light by way of a light interaction function in accordance with the one or more initial scene reconstruction data elements,
wherein
scene reconstruction data elements comprise i) scene media data elements representing a matter field of the scene comprising geometric and material properties of said scene media; ii) light data elements representing a light field of the scene comprising geometric and radiometric characteristics of said scene light; and iii) camera data elements representing said camera poses,
initial scene reconstruction data elements are scene reconstruction data elements that serve as input to the determining process if they exist, and
updated scene reconstruction data elements are scene reconstruction data elements output from the determining process.
32. The non-transitory computer program storage media of claim 31 , wherein:
acquiring one or more sensed digital images comprises capturing and/or accessing one or more sensed digital images,
determining one or more updated scene reconstruction data elements comprises using non-linear optimization and/or using machine learning, and
at least one of said scene entities is represented by at least one of said scene reconstruction data elements using at least one of said sensed pixel data elements.
33. The non-transitory computer program storage media of claim 32 , wherein the executed program instructions effect a method further comprising:
accessing at least one user-identified scene reconstruction goal;
repeating the process of claim 32 until the scene reconstruction goal is satisfied, and
generating output data representing at least one of (aa) at least one scene entity associated at least in part with the user-identified goal, (bb) at least one camera data element representing at least part of said camera poses and at least part of a camera model, and (cc) at least one camera data element representing at least part of said camera model, wherein the output data is determined at least in part using at least one scene reconstruction data element.
34. The non-transitory computer program storage media of claim 32 , wherein:
at least one of the sensed digital images is a substantially omnidirectional image sensed by at least one camera.
35. The non-transitory computer program storage media of claim 32 , wherein at least one of the sensed pixel data elements represents a plurality of characteristics of polarized light, and at least one of the scene reconstruction data elements is determined at least in part using characteristics of polarized light.
36. The non-transitory computer program storage media of claim 35 , wherein:
the at least one scene reconstruction data element is determined at least in part using characteristics of polarized light representing at least one spatially localized scene feature sensitive to polarized light.
37. The non-transitory computer program storage media of claim 33 , wherein the at least one scene reconstruction data element represents the at least one spatially localized scene feature sensitive to polarized light and is used at least in part to perform any one or any combination of:
labeling scene features across a plurality of images,
image registration,
scene initialization, and
scene reconstruction.
38. The non-transitory computer program storage media of claim 35 , wherein:
the scene surface is a vehicle surface,
the scene feature is damage to the vehicle surface comprising at least one of a dent, an anomaly, and/or debris, and
one or more characteristics of the damage are determined using any one or any combination of
at least one of the scene reconstruction data elements,
a 3D model of the associated undamaged vehicle surface,
non-linear optimization, and
a machine learning model.
39. The non-transitory computer program storage media of claim 32 , wherein:
the scene media comprises one or more volumetric scene media subregions comprising homogeneous or heterogeneous media,
the scene light interacts with the scene media subregions by phenomena comprising any one of or any combination of absorption, emission, reflection, transmission, and scattering,
the scene media data elements representing geometric properties of the scene media subregions comprise at least one of a value, a distance, an angle, a point, a vector, a plane, a surfel, a voxel, and a reference, and
the scene media data elements representing material properties of the scene media subregions comprise at least one of a value, a gradient, a bi-directional reflectance distribution function (BRDF), a bi-directional light interaction function (BLIF), and a reference.
40. The non-transitory computer program storage media of claim 32 , wherein:
the scene light flows in at least one frusta,
the scene light interacts with the scene media by phenomena comprising any one or any combination of absorption, emission, reflection, transmission, and scattering,
the scene light data elements representing geometric properties of the frusta comprise at least one of a value, a point, a voxel, a direction, an angle, a vector, a solid angle, and a reference, and
the scene light data elements representing radiometric properties of the frusta comprising at least one of a value, a gradient, a radiant flux, a radiant power, and a reference.
41. The non-transitory computer program storage media of claim 32 , wherein:
at least one of the geometric characteristics of the scene media form an octree subdivided into octree cells organized in a spatially sorted and hierarchical manner.
42. The non-transitory computer program storage media of claim 41 , wherein:
the geometric characteristics of the scene light form a solid-angle octree,
directions at a point are subdivided into solid-angle elements, and
the solid-angle elements are organized in a spatially sorted and hierarchical manner.
43. The non-transitory computer program storage media of claim 42 , further comprising computing synthetic images with perspective projection using at least in part the solid-angle octree, wherein the computations use shift operations rather than general purpose divide operations.
44. The non-transitory computer program storage media of claim 43 , further comprising using spans to perform perspective projection.
45. The non-transitory computer program storage media of claim 44 , wherein:
at least one excludable octree cell is excluded from processing during computation of the synthetic images,
determination of the at least one excludable octree cell comprises any one or any combination of: i) determining occluded octree cells using a quadtree or equivalent data structure that enables a ‘front-to-back’ traversal of the scene media for accessing occlusion, and ii) determining lesser-sized octree cells using a comparison of a window size with respect to a size that may vary within a synthetic image computation that is determined at least in part with respect to a pixel size, wherein the window size is determined at least in part with respect to a perspective projection onto a display plane.