Methods and apparatus for light field photography
In exemplary implementations of this invention, a light field camera uses a light field dictionary to reconstruct a 4D light field from a single photograph. The light field includes both angular and spatial information and has a spatial resolution equal to the spatial resolution of the imaging sensor. Light from a scene passes through a coded spatial light modulator (SLM) before reaching an imaging sensor. Computer processors reconstruct a light field. This reconstruction includes computing a sparse or compressible coefficient vector using a light field dictionary matrix. Each column vector of the dictionary matrix is a light field atom. These light field atoms each, respectively, comprise information about a small 4D region of a light field. Reconstruction quality may be improved by using an SLM that is as orthogonal as possible to the dictionary.
1. A method comprising, in combination:
(a) using an imaging sensor to take measurements of light from a first scene, which light has, after leaving the first scene, passed through or been reflected from a SLM; and
(b) using one or more computer processors to use the measurements of the light as inputs to perform calculations to compute a first light field function of the light;
wherein
(i) the calculations include multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix, which light field dictionary matrix has, before the imaging sensor takes the measurements, been computed from training sets, which training sets comprise or were extracted from light field functions of light from other scenes, the first scene being at least partially different in appearance than the other scenes,
(ii) the first light field function has a first spatial resolution, the imaging sensor has a second spatial resolution, and the first and second spatial resolutions are equal,
(iii) the measurements are taken during a single exposure, and
(iv) the calculations do not use, as inputs, any sensor readings by the imaging sensor during any exposure other than the single exposure, except to the extent, if any, that any prior sensor readings by the imaging sensor affect any information contained in the light field dictionary or affect any information, not specific to the first scene, contained in any measurement matrix.
2. A method comprising, in combination:
(a) using an imaging sensor to take measurements of light from a first scene, which light has, after leaving the first scene, passed through or been reflected from a SLM; and
(b) using one or more computer processors to use the measurements of the light as inputs to perform calculations to compute a first light field function of the light;
wherein
(i) the calculations include multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix, which light field dictionary matrix has, before the imaging sensor takes the measurements, been computed from training sets, which training sets comprise or were extracted from light field functions of light from other scenes, the first scene being at least partially different in appearance than the other scenes, and
(ii) the SLM is an Optimized SLM with respect to the light field dictionary matrix.
3. A method comprising, in combination:
(a) using an imaging sensor to take measurements of light from a first scene, which light has, after leaving the first scene, passed through or been reflected from a SLM; and
(b) using one or more computer processors to use the measurements of the light as inputs to perform calculations to compute a first light field function of the light;
wherein
(i) the calculations include multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix, which light field dictionary matrix has, before the imaging sensor takes the measurements, been computed from training sets, which training sets comprise or were extracted from light field functions of light from other scenes, the first scene being at least partially different in appearance than the other scenes,
(ii) the calculations further include computing a coefficient vector αε d that satisfies an equation i=Φl=ΦDα, where Dε n×d is the light field dictionary matrix, iε m is a matrix that comprises information about a sensor image, lε n is a matrix that is the first light field function, and Φ is a matrix that comprises information about a projection of light, modulated by the SLM, onto the imaging sensor,
(iii) a first light field atom in the light field dictionary matrix comprises information about one or more light rays, or absence of light rays, in a first spatial region,
(iv) a second light field atom in the light field dictionary matrix comprises information about one or more light rays, or absence of light rays, in a second spatial region, and
(v) the first and second spatial regions partially overlap each other.
4. A method comprising, in combination:
(a) using an imaging sensor to take measurements of light from a first scene, which light has, after leaving the first scene, passed through or been reflected from a SLM; and
(b) using one or more computer processors to use the measurements of the light as inputs to perform calculations to compute a first light field function of the light;
wherein
(i) the calculations include multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix, which light field dictionary matrix has, before the imaging sensor takes the measurements, been computed from training sets, which training sets comprise or were extracted from light field functions of light from other scenes, the first scene being at least partially different in appearance than the other scenes,
(ii) the calculations include a first computation of light field information for a first spatial area, and also include a second computation of light field information for a second spatial area, the first and second spatial areas being different, but not necessarily disjoint, sets of points, and
(iii) the first and second computations are performed in parallel.
5. A method comprising, in combination:
(a) using an imaging sensor to take measurements of light that has passed through or been reflected from a SLM; and
(b) using one or more computer processors to perform calculations to compute a light field function of light incident on the imaging sensor;
wherein
(i) the calculations include
(A) multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix, and
(B) computing a coefficient vector αε d that satisfies an equation i=Φl=ΦDα, where Dε n×d is the light field dictionary matrix, iε m is a matrix that comprises information about a sensor image, lε n is a matrix that is the light field function, and Φ is a matrix that comprises information about a projection of light, modulated by the SLM, onto the imaging sensor, and
(ii) the SLM is an Optimized SLM with respect to the light field dictionary matrix.
6. A method comprising, in combination:
(a) using an imaging sensor to take measurements of light that has passed through or been reflected from a SLM; and
(b) using one or more computer processors to perform calculations to compute a light field function of light incident on the imaging sensor;
wherein
(i) the calculations include
(A) multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix, and
(B) computing a coefficient vector αε d that satisfies an equation i=Φl=ΦDα, where Dε n×d is the light field dictionary matrix, iε m is a matrix that comprises information about a sensor image, lε n is a matrix that is the light field function, and Φ is a matrix that comprises information about a projection of light, modulated by the SLM, onto the imaging sensor,
(ii) the imaging sensor detects light during multiple exposures, and
(iii) the one or more processors use data collected by the imaging sensor during the multiple exposures to compute a light field function.
7. A method comprising, in combination:
(a) using an imaging sensor to take measurements of light that has passed through or been reflected from a SLM; and
(b) using one or more computer processors to perform calculations to compute a light field function of light incident on the imaging sensor;
wherein
(i) the calculations include
(A) multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix, and
(B) computing a coefficient vector αε d that satisfies an equation i=Φl=ΦDα, where Dε n×d is the light field dictionary matrix, iε m is a matrix that comprises information about a sensor image, lε n is a matrix that is the light field function, and Φ is a matrix that comprises information about a projection of light, modulated by the SLM, onto the imaging sensor,
(ii) a first light field atom in the light field dictionary matrix comprises information about one or more light rays, or about absence of light rays, in a first spatial region,
(iii) a second light field atom in the light field dictionary matrix comprises information about one or more light rays, or about absence of light rays, in a second spatial region, and
(iv) the first and second spatial regions partially overlap each other.
8. A method comprising, in combination:
(a) using an imaging sensor to take measurements of light that has passed through or been reflected from a SLM; and
(b) using one or more computer processors to perform calculations to compute a light field function of light incident on the imaging sensor;
wherein
(i) the calculations include
(A) multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix,
(B) computing a coefficient vector αε d that satisfies an equation i=Φl=ΦDα, where Dε n×d is the light field dictionary matrix, iε m is a matrix that comprises information about a sensor image, lε n is a matrix that is the light field function, and Φ is a matrix that comprises information about a projection of light, modulated by the SLM, onto the imaging sensor, and
(C) a first computation of light field information for a first spatial area, and
(D) a second computation of light field information for a second spatial area,
(ii) the first and second spatial regions partially overlap each other, and
(iii) the first and second computations are performed in parallel.
9. An apparatus comprising, in combination:
(a) a SLM;
(b) an imaging sensor configured to take measurements of light from a first scene, which light has, after leaving the first scene, passed through or been reflected from the SLM; and
(c) one or more computer processors configured to use the measurements of the light as inputs to perform calculations to compute a first light field function of the light;
wherein
(i) the calculations include multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix, which light field dictionary matrix has, before the measurements, been computed from training sets, which training sets comprise or were extracted from light field functions of light from other scenes, the first scene being at least partially different in appearance than the other scenes, and
(ii) the SLM is an Optimized SLM with respect to the light field dictionary matrix.
10. An apparatus comprising, in combination:
(a) a SLM;
(b) an imaging sensor configured to take measurements of light that has passed through or been reflected from the SLM; and
(c) one or more computer processors configured to perform calculations to compute a light field function of light incident on the imaging sensor;
wherein
(i) the calculations include
(A) multiplying a light field dictionary matrix or multiplying a light field atom in the light field dictionary matrix, and
(B) computing a coefficient vector αε d that satisfies an equation i=Φl=ΦDα, where Dε n×d is the light field dictionary matrix, iε m is a matrix that comprises information about a sensor image, lε n is a matrix that is the light field function, and Φ is a matrix that comprises information about a projection of light, modulated by the SLM, onto the imaging sensor, and
(ii) the SLM is an Optimized SLM with respect to the light field dictionary matrix.