Time interleaved exposures and multiplexed illumination
View Patent ↗Samples of a scene are acquired in synchronization with finely interleaved varying conditions such as lighting, aperture, focal length, and so forth. A time-varying sample at each pixel for each condition is reconstructed. Time multiplexed interleaving allows for real-time, live applications, while handling motion blur naturally. Effects such as real-time video relighting of a scene is possible. Time interleaved exposures also allow segmentation of a scene, and allows for constrained special effects such as triangulation matting using a multi-color interleaved chroma key.
1. One or more computer-readable storage devices storing instructions that, when executed by a processor, cause the processor to perform acts comprising:
illuminating a portion of a scene with different interleaved lighting states, wherein each of a plurality of the different interleaved lighting states illuminates the portion of the scene a plurality of times, and wherein a same portion of the scene is illuminated by each of the plurality of the different interleaved lighting states;
collecting data from the scene with an image sensor comprising a plurality of pixels, the data corresponding to each of the different interleaved lighting states;
reconstructing a time-varying signal from the data using a reconstruction filter, the reconstructing based at least in part on data collected from a plurality of times during which the scene is illuminated with a particular interleaved lighting state; and
generating, based at least in part upon the reconstructed time-varying signal, an output image of the scene including motion blur for each of the plurality of different interleaved lighting states.
2. The one or more computer-readable storage devices of claim 1 , wherein reconstructing the time-varying signal using the reconstruction filter comprises determining a weighted sum of pixel values for the collected data of interleaved lighting states over a pre-determined amount of time.
3. The one or more computer-readable storage devices of claim 1 , wherein the different interleaved lighting states comprise an ambient light state and a state that includes illumination from one or more illuminators.
4. The one or more computer-readable storage devices of claim 3 , wherein the state that includes illumination from one or more illuminators comprises monochromatic illuminations.
5. The one or more computer-readable storage devices of claim 1 , wherein the illuminating comprises cycling the plurality of the different interleaved lighting states a plurality of times at a rate of at least about 100 times per second.
6. A system comprising:
an imaging device comprising a plurality of pixels;
an illuminator; and
a computing device coupled to the imaging device and the illuminator and configured to:
cycle illumination of a portion of a scene between pre-determined illumination states a plurality of times at a rate of at least about 100 times per second, wherein the portion of the scene is illuminated with each of a plurality of the pre-determined illumination states a plurality of times, and wherein a same portion of the scene is illuminated with each of the plurality of the pre-determined illumination states;
synchronize with the plurality of pre-determined illumination states and sample at least a portion of the plurality of pixels during at least a portion of the pre-determined illumination states to create a plurality of samples; and
reconstruct a time-varying signal from the plurality of samples.
7. The system of claim 6 , wherein the reconstruction comprises a summation of at least a portion of the plurality of samples from same illumination states.
8. The system of claim 6 , wherein the imaging device is analog and configured to pre-process image data and output low-speed image data.
9. The system of claim 6 , wherein the plurality of pixels comprise imaging elements configured to be alternatively and selectively coupled to a first charge accumulator or a second charge accumulator, and each of the plurality of samples comprises a charge stored in one of the first or second charge accumulators and read out by the imaging device after a plurality of alternations.
10. The system of claim 6 , wherein the plurality of pixels each comprise a first imaging element coupled alternately between a first charge accumulator and a second imaging element coupled to a second charge accumulator, and each of the plurality of samples comprises a charge stored in one of the first or second charge accumulators and read out by the imaging device after a plurality of alternations.
11. The system of claim 6 , wherein the illuminator is configured to generate a plurality of different colors and each of the plurality of pre-determined illumination states comprise monochromatic illumination.
12. A method comprising:
generating interleaved image data of a scene, the generating comprising:
imaging the scene with a plurality of interleaved states, the scene being imaged a plurality of times in each of at least two of a plurality of lighting states;
sampling pixels in an imaging device to acquire state data associated with the each of the plurality of interleaved states; and
calculating an image based at least in part upon the generated interleaved image data, wherein the calculating the image comprises:
computing normals and reflectances for individual color channels; and
fitting a reflectance model to the individual color channels.
13. The method of claim 12 , wherein the imaging the scene with the plurality of the interleaved states comprises varying an aperture of the imaging device.
14. The method of claim 12 , wherein the imaging the scene with the plurality of the interleaved states comprises varying a color or opacity of a background of the scene.
15. The method of claim 12 , wherein the imaging the scene with the plurality of the interleaved states comprises varying a focal length of the imaging device.
16. The method of claim 12 , wherein the calculating the image further comprises computing a desired exposure based at least in part upon the fitting.
17. The method of claim 12 , wherein the calculating comprises computing a desired exposure to relight the scene.
18. The method of claim 12 , wherein the plurality of interleaved states comprises a flash state and a no-flash state, and further comprising segmenting the scene into a foreground and a background based at least in part upon one or more pixels exceeding a pre-determined threshold.
19. The method of claim 12 , wherein the imaging the scene with the plurality of the interleaved states comprises varying one or more illumination characteristics.
20. The method of claim 19 , wherein the one or more illumination characteristics comprise color, duration, intensity, or physical location of one or more illuminators relative to one or more illuminated objects.
21. The method of claim 12 , wherein the reflectance model includes a Lambertian reflectance model.