System and method for event-based double integral imaging
EVS event data may be stored in an event buffer as a plurality of event values, each event value corresponding to a respective event frame of the plurality of event frames. For each event frame, a processor may implement an improved EDI model to update the event-frame count, for each event frame after a first event frame, and only when a current event value is different than an immediately preceding event value: (1) updating a first integral of event data based on the current event value and all preceding event values, and (2) updating a second integral of event data based on at least the first integral of event data and the event frame count, and using at least the second integral of event data, deblurring raw image data to yield a latent deblurred image.
1 . A camera, comprising:
an image-generation sensor designated to generate a raw image data based on received light received over an exposure interval;
an event-detection sensor spatially corresponding to the image-generation sensor and designated to output event data over a plurality of event frames, each event frame corresponding to a period of the exposure interval;
memory defining:
an event buffer storing a plurality of event values, each event value corresponding to a respective event frame of the plurality of event frames based on the event data,
a first-integral buffer storing first integral of event data based on the event values,
a second-integral buffer storing second integral of event data based on the first integral of event data, and
a counter-buffer defining an event-frame count of the plurality of event frames; and
a processor configured to:
for each event frame, update the event-frame count,
for each event frame after a first event frame, only when a current event value is different than an immediately preceding event value: (1) executing read/write on the first-integral buffer to update the first integral of event data based on the current event value and all preceding event values, and (2) executing read/write on the second-integral buffer to update the second integral of event data based on at least the first integral of event data and the event frame count, and
using at least the second integral of event data, deblurring the raw image data to yield a latent deblurred image.
2 . The camera of claim 1 , comprising: a pixel array having a plurality of pixels,
the image-generation sensor being an image-generation subset of the plurality of pixels each designated to integrate the received light over the exposure interval,
the event-detection sensor being an event-detection subset of the plurality of pixels being designated to output event data over the plurality of event frames.
3 . The camera of claim 2 , the event-detection subset comprising a plurality of event-detection subsets and the image-generation subset comprising a plurality of image-generation subsets, each image-generation subset corresponding to at least one row of the plurality of pixels.
4 . The camera of claim 1 ,
the image-generation sensor comprising a plurality of image-generation subsets;
the event-detection sensor comprising a plurality of event-detection subsets.
5 . The camera of claim 4 , wherein each subset of the plurality of event-detection subsets has same event frame quantity; a single count-buffer is used for analysis of event frames corresponding to more than one of the plurality of event-detection subsets.
6 . The camera of claim 1 , the latent deblurred image corresponding to beginning of the exposure interval.
7 . The camera of claim 1 , the latent deblurred image including a plurality of latent deblurred images corresponding to respective time of corresponding event frame in which event data value indicates an event occurred.
8 . The camera of claim 1 , exposure control of the image-generation sensor and the event-detection sensor being synchronized.
9 . The camera of claim 1 , wherein update the event-frame count comprises incrementally decreasing the frame count from a max frame count to zero.
10 . The camera of claim 1 , wherein update the frame count comprises incrementally increasing the frame count from zero to a max frame count.
11 . A method for producing an image, comprising:
(a) receiving raw image data, captured using an image-generation sensor over an exposure interval;
(b) receiving event data, captured using an event-detection sensor over a plurality of event frames, each event frame corresponding to a period of the exposure interval;
(c) storing a plurality of event values in an event buffer, each event value corresponding to a respective event frame of the plurality of event frames based on the event data;
(d) for each event frame, updating an event-frame count of the plurality of event frames stored in a counter-buffer;
(e) when a current event value is different from an immediately preceding event value, (1) updating a first integral of event data using at least the current event value and all preceding event values, and (2) updating a second integral of event data using at least the first integral of event data and the event frame count; and
(f) using at least the second integral of event data, deblurring the raw image data to yield a latent deblurred image.
12 . The method of claim 11 , wherein:
the image-generation sensor comprises a plurality of image-generation subsets, and the event-detection sensor comprises a plurality of event-detection subsets; and,
a single count-buffer is used for analysis of event frames corresponding to more than one of the plurality of event-detection subsets.
13 . The method of claim 11 , the latent deblurred image corresponding to beginning of the exposure interval.
14 . The method of claim 11 , the latent deblurred image including a plurality of latent deblurred images each corresponding to respective time of the corresponding event frame.
15 . The method of claim 11 , further comprising synchronizing exposure control of the image-generation sensor and the event-detection sensor.
16 . The method of claim 11 , updating an event-frame count comprises incrementally decreasing the frame count from a max frame count to zero.
17 . The method of claim 11 ,
wherein updating first integral of event data using at least the current event value and all preceding event values comprises: summing the current event value and all preceding event values;
wherein each event value is one of a −1 for a negative change in light intensity detected by the event-detection sensor exceeding a threshold, a 0 for no change in light intensity detected by the event-detection sensor exceeding the threshold, and a +1 for a positive change exceeding the threshold in light intensity detected by the event-detection sensor.
18 . The method of claim 11 , wherein updating the second integral of event data comprises calculating:
acc( t )=event( t )!=0?acc( t −1)+dn_count( t )*(2 c′E(t) −2 c′E(t-1) ):acc( t− 1)
where c′ is a contrast threshold that defines whether an event occurs in the event data, t is the frame count, and E(t) is the first integral of event data.
19 . The method of claim 11 , wherein deblurring the raw image data to yield a latent deblurred image includes, calculating expected intensity value of each pixel in the deblurred image corresponding to time (t) based on
L
(
t
)
=
L
(
0
)
*
exp
cE
(
t
)
;
where
L
(
0
)
=
BT
′
*
(
∫
0
T
2
∧
(
c
′
E
(
t
)
)
dt
)
-
1
=
BT
′
*
acc
(
T
)
-
1
;
where BT′ is the value of pixel in the raw image data at end of exposure time T′, and c′ is a contrast threshold,
a
c
c
(
t
)
=
∫
0
c
2
∧
(
c
′
E
(
s
)
)
ds
,
and
E
(
t
)
=
∫
0
t
e
(
s
)
d
s
.
20 . The method of claim 11 , further comprising repeating (a)-(f) over additional exposure intervals to yield additional latent deblurred images; and combining the latent deblurred image and the additional latent deblurred images into a video stream.