Method and apparatus of image representation and processing for dynamic vision sensor
An apparatus and a method. The apparatus includes an image representation unit configured to receive a sequence of frames generated from events sensed by a dynamic vision sensor (DVS) and generate a confidence map from non-noise events; and an image denoising unit connected to the image representation unit and configured to denoise an image in a spatio-temporal domain. The method includes receiving, by an image representation unit, a sequence of frames generated from events sensed by a DVS, and generating a confidence map from non-noise events; and denoising, by an image denoising unit connected to the image representation unit, images formed from the frames in a spatio-temporal domain.
1. An apparatus, comprising:
an image representation unit configured to receive a sequence of frames generated from events sensed by a dynamic vision sensor (DVS), determine N 1 , events within a neighborhood and a time W of each event, if N 1 is below a threshold T 1 discard an associated event as noise, determine a neighborhood density for each non-noise event, determine confidence events as events with neighborhood densities greater than threshold T 2 , and generate a confidence map from the confidence events, where N 1 , N 2 , T 1 , and T 2 are each integers; and
an image denoising unit connected to the image representation unit and configured to determine N 2 events within the neighborhood and the time W of each event, determine C confidence events in a previous frame in an equivalent neighbourhood, determine (N 2 +(α×C)), determine noise events by comparing (N 2 +(α×C)) to a threshold T 3 , and denoise an image in a spatio-temporal domain by discarding the noise events, where α, C, and T 3 are each integers.
2. The apparatus of claim 1 , wherein each event is associated with four values representing an event state, wherein the four values include an x and a y coordinate of a pixel indicating a location of the event, either a value of +1 to indicate a positive change in luminance of the event, a value of −1 to indicate a negative change in luminance of the event, or a value of 0 to indicate no change in luminance of the event as compared to an immediately preceding event state of the associated location, and a timestamp of the event.
3. The apparatus of claim 1 , wherein the image representation unit is further configured to determine, for each event not discarded, a neighborhood density I(valid_evt) as follows;
I
(
valid
evt
)
=
{
M
max
(
R
1
,
S
)
,
when
M
≤
R
2
M
k
,
when
M
>
R
2
,
where M is a number of events within a predetermined neighborhood of the event, including the event, S is a predetermined time window, R 1 is a threshold for avoiding values of S that are less than R 1 , R 2 is a threshold when M is larger than a predetermined value, and k is a predetermined constant.
4. An apparatus, comprising:
a dynamic vision sensor (DVS) configured to generate a stream of events;
a sampling unit connected to the DVS and configured to sample the stream of events;
an image formation unit connected to the sampling unit and configured to form an image for each sample of the stream of events by determining N 1 events within a neighborhood and a time W of each event, if N 1 is below a threshold T 1 , discarding an associated event as noise, determining a neighborhood density for each non-noise event, and determining confidence events as events with neighborhood densities greater than threshold T 2 , where N 1 , N 2 , T 1 , and T 2 are each integers;
an image representation unit connected to the image formation unit and configured to generate a confidence map from confidence events;
an image undistortion unit connected to the image representation unit and configured to compensate for distortion in frames by determining N 2 events within the neighborhood and the time W of each event, determining C confidence events in a previous frame in an equivalent neighbourhood, determine (N 2 +(α×C)), determining noise events by comparing (N 2 +(α×C)) to a threshold T 3 , and denoising an image in a spatio-temporal domain by discarding the noise events, where α, C, and T 3 are each integers; and
an image matching unit connected to the image undistortion unit and the sampling unit and configured to match frames and adjust a sampling method of the sampling unit, if necessary.
5. The apparatus of claim 4 , wherein the image matching unit is further configured to match frames using weighted frame-to-frame matching as follows:
arg min½Σ i C ( p i )∥ I (1) ( p i )− I (2) ( T×p i )∥ 2 ,
where arg min ƒ(x) is an argument-of-a-minimum function that determines a value x for which a function ƒ(x) attains its minimum, where i is a location in a confidence map C (1) of a first image I (1) , p i is a coordinate for location i, I (2) is a second image, and T is a transformation matrix that minimizes the arg min function.
6. A method, comprising:
receiving, by an image representation unit, a sequence of frames generated from events sensed by a dynamic vision sensor (DVS), determining N 1 events within a neighborhood and a time W of each event, if N 1 is below a threshold T 1 , discarding an associated event as noise, determining a neighborhood density for each non-noise event, and determining confidence events as events with neighborhood densities greater than threshold T 2 , and generating a confidence map from non-noise events, where N 1 , N 2 , T 1 , and T 2 are each integers; and
denoising, by an image denoising unit connected to the image representation unit, images formed from the frames in a spatio-temporal domain by determining N 2 events within the neighborhood and the time W of each event, determining C confidence events in a previous frame in an equivalent neighbourhood, determine (N 2 +(α×C)), determining noise events by comparing (N 2 +(α×C)) to a threshold T 3 , and denoising the image in a spatio-temporal domain by discarding the noise events, where α, C, and T 3 are each integers.
7. The method of claim 6 , wherein each event is associated with four values representing an event state, wherein the four values include an x and a y coordinate of a pixel indicating a location of the event, either a value of +1 to indicate a positive change in luminance of the event, a value of −1 to indicate a negative change in luminance of the event, or a value of 0 to indicate no change in luminance of the event as compared to an immediately preceding event state of the associated location, and a timestamp of the event.
8. The method of claim 6 , further comprising determining, by the image representation unit, for each event not discarded, a neighborhood density I(valid_evt) as follows:
I
(
valid
evt
)
=
{
M
max
(
R
1
,
S
)
,
when
M
≤
R
2
M
k
,
when
M
>
R
2
,
where M is a number of events within a predetermined neighborhood of the event, including the event, S is a predetermined time window, R 1 is a threshold for avoiding values of S that are less than R 1 , R 2 is a threshold when M is larger than a predetermined value, and k is a predetermined constant.
9. A method, comprising
generating, by a dynamic vision sensor (DVS), a stream of events;
sampling, by a sampling unit connected to the DVS, the stream of events;
forming, by an image formation unit connected to the sampling unit, an image for each sample of the stream of events;
receiving, by an image representation unit connected to the image formation unit, images formed by the image formation unit and generating a confidence map from non-noise events by determining N 1 events within a neighborhood and a time W of each event, if N 1 is below a threshold T 1 , discarding an associated event as noise, determining a neighborhood density for each non-noise event, and determining confidence events as events with neighborhood densities greater than threshold T 2 , and generating the confidence map from non-noise events, where N 1 , N 2 , T 1 , and T 2 are each integers;
compensating for distortion, by an image undistortion unit connected to the image representation unit, in frames by determining N 2 events within the neighborhood and the time W of each event, determining C confidence events in a previous frame in an equivalent neighbourhood, determine (N 2 +(α×C)), determining noise events by comparing (N 2 +(α×C)) to a threshold T 3 , and denoising the image in a spatio-temporal domain by discarding the noise events, where α, C, and T 3 are each integers; and
matching, by an image matching unit connected to the image undistortion unit and the sampling unit, frames and adjusting a sampling method of the sampling unit, if necessary.
10. The method of claim 9 , wherein matching frames is comprised of weighted frame-to-frame matching as follows:
arg min½Σ i C ( p i )∥ I (1) ( p i )− I (2) ( T×p i )∥ 2 ,
where arg min ƒ(x) is an argument-of-a-minimum function that determines a value x for which a function ƒ(x) attains its minimum, where i is a location in a confidence map C (1) of a first image I (1) , p i is a coordinate for location i, I (2) is a second image, and T is a transformation matrix that minimizes the arg min function.