Method for color imaging using arbitrary-color-filter-array event data and image sensor
An image sensor for color imaging using arbitrary-color-filter-array event data is provided. The image sensor comprises: a plurality of color imaging pixels and a plurality of color event pixels. An image signal of a first color imaging pixel included in the plurality of color imaging pixels is determined based on a first color signal of the first color image pixel and at least a color event data of one or more color event pixels included in the plurality of color event pixels. The color event data is generated in a temporal relation to the generation of the first color signal.
1 . An image sensor, comprising:
a plurality of color imaging pixels; and
a plurality of color event pixels,
wherein an image signal of a first color imaging pixel included in the plurality of color imaging pixels is determined based on a first color signal of the first color imaging pixel and at least a color event data of a set of color event pixels included in the plurality of color event pixels, wherein the color event data is generated in a temporal relation to the generation of the first color signal, and wherein the set of color event pixels includes at least three different colors.
2 . The image sensor according to claim 1 , wherein the set of color event pixels are ones of the plurality of color event pixels being correlated to the first color imaging pixel.
3 . The image sensor according to claim 2 , wherein the set of color event pixels are ones of the plurality of color event pixels being located in close proximity to the first color imaging pixel.
4 . The image sensor according to claim 1 , wherein the color event data is generated during an integration time of the first color imaging pixel.
5 . The image sensor according to claim 4 , wherein the color event data is generated within a time interval that is the same as the integration time of the first color imaging pixel.
6 . The image sensor according to claim 1 , wherein the color event data is generated within a time interval before and/or after a time at which the first color signal is received.
7 . The image sensor according to claim 1 , wherein a first color sensed by the first color imaging pixel is the same as a color of one of the one or more color event pixels.
8 . The image sensor according to claim 1 , wherein a first color sensed by the first color imaging pixel is different from a color of each of the one or more color event pixels.
9 . The image sensor according to claim 8 , wherein image signals of each individual color imaging pixel included in the plurality of color imaging pixels are determined based on a color signal of each corresponding color imaging pixel and the color event data of the one or more color event pixels included in the plurality of color event pixels.
10 . The image sensor according to claim 1 , wherein the one or more color event pixels are correlated to each color imaging pixel of the plurality of color imaging pixels.
11 . The image sensor according to claim 1 , wherein the first color imaging pixel included in the plurality of color imaging pixels is located within a region of interest of a pixel array of the image sensor.
12 . A method of color imaging, the method comprising:
defining a target CMOS image sensor (CIS) color channel;
selecting one or more event-based vision sensor (EVS) pixels depending on proximity to the target CIS color channel;
applying a set of color conversion model parameters to an event data associated with the one or more EVS pixels to generate color space data, wherein associated with the one or more EVS pixels is captured within a time interval before or after a time at which a color image signal is received; and
fusing the color space data with the color image signal of the target CIS color channel,
wherein an error minimization process comprising:
computing differences between color CIS data of consecutive CIS frames;
computing integrals of color EVS data within time intervals between the consecutive CIS frames;
optimizing the set of color conversion model parameters that minimizes the errors between the differences and the integrals.
13 . The method according to claim 12 , wherein the set of color conversion model parameters is stored in a register bank.
14 . The method according to claim 13 , wherein the set of color conversion model parameters comprising a set of weight factors associated with the one or more EVS pixels is predetermined through the error minimization process.
15 . The method according to claim 12 , wherein imaging different illuminants requires different sets of color conversion model parameters.
16 . The method according to claim 12 , wherein the set of color conversion model parameters depends on an illumination level, a noise level or a motion speed.
17 . The method according to claim 12 , wherein the set of color conversion model parameters is in a form of one of a linear combination, a quadratic combination, a neural network or a look-up table.
18 . The method according to claim 12 , wherein the set of EVS pixels includes at least three different colors.
19 . An image system, comprising:
an image sensor having a pixel array comprising of a plurality of color imaging pixels and a plurality of color event pixel;
a host device connected to the image sensor; and
a processor programed with firmware to perform:
defining a target CIS color channel;
selecting a set of EVS pixels depending on proximity to the target CIS color channel, wherein the set of EVS pixels includes at least three different colors;
applying a set of color conversion model parameters to EVS data of the set of EVS pixels to generate color space data; and
outputting the generated color space data.
20 . The image system according to claim 19 , wherein the set of color conversion model parameters is stored in a register bank.
21 . The image system according to claim 20 , wherein the processor and the register bank are included in the image sensor.