Image sensor

1 . An image sensor comprising:

a plurality of pixels; and

processing circuitry configured to,

generate first image data by converting optical signals received by each of the plurality of pixels during a first exposure time into electrical signals,

generate event data including information related to a change in intensity of the optical signals received by each of the plurality of pixels during the first exposure time, the event data having a frame rate greater than a frame rate of the first image data,

receive the first image data and the event data, and

generate output data by sequentially packing the first image data and the event data corresponding to the first exposure time,

wherein the processing circuitry is further configured to,

generate second image data corresponding to a plurality of sub-frames of an image frame, the plurality of sub-frames corresponding to a plurality of sub-exposure times included in the first exposure time,

merge at least a portion of the second image data corresponding to each of the plurality of sub-frames to generate the first image data, and

generate the event data corresponding to each of the plurality of sub-frames based on the second image data generated from the plurality of sub-frames and the second image data generated from at least one reference frame continuous with each of the plurality of sub-frames.

2 . The image sensor of claim 1 , further comprising:

a memory configured to store the second image data.

3 . The image sensor of claim 2 , wherein the processing circuitry is further configured to:

receive the at least a portion of the second image data corresponding to each of the plurality of sub-frames from the memory.

4 . The image sensor of claim 1 , wherein the processing circuitry is further configured to:

generate conversion image data corresponding to each of the plurality of sub-frames based on the second image data corresponding to each of the plurality of sub-frames;

generate difference data corresponding to each of the plurality of sub-frames, the difference data representing differences between the conversion image data of each sub-frame and conversion image data of a reference frame of the respective sub-frame; and

generate the event data corresponding to each of the plurality of sub-frames based on the difference data.

5 . The image sensor of claim 4 , wherein the processing circuitry is further configured to:

convert the second image data corresponding to each of the plurality of sub-frames into gray scale;

convert a result of the conversion into gray scale into a log scale; and

generate the conversion image data corresponding to each of the plurality of sub-frames.

6 . The image sensor of claim 4 , wherein the processing circuitry is further configured to:

generate the event data corresponding to each of the plurality of sub-frames by comparing the difference data corresponding to each of the plurality of sub-frames with a desired threshold value.

7 . The image sensor of claim 4 , wherein the processing circuitry is further configured to:

compare the difference data corresponding to each of the plurality of sub-frames with a desired threshold value in units of pixels, wherein a size of the event data corresponding to the unit of pixels is 1 bit.

8 . The image sensor of claim 1 , wherein the processing circuitry is further configured to:

group the second image data of each of the plurality of sub-frames in units of k pixels, wherein k is a positive integer greater than or equal to 2; and

in units of grouped pixels, generate the event data corresponding to each of the plurality of sub-frames based on the second image data corresponding to each of the plurality of sub-frames and the second image data of a reference frame corresponding to each of the plurality of sub-frames.

9 . The image sensor of claim 1 , wherein

the plurality of pixels comprise,

at least one complementary metal-oxide semiconductor (CMOS) image sensor (CIS) pixel configured to convert the optical signals into the electrical signals, and

at least one dynamic vision sensor (DVS) pixel configured to sense the change in intensity of the optical signals; and

the processing circuitry is further configured to,

generate the first image data using the at least one CIS pixel of the plurality of pixels, and

generate the event data using the at least one DVS pixel.

10 . The image sensor of claim 9 , wherein the processing circuitry is further configured to:

generate the event data in units of sub-frames at desired time points; and

generate the output data by sequentially packing the first image data and the event data during the first exposure time.

11 . An image sensor comprising:

processing circuitry configured to generate initial image data in units of sub-frames by reading out electrical signals from a pixel array, the pixel array including a plurality of pixels each configured to convert optical signals into the electrical signals; and

a memory configured to store the initial image data, wherein

the processing circuitry is further configured to,

generate first image data based on the initial image data, the generating the first image data including merging the initial image data of each of m sub-frames included in a frame according to a plurality of sub-exposure times included in a first exposure time associated with the frame, wherein m is a positive integer,

generate event data corresponding to each of the m sub-frames based on the initial image data corresponding to each of the m sub-frames, and

generate output data by sequentially packing the first image data and the event data.

12 . The image sensor of claim 11 , wherein the processing circuitry is further configured to:

merge the initial image data using a first frame rate;

generate the first image data based on the merged initial image data, the first image data having a second frame rate which is less than the first frame rate; and

pack the first image data of the second frame rate and the event data, the event data having the first frame rate.

13 . The image sensor of claim 11 , wherein

the m sub-frames comprise a first sub-frame and a second sub-frame continuous with the first sub-frame; and

the processing circuitry is further configured to generate the event data corresponding to the first sub-frame based on the initial image data corresponding to the first sub-frame and the second sub-frame.

14 . The image sensor of claim 13 , wherein the processing circuitry is further configured to:

convert the initial image data corresponding to the first sub-frame and the initial image data corresponding to the second sub-frame to generate conversion image data;

generate difference data representing differences between the conversion image data corresponding to the first sub-frame and the conversion image data corresponding to the second sub-frame; and

generate the event data corresponding to the first sub-frame based on the difference data.

15 . The image sensor of claim 14 , wherein the processing circuitry is further configured to:

convert the initial image data corresponding to the first sub-frame and the initial image data corresponding to the second sub-frame into gray scale;

convert a result of the conversion into gray scale into a log scale; and

generate the conversion image data corresponding to the first sub-frame and the conversion image data corresponding to the second sub-frame.

16 . The image sensor of claim 14 , wherein

the difference data includes difference values corresponding to each pixel of the plurality of pixels; and

the processing circuitry is further configured to:

compare the difference values corresponding to each pixel of the plurality of pixels with a desired threshold value to generate the event data corresponding to the first sub-frame.

17 . The image sensor of claim 13 , wherein the processing circuitry is further configured to:

group the initial image data corresponding to the first sub-frame and the initial image data corresponding to the second sub-frame in units of k pixels, wherein k is a positive integer greater than or equal to 2; and

generate the event data corresponding to the first sub-frame based on the grouped initial image data corresponding to the first sub-frame and the grouped initial image data corresponding to the second sub-frame.

18 . An image sensor comprising:

a pixel array including at least one complementary metal-oxide semiconductor (CMOS) image sensor (CIS) pixel and at least one dynamic vision sensor (DVS) pixel, the at least one CIS pixel configured to convert optical signals into electrical signals, and the at least one DVS pixel configured to generate event signals by sensing a change in intensity of the optical signals; and

processing circuitry configured to,

generate first image data based on the electrical signals, the first image data having a first frame rate,

generate event data based on the event signals, the event data having a second frame rate higher than the first frame rate of the first image data, and

generate output data by sequentially packing the first image data and the event data corresponding to an exposure time of the first image data-,

wherein the processing circuitry is further configured to,

generate second image data corresponding to a plurality of sub-frames of an image frame, the plurality of sub-frames corresponding to a plurality of sub-exposure times included in a first exposure time associated with the first image data,

merge at least a portion of the second image data corresponding to each of the plurality of sub-frames to generate the first image data, and

generate the event data corresponding to each of the plurality of sub-frames based on the second image data generated from the plurality of sub-frames and the second image data generated from at least one reference frame continuous with each of the plurality of sub-frames.

19 . The image sensor of claim 18 , wherein the processing circuitry is further configured to:

generate conversion image data corresponding to each of the plurality of sub-frames based on the second image data corresponding to each of the plurality of sub-frames;

generate difference data corresponding to each of the plurality of sub-frames, the difference data representing differences between the conversion image data of each sub-frame and conversion image data of a reference frame of the respective sub-frame; and

generate the event data corresponding to each of the plurality of sub-frames based on the difference data.

20 . The image sensor of claim 19 , wherein the processing circuitry is further configured to:

convert the second image data corresponding to each of the plurality of sub-frames into gray scale;

convert a result of the conversion into gray scale into a log scale; and

generate the conversion image data corresponding to each of the plurality of sub-frames.