Image sensor, data processing device, and image sensor system
The present disclosure relates to an image sensor, a data processing device, and an image sensor system capable of further improving versatility. An image sensor includes: an event detecting unit configured to detect an occurrence of an event that is a luminance change of light received by a photodiode; and a data transmitting unit configured to set event data representing details of the event as a part of payload data and transmit line information that is added to a line and relates to the event data in a frame structure in which the line information is stored in a beginning part of the payload data. In addition, a luminance detecting unit configured to detect luminance of light received by the photodiode and output a luminance signal representing a luminance value and an additional information generating unit configured to generate the line information added to a line as additional information that is additionally disposed in the event data on the basis of the event data are further included. The present technology for example, can be applies to an event-based vision sensor (EVS).
1 . An image processing device comprising:
an event detecting circuit configured to detect an occurrence of an event that is a luminance change of light received by a photodiode; and
a data transmitting circuit configured to set event data representing details of the event as a part of payload data and transmit a line information that is added to a line and relates to the event data in a frame structure in which the line information is stored in a beginning part of the payload data,
wherein the line information includes a compression technique for compressing the event data.
2 . The image processing device according to claim 1 , wherein additional line information is stored as embedded data specified in CSI-2 or SLVS-EC.
3 . The image processing device according to claim 1 , wherein the line information includes at least one of a time stamp, coordinates of the line, or an event polarity of the line.
4 . The image processing device according to claim 1 , wherein the line information includes an event detection threshold of the line.
5 . The image processing device according to claim 1 , wherein the line information includes flicker information of the line that is generated on a basis of the event data.
6 . An image processing device comprising:
an event detecting circuit configured to detect an occurrence of an event that is a luminance change of light received by a photodiode; and
a data transmitting circuit configured to set event data representing details of the event as a part of payload data and transmit a line information that is added to a line and relates to the event data in a frame structure in which the line information is stored in a beginning part of the payload data,
wherein the line information includes a data amount of the line.
7 . The image processing device according to claim 1 , further comprising:
a luminance detecting circuit configured to detect a luminance of light received by the photodiode and output a luminance signal representing a luminance value; and
an additional information generating circuit configured to generate the line information as additional information that is additionally disposed in the event data on a basis of the event data,
wherein the event detecting circuit acquires a difference between the luminance value represented by the luminance signal and a predetermined reference value, and in a case in which the difference exceeds an event detection threshold of a positive side or an event detection threshold of a negative side, detects an occurrence of the event, and outputs the event data representing details of the event.
8 . The image processing device according to claim 7 , wherein, in a case in which the event detecting circuit is an arbiter type, the additional information generating circuit has a frame generating circuit generating a frame from the event data that is output at a timing at which an event has occurred from the event detecting circuit.
9 . The image processing device according to claim 7 ,
wherein the additional information generating circuit sets area information corresponding to an area set in an image formed from the event data for each row of the image and transmits the set area information and the event data that is area data corresponding to the area for each row, and
wherein information representing a position of a row and information representing a position of a column of the area included in the row are included in the area information.
10 . The image processing device according to claim 7 , wherein the luminance detecting circuit includes an event data acquisition circuit for acquiring event data and an image data acquisition circuit for acquiring image data, and the event data acquisition circuit and the image data acquisition circuit are arranged on a same pixel chip.
11 . The image processing device according to claim 1 , wherein the line information includes data after compression.
12 . The image processing device according to claim 4 , wherein the line information includes an event number of the line.
13 . The image processing device according to claim 1 , wherein the data transmitting circuit is configured to operate in a first mode and a second mode, the first mode implementing a data amount of a pixel of two bits, and the second mode implementing a data amount of a pixel of other than two bits.
14 . The image processing device according to claim 1 , wherein the data transmitting circuit is configured to implement a data amount of the event data of two bits or three bits.
15 . The image processing device according to claim 1 , wherein the frame structure for the image processing device is configured to store the line information of a frame according to a first virtual channel value, and another image processing device is configured to store the line information of a frame according to a second virtual channel value.
16 . The image processing device according to claim 1 , wherein the data transmitting circuit is configured to transmit frame information that is added to a frame as additional information that is disposed in the event data in the frame structure.
17 . The image processing device according to claim 7 , wherein the additional information includes a time stamp or a frame number relating to the event data.
18 . The image processing device according to claim 1 , wherein the data transmitting circuit is configured to embed pixel information in the event data.
19 . The image processing device according to claim 1 , wherein the event detecting circuit is configured to perform a region of interest (ROI) process that acquires coordinate information of an area desired to be obtained, and to output the coordinate information of the area.
20 . The image processing device according to claim 1 , wherein the event detecting circuit is configured to perform a target object recognition process for a target object that has generated the event.
21 . The image processing device according to claim 1 , wherein the event detecting circuit is configured to output outputs distance information representing a distance to a target that is acquired in an exposure/auto focus (AE/AF) process.
22 . The image processing device according to claim 1 , wherein the event detecting circuit is configured to perform a gesture detecting process.
23 . The image processing device according to claim 1 , wherein the event detecting circuit is configured to output movement amount information representing a movement amount of a subject per unit time, the movement amount information being acquired in a deblur process corresponding to the subject.
24 . An image processing device comprising:
an event detecting circuit configured to detect an occurrence of an event that is a luminance change of light received by a photodiode; and
a data transmitting circuit configured to set event data representing details of the event as a part of payload data and transmit a line information that relates to the event data in a frame structure in which the line information is stored in a beginning part of the payload data,
wherein the line information includes a compression technique for compressing the event data, and
the line information is stored in a packet header of a packet storing the event data specified in CSI-2 or SLVS-EC.
25 . An image processing device comprising:
an event detecting circuit configured to detect an occurrence of an event that is a luminance change of light received by a photodiode;
a data transmitting circuit configured to set event data representing details of the event as a part of payload data and transmit a line information that is added to a line and relates to the event data in a frame structure in which the line information is stored in a beginning part of the payload data;
a luminance detecting circuit configured to detect a luminance of light received by the photodiode and output a luminance signal representing a luminance value;
an additional information generating circuit configured to generate the line information as additional information that is additionally disposed in the event data on a basis of the event data,
wherein the event detecting circuit acquires a difference between the luminance value represented by the luminance signal and a predetermined reference value, and in a case in which the difference exceeds an event detection threshold of a positive side or an event detection threshold of a negative side, detects an occurrence of the event, and outputs the event data representing details of the event; and
a pixel chip in which a first luminance detecting circuit used for acquisition of the event data and a second luminance detecting circuit used for acquisition of image data are disposed as the luminance detecting circuit,
wherein the additional information generating circuit generates a frame number used for synchronizing the event data with the image data and interference information used for identifying the event data acquired at approximately a same timing as a timing at which the second luminance detecting circuit operates as the additional information that is additionally disposed in the event data.
26 . The image processing device according to claim 25 , wherein an operation of the first luminance detecting circuit for event data acquisition is stopped at an interval between a first operation of the second luminance detecting circuit for image data acquisition for a first frame and a second operation of the second luminance detecting circuit for image data acquisition of a next frame that follows the first frame.
27 . The image processing device according to claim 25 , wherein the frame number and the interference information are generated in units of lines of the event data.
28 . The image processing device according to claim 25 , wherein, in a case in which the first luminance detecting circuit and the second luminance detecting circuit do not operate at approximately the same timing over one frame, the frame number is included in frame information in units of frames.
29 . The image processing device according to claim 25 , wherein the additional information generating circuit has:
a pixel accessing circuit configured to perform an access to the second luminance detecting circuit and control a timing at which the second luminance detecting circuit operates; and
an event accessing circuit configured to generate the frame number and the interference information on the basis of a timing at which the pixel accessing circuit controls an operation of the second luminance detecting circuit together with generating a time stamp, coordinates of the line, and an event polarity of the line used as the line information.