Photoelectric conversion device, movable apparatus, processing method, and storage medium for generating a differential image
A photoelectric conversion device according to one embodiment of the present disclosure has a plurality of pixels, a control unit, and a generating unit. Each pixel of the plurality of pixels includes a sensor unit that emits a pulse at a frequency corresponding to a photon reception frequency, a first counter and a second counter that count a number of the pulses, and a first memory that stores the count value for the first counter, and a second memory that stores the count value for the second counter. The control unit controls the plurality of pixels such that during a read-out operation for the count values that have been accumulated during a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, a count value for a first accumulation period is read out before an end time of a second accumulation period that is longer than the first accumulation period, and performs control such that a read-out timing for each accumulation period becomes different between the first counter and the second counter. The generating unit generates a differential image based on a difference in signals corresponding to the count values that are output from the first memory and the second memory respectively.
1 . A photoelectric conversion device comprising:
a plurality of pixels, each of which is configured to include an avalanche photodiode that generates a pulse at a frequency corresponding to a photon reception frequency, a first counter and a second counter that both count a number of the pulses, a first memory that stores count values for the first counter, and a second memory that stores count values for the second counter; and
one or more hardware processors; and
one or more memories storing instructions that can be executed by the one or more hardware processors, the instructions including instructions for:
controlling the plurality of pixels such that in a read-out operation for the count values that have been accumulated during a plurality of different accumulation periods within a full frame period from each of the first counter and the second counter, a count value for a first accumulation period is read out up until an end time of a second accumulation period that is longer than the first accumulation period, and to perform control such that read-out timings for each of the accumulation periods are different for the first counter and the second counter;
generating a differential image based on a difference in signals corresponding to the count values that are output from the first memory and the second memory respectively;
performing recognition processing for a subject based on an image signal that is output from the plurality of pixels; and
performing switching between a first mode in which the recognition processing is performed on the differential image and a second mode in which the recognition processing is performed on an image signal that is output from the plurality of pixels without performing output processing for the differential image.
2 . The photoelectric conversion device according to claim 1 ,
wherein the instructions further include instructions for performing control such that as the timings for the read-outs, the count value for the first counter is read out at a first timing, and the count value for the second counter is read out at a second timing that is delayed from the first timing by a time less than the first accumulation period.
3 . The photoelectric conversion device according to claim 1 ,
wherein intervals between the read-out timings corresponding to each of the first counter and the second counter are fixed.
4 . The photoelectric conversion device according to claim 1 ,
wherein in each of the first counter and the second counter, the first accumulation period and the second accumulation period overlap.
5 . The photoelectric conversion device according to claim 1 ,
wherein in each of the first counter and the second counter, the first accumulation period and the second accumulation period start at the same time.
6 . The photoelectric conversion device according to claim 1 ,
wherein in each of the first counter and the second counter, an end time of the second accumulation period coincides with an end time of the full frame period.
7 . The photoelectric conversion device according to claim 1 ,
wherein the instructions further include instructions for displaying a signal that is generated during at least the second accumulation period as an image.
8 . The photoelectric conversion device according to claim 1 ,
wherein the instructions further include instructions for performing switching between the first mode and the second mode based on at least one of control information for a movable apparatus and movement information for the subject.
9 . A processing method for a photoelectric conversion device that includes a plurality of pixels, each of which is configured to include an avalanche photodiode that generates a pulse at a frequency corresponding to a photon reception frequency, a counter that counts a number of the pulses, and a memory that stores count values for the counter, the method comprising:
controlling the plurality of pixels such that in a read-out operation for the count values that have been accumulated during a plurality of different accumulation periods within a full frame period from each of a first counter and a second counter, wherein the counter includes the first counter and the second counter, a count value for a first accumulation period is read out up until an end time of a second accumulation period that is longer than the first accumulation period;
performing control such that read-out timings for each of the accumulation periods are different for the first counter and the second counter;
generating a differential image based on a difference in signals corresponding to the count values that are output from a first memory and a second memory respectively, wherein the memory includes the first memory which stores the count value for the first counter and the second memory which stores the count value for the second counter;
performing recognition processing for a subject based on an image signal that is output from the plurality of pixels; and
performing switching between a first mode in which the recognition processing is performed on the differential image and a second mode in which the recognition processing is performed on an image signal that is output from the plurality of pixels without performing output processing for the differential image.
10 . A non-transitory storage medium storing a control program of a photoelectric conversion device that includes a plurality of pixels, each of which is configured to include an avalanche photodiode that generates a pulse at a frequency corresponding to a photon reception frequency, a counter that counts a number of the pulses, and a memory that stores count values for the counter, the control program causing a computer to perform each step of a processing method for the photoelectric conversion device, the method comprising:
controlling the plurality of pixels such that in a read-out operation for the count values that have been accumulated during a plurality of different accumulation periods within a full frame period from each of a first counter and a second counter, wherein the counter includes the first counter and the second counter, a count value for a first accumulation period is read out before an end time of a second accumulation period that is longer than the first accumulation period;
performing control such that read-out timings for each of the accumulation periods are different for the first counter and the second counter;
generating a differential image based on a difference in signals corresponding to the count values that are output from a first memory and a second memory, respectively wherein the memory includes a first memory, which stores the count value for the first counter, and a second memory that stores the count value for the second counter;
performing recognition processing for a subject based on an image signal that is output from the plurality of pixels; and
performing switching between a first mode in which the recognition processing is performed on the differential image and a second mode in which the recognition processing is performed on an image signal that is output from the plurality of pixels without performing output processing for the differential image.