Photodetection apparatus and electronic equipment
The present disclosure relates to a photodetection apparatus and electronic equipment that are allowed to achieve increased efficiency. Bidirectional communication between an image sensor provided with a pixel group including N pixels from among a plurality of pixels arranged on a sensor surface in an array and a processor in which a neural network is formed by a neuron group with M neurons is performed by a group interface provided between the image sensor and the processor. The present technology can be applied to, for example, a stacked CMOS image sensor.
1 . A photodetection apparatus comprising:
an image sensor provided with a pixel group including N (N is a positive integer) pixels from among a plurality of pixels arranged on a sensor surface in an array;
a processor in which a neural network is formed by a neuron group with M (M is a positive integer) neurons;
a group interface provided between the pixel group and the neuron group and performing bidirectional communication between the image sensor and the processor;
circuitry configured to
use a processing result of processing executed in the neural network to control properties of the pixel,
for each of the neurons in the neuron group, in a case where intermediate data determined by each of the neurons exceeds a predetermined threshold, detect firing of the neuron for which the intermediate data exceeds the threshold, and
determine a learning signal based on a firing rate of the neuron detected; and
a memory that accumulates the learning signal output over time, wherein
a value obtained by accumulating the learning signal in the memory is fed to the circuitry as a control signal and used to control the properties of the pixel.
2 . The photodetection apparatus according to claim 1 , wherein
the circuitry is configured to adjust any one or more of a gain, a delay, and a noise in a pixel signal output from the pixel and an unresponsive period of the pixel signal.
3 . The photodetection apparatus according to claim 1 , wherein
the neural network executes processing of detecting a pixel signal with a fixed, temporally periodic pattern acquired by the image sensor.
4 . The photodetection apparatus according to claim 1 , wherein
the circuitry is configured to determine or change a region-of-interest in which a subject of interest appears in an image captured by the image sensor, and set the pixels other than those in the region-of-interest into a standby state.
5 . The photodetection apparatus according to claim 1 , wherein the circuitry is configured to perform:
a differential of the firing rate of the neuron or an operation corresponding to the differential and feeds a result of the operation to the firing detection section.
6 . The photodetection apparatus according to claim 5 , wherein
the circuitry is configured to perform the operation corresponding to the differential by use of at least any one of a leaky integrate value of the number of firings of the neuron and a membrane potential of the neuron.
7 . The photodetection apparatus according to claim 5 , wherein
the circuitry is configured to use the operation corresponding to the differential to determine a time moving rate of the firing rate that is a differential, in a spatial direction, of a spatial distribution of firing rates of the neurons.
8 . The photodetection apparatus according to claim 1 , wherein
the group interface accumulates the learning signal in the learning memory and performs, at a predetermined time, communication of transmitting the control signal.
9 . The photodetection apparatus according to claim 1 ,
the circuitry is configured to address conversion processing of changing a virtual address of the corresponding neuron group to newly specifically associate the pixel group with the neuron group in response to receiving a content of a change in the pixel group transmitted from the image sensor in a case where the pixel group is changed, the content of the change being included in the communication.
10 . The photodetection apparatus according to claim 9 , wherein,
when executing the address conversion processing, the circuitry is configured to realize the change in the pixel group by referencing a pre-held address table to replace the address.
11 . The photodetection apparatus according to claim 1 , wherein,
at an interface between a semiconductor substrate on which the image sensor is provided and a semiconductor substrate on which the processor is provided, pieces of wiring are stacked by Cu—Cu bonding, and the group interface is arranged on the processor side of the interface.
12 . The photodetection apparatus according to claim 11 , wherein
a neural network circuit different from the processor is connected to an outside of the photodetection apparatus, both the processor and the neural network circuit different from the processor communicate with each other, and both the image sensor and the neural network circuit different from the processor communicate with each other.
13 . The photodetection apparatus according to claim 4 , wherein,
when the region-of-interest is output from the image sensor and input to the processor, the processor performs an operation of recognizing the subject appearing in the region-of-interest, and notifies the image sensor of the region-of-interest at a next time according to a result of the operation to change the region-of-interest to be output by the image sensor at the next time.
14 . The photodetection apparatus according to claim 1 , wherein the circuitry is configured to predict pixels requiring state transition at a next time based on the differential of frequency of the firing rate and preliminarily prepare the pixels for which driving is started at the next time.
15 . The photodetection apparatus according to claim 1 , wherein the processor is configured to perform a multiply and accumulate operation between data fed via the group interface and weights stored in the memory.
16 . The photodetection apparatus according to claim 1 , wherein the circuitry is configured to transmit a control signal to the image sensor to adjust a signal amount to prevent a wait to access the memory in the processor.
17 . The photodetection apparatus according to claim 11 , wherein the processor and the image sensor communicate bidirectionally via the group interface arranged on the semiconductor substrate on which the processor is provided to achieve an increase in efficiency of power consumption.
18 . The photodetection apparatus according to claim 1 , wherein the circuitry is further configured to unroll a plurality of the neurons in a time direction to acquire and accumulate values of learning information at every time step.
19 . The photodetection apparatus according to claim 1 , wherein the circuitry is further configured to replace the determined learning signal with a firing rate and use the firing rate to pre-drive the pixels in a high firing region at a next time.
20 . Electronic equipment comprising:
an image sensor provided with a pixel group including N (N is a positive integer) pixels from among a plurality of pixels arranged on a sensor surface in an array;
a processor in which a neural network is formed by a neuron group with M (M is a positive integer) neurons;
a group interface provided between the pixel group and the neuron group and performing bidirectional communication between the image sensor and the processor; and
an address control circuit that executes address conversion processing of changing a virtual address of the corresponding neuron group to newly specifically associate the pixel group with the neuron group in response to receiving a content of a change in the pixel group transmitted from the image sensor in a case where the pixel group is changed the content of the change being included in the communication.