Artificial intelligence pixel sensor systems and methods
Intelligent sensor systems and methods are provided. An intelligent sensor system as described herein includes an intelligent sensor array in which at least portions of a neural network are implemented in hardware. More particularly, multiple artificial intelligence pixels, each providing a neural network, can be included in the intelligent sensor system. Each artificial intelligence pixel includes a plurality of sub-pixel elements disposed across a portion of the intelligent sensor array. Each sub-pixel element includes a photodetector and a neuromorphic element. Weighted outputs from the sub-pixel elements within an artificial intelligence pixel are summed and compared to a reference value. Depending on that comparison, a pixel output is or is not provided to a subsequent layer of a larger neural network or to a data consumer.
1. An intelligent sensor system, comprising:
a sensor array, including:
a plurality of artificial intelligence pixels, each artificial intelligence pixel in the plurality of artificial intelligence pixels including:
a plurality of sub-pixel elements, each sub-pixel element in the plurality of sub-pixel elements including:
a photodetector; and
a neuromorphic element;
an artificial intelligence pixel summing node, wherein a weighted output from each sub-pixel element in the plurality of sub-pixel elements is provided to the artificial intelligence pixel summing node; and
an artificial intelligence pixel comparator, wherein the artificial intelligence pixel comparator operates to compare a summed output of the sub-pixel elements provided to the artificial intelligence pixel summing node to a first reference value established for the artificial intelligence pixel comparator, and wherein the artificial intelligence pixel comparator further operates to provide an output in response to a first comparison result; and
a plurality of summation and data reduction blocks, wherein each summation and data reduction block includes:
a summation and data reduction block summing node, wherein an output from at least some of the artificial intelligence pixels is received at the summation and data reduction block summing node; and
a summation and data reduction block comparator, wherein the summation and data reduction block comparator operates to compare a summed output of the at least some of the artificial intelligence pixels to a second reference value established for the summation and data block comparator, and wherein the summation and data reduction block comparator further operates to provide an output in response to a second comparison result.
2. The intelligent sensor system of claim 1 , further comprising:
a digital physical infrastructure layer, wherein outputs from the artificial intelligence pixels are received at the digital physical infrastructure layer, and wherein the digital physical infrastructure layer formats the outputs from the artificial intelligence pixels.
3. The intelligent sensor system of claim 2 , further comprising:
a communication interface, wherein the communication interface provides a formatted version of the output from the digital physical infrastructure layer to a data consumer.
4. The intelligent sensor system of claim 1 , wherein, in each sub-pixel element, the photodetector is integrated with the neuromorphic element, and wherein the neuromorphic element is one of a memristor, a memcapacitor, or a meminductor.
5. The intelligent sensor system of claim 4 , wherein a value of each neuromorphic element is set during a training process.
6. The intelligent sensor system of claim 5 , wherein a value of each reference value is set during a training process.
7. The intelligent sensor system of claim 6 , wherein at least some of the artificial intelligence pixels are trained to perform change detection.
8. The intelligent sensor system of claim 6 , wherein at least some of the artificial intelligence pixels are trained to perform object recognition.
9. The intelligent sensor system of claim 6 , wherein at least some of the artificial intelligence pixels are trained to perform segmentation.
10. The intelligent sensor system of claim 1 , further comprising:
a platform, wherein the sensor array is carried by the platform;
an optical assembly, wherein the optical assembly directs light to the sensor array; and
a communication interface, wherein an output from at least some of the artificial intelligence pixels is provided to a data consumer by the communication interface.
11. A method of providing an intelligent sensor, comprising:
providing a sensor array, including:
a plurality of artificial intelligence pixels, each artificial intelligence pixel in the plurality of artificial intelligence pixels including:
a plurality of sub-pixel elements, each sub-pixel element in the plurality of sub-pixel elements including:
a photodetector; and
a neuromorphic element;
an artificial intelligence pixel summing node, wherein a weighted output from each sub-pixel element in the plurality of sub-pixel elements is provided to the artificial intelligence pixel summing node; and
an artificial intelligence pixel comparator, wherein the artificial intelligence pixel comparator operates to compare a summed output of the sub-pixel elements provided to the artificial intelligence pixel summing node to a first reference value established for the artificial intelligence pixel comparator, and wherein the artificial intelligence pixel comparator further operates to provide an output in response to a first comparison result; and
a plurality of summation and data reduction blocks, wherein each summation and data reduction block includes:
a summation and data reduction block summing node, wherein an output from at least some of the artificial intelligence pixels is received at the summation and data reduction block summing node; and
a summation and data reduction block comparator, wherein the summation and data reduction block comparator operates to compare a summed output of the at least some of the artificial intelligence pixels to a second reference value established for the summation and data block comparator, and wherein the summation and data reduction block comparator further operates to provide an output in response to a second comparison result;
exposing the plurality of photodetectors of the sub-pixel elements of an artificial intelligence pixel included in the plurality of artificial intelligence pixels to light, wherein, for each photodetector in the plurality of photodetectors, a value of a neuromorphic element integrated with the photodetector is selected during a training process to determine a responsivity of the photodetector to incident light;
summing a weighted output from each of the photodetectors in the plurality of photodetectors produced in response to the photodetectors being exposed to the light in the artificial intelligence pixel summing node;
comparing the summed and weighted output to the first reference value in the artificial intelligence comparator, wherein the first reference value is selected during the training process; and
providing an artificial intelligence pixel output in response to a first comparison result.
12. The method of providing an intelligent sensor of claim 11 , wherein the sensor array is trained to perform change detection.
13. The method of providing an intelligent sensor of claim 11 , wherein the sensor array is trained to perform object recognition.
14. The method of providing an intelligent sensor of claim 11 , wherein the sensor array is trained to perform segmentation.
15. The method of providing an intelligent sensor of claim 11 , wherein the sensor array is trained to perform a wide variety of signal and image processing.