Non-volatile processing-in-sensor accelerator for imaging systems
View Patent ↗Disclosed is a Processing-In-Sensor Accelerator (PISA) that provides a flexible, energy-efficient, and high-performance solution for real-time and smart image processing in AI devices. PISA implements a coarse-grained convolution operation in Binarized-Weight Neural Networks (BWNNs) leveraging a novel compute-pixel with non-volatile weight storage at the sensor side. This reduces power consumption of data conversion and transmission to an off-chip processor. A bit-wise near-sensor in-memory computing unit processes the remaining network layers. Once the object is detected, PISA switches to typical sensing mode to capture the image for a fine-grained convolution using only a near-sensor processing unit. The circuit-to-application co-simulation results on a BWNN acceleration demonstrate minor accuracy degradation on various image datasets in coarse-grained evaluation compared to baseline BWNN models. PISA achieves a frame rate of 1000 and efficiency of 1.74 TOp/s/W. PISA reduces data conversion and transmission energy by at least 84% compared to a baseline.
1 . A method for a Processing-In-Sensor architecture (PISA), comprising:
inputting data that represents an image into an edge Internet-of-Things (IoT) device;
enabling integrated sensing and processing of a 1 st layer of a Binarized-Weight Neural Networks (BWNN) with weights stored in a non-volatile magnetic memory component for offering energy-efficiency and speed-up at the edge IoT device; and
implementing a convolutional operation, wherein the convolutional operation includes implementing a coarse-grained convolution operation in the Binarized-Weight Neural Networks (BWNN) through leveraging a compute-pixel with a non-volatile weight storage at a sensor side.
2 . The method of claim 1 , further comprises reducing power consumption of data conversion and transmission to an off-chip processor.
3 . The method of claim 1 , further comprises processing network layers with a bit-wise near-sensor in memory computing unit or a near-sensor processing unit.
4 . The method of claim 3 , further comprises detecting an object and switching to a sensing mode to capture an image for a fine-grained convolution using only the near-sensor processing unit.
5 . A processing-in-sensor architecture (PISA) system comprising:
a network structure with at least four convolutional layers and one Fully Connected (FC) layer with an image input data being processed therein;
a non-volatile memory (NVM) element for pre-storing a plurality of pixels from the image input data;
wherein, the four convolutional layers include a 1 st -layer, a 2 nd layer, a 3 rd layer and a 4 th layer, and the 1st-layer is processed in a Binarized-Weight Neural Network (BWNN); and
a Process Near Sensor (PNS) unit to perform a low bit-width coarse-grained convolution on the remaining 2 nd layer, 3 rd layer and 4 th layer.
6 . The system of claim 5 , further comprises a binarized weight included in the 1 st layer corresponding to the pixels pre-stored into the NVM, and a coarse-grained multiply-and-accumulate (MAC) operation accomplished in a voltage-controlled crossbar process.
7 . The system of claim 6 , wherein an output of the 1st layer is transmitted to the PNS unit for computation of the 2 nd layer, the 3 rd layer, and the 4th layer.
8 . The system of claim 5 , further comprising a switch for switching to a sensing mode once an object is roughly detected.
9 . The system of claim 8 , wherein the PNS unit provides a fine-grained convolution after capture of an image of the object.
10 . The system of claim 9 , further comprises a sense bit-line (SBL) that is shared across pixels on a same column connected to a sensor for the sensing mode.
11 . The system of claim 5 , further comprising a Compute Focal Plane (CFP), a row and column controllers (Ctrl), a command decoder, a sensor timing control, and a sensor I/O that operates in two modes.
12 . The system of claim 11 , wherein, the CFP is cointegrated for sensing and processing the 1 st layer of the BWNN targeting a low-power and coarse grained classification.
13 . The system of claim 11 , further comprising a Compute Pixel (CP) unit that includes a pixel connected to the Non-Volatile Memory (NVM) element.
14 . A method for a Processing-In-Sensor architecture (PISA), comprises:
performing an image processing method of an object by processing an image through a 1 st convolution layer in a Binarized-Weight Neural Networks (BWNN) for implementing a coarse grained convolution operation;
performing a low bit-width coarse-grained convolution in a Processor Near Sensor (PNS) unit on a plurality of other convolution layers;
switching a Processing-In-Sensor architecture (PISA) to a sensing mode to capture the image when the object is roughly detected after the low bit-width coarse-grained convolution; and
using on the image a fine-grained convolution using the PNS unit.
15 . The method of claim 14 further includes using the image processing method for real-time and smart image processing in an artificial intelligence (AI) device.
16 . The method of claim 14 , wherein the PISA intrinsically performs a coarse-grained convolution operation on the 1st-layer of the BWNN leveraging a compute-pixel with a nonvolatile weight storage.
17 . The method of claim 14 , further includes performing the fine-grained convolution operation though the PNS on the plurality of other layers.
18 . The method of claim 14 , further includes achieving by the PISA a frame rate of 1000 and an efficiency of about 1.74 TOp/s/W.
19 . The method of claim 14 wherein the PNS is an off-chip processor that reduces power consumption of data conversion and transmission as compared to not using the PNS.