Memory bandwidth reduction techniques for low power convolutional neural network inference
View Patent ↗Systems, apparatuses, and methods for implementing memory bandwidth reduction techniques for low power convolutional neural network inference applications are disclosed. A system includes at least a processing unit and an external memory coupled to the processing unit. The system detects a request to perform a convolution operation on input data from a plurality of channels. Responsive to detecting the request, the system partitions the input data from the plurality of channels into 3D blocks so as to minimize the external memory bandwidth utilization for the convolution operation being performed. Next, the system loads a selected 3D block from external memory into internal memory and then generates convolution output data for the selected 3D block for one or more features. Then, for each feature, the system adds convolution output data together across channels prior to writing the convolution output data to the external memory.
1 . An apparatus comprising:
a plurality of channel processing engines, each of the channel processing engines comprising circuitry configured to perform convolution computations for a channel of a plurality of channels;
wherein a first channel processing engine of the plurality of channel processing engines is configured to:
read first data corresponding to a first channel of the plurality of channels; and
perform computations for a plurality of features using the first data, prior to performing a read of second data for a second channel of the plurality of channels.
2 . The apparatus as recited in claim 1 , wherein the first channel processing engine is configured to generate first output data responsive to the computations.
3 . The apparatus as recited in claim 2 , wherein subsequent to the computations for the plurality of features, a second channel processing engine of the plurality of channel processing engines is configured to:
read second data corresponding to a second channel of the plurality of channels; and
perform computations for the plurality of features using the second data, prior to performing a read of data for a third channel of the plurality of channels.
4 . The apparatus as recited in claim 3 , wherein the second channel processing engine is configured to generate second output data responsive to the computations using the second data.
5 . The apparatus as recited in claim 4 , further comprising a plurality of adders, each corresponding to a feature of the plurality of features.
6 . The apparatus as recited in claim 5 , wherein a first adder of the plurality of adders is configured to generate a sum of the first output data and the second output data.
7 . The apparatus as recited in claim 4 , wherein the apparatus is configured to read the first data corresponding to the first channel only once to produce both the first output data and the second output data.
8 . The apparatus as recited in claim 2 , wherein the first channel processing engine is configured to cause the first output data to be stored in an external memory.
9 . A method comprising:
reading, by a first channel processing engine of a plurality of channel processing engines, first data corresponding to a first channel of a plurality of channels, wherein each of the channel processing engines comprises circuitry configured to perform convolution computations for a channel of a plurality of channels; and
performing, by the first channel processing engine, computations for a plurality of features using the first data, prior to performing a read of second data for a second channel of the plurality of channels.
10 . The method as recited in claim 9 , further comprising generating, by the first channel processing engine, first output data responsive to the computations.
11 . The method as recited in claim 10 , wherein subsequent to the computations for the plurality of features, the method comprises a second channel processing engine of the plurality of channel processing engines:
reading second data corresponding to a second channel of the plurality of channels; and
performing computations for the plurality of features using the second data, prior to performing a read of data for a third channel of the plurality of channels.
12 . The method as recited in claim 11 , further comprising generating, by the second channel processing engine, second output data responsive to the computations using the second data.
13 . The method as recited in claim 12 , further comprising generating, by an adder of a plurality of adders, a sum of the first output data and the second output data.
14 . The method as recited in claim 13 , further comprising reading the first data corresponding to the first channel only once to produce both the first output data and the second output data.
15 . The method as recited in claim 10 , further comprising causing, by the first channel processing engine, the first output data to be stored in an external memory.
16 . A system comprising:
a processing unit comprising internal memory;
an external memory coupled to the processing unit; and
an inference engine comprising:
a plurality of channel processing engines, each of the channel processing engines comprising circuitry configured to perform convolution computations for a channel of a plurality of channels;
wherein a first channel processing engine of the plurality of channel processing engines comprises circuitry configured to:
read first data corresponding to a first channel of the plurality of channels from the external memory into the internal memory; and
perform computations for a plurality of features using the first data, prior to performing a read of second data for a second channel of the plurality of channels.
17 . The system as recited in claim 16 , wherein the first channel processing engine is configured to generate first output data responsive to the computations.
18 . The system as recited in claim 17 , wherein the first channel processing engine is configured to cause the first output data to be stored in the external memory.
19 . The system as recited in claim 16 , wherein subsequent to the computations for the plurality of features, a second channel processing engine of the plurality of channel processing engines is configured to:
read second data corresponding to a second channel of the plurality of channels from the external memory into the internal memory; and
perform computations for the plurality of features using the second data, prior to performing a read of data for a third channel of the plurality of channels.
20 . The system as recited in claim 19 , wherein the second channel processing engine is configured to generate second output data responsive to the computations using the second data.