Ultra pipelined accelerator for machine learning inference
A method of pipelining inference of a neural network, which includes an i-th layer (i being an integer greater than zero), an (i+1)-th layer, and an (i+2)-th layer, includes processing a first set of i-th values of the i-th layer to generate (i+1)-th values for the (i+1)-th layer, determining a quantity of the (i+1)-th values as being sufficient for processing, and in response to the determining, processing the (i+1)-th values to generate an output value for the (i+2)-th layer while concurrently processing a second set of i-th values of the i-th layer.
1. A method of pipelining inference of a neural network comprising a plurality of layers comprising an i-th layer (i being an integer greater than zero), an (i+1)-th layer, and an (i+2)-th layer, the method comprising:
processing, by a controller, a first set of i-th values of the i-th layer via an i-th composite filter to generate (i+1)-th values for the (i+1)-th layer;
determining, by the controller, a quantity of the (i+1)-th values as being sufficient for processing by an (i+1)-th filter associated with the (i+1)-th layer, based on generating the (i+1)-th values to correspond to each unit being operated on by the (i+1)-th filter, the i-th composite filter comprising a first i-th filter and a second i-th filter, and generating more than one (i+1)-th value per clock cycle based on the second i-th filter being offset from the first i-th filter on the i-th layer; and
in response to the determining, processing, by the controller, the (i+1)-th values via the (i+1)-th filter to generate an output value for the (i+2)-th layer while concurrently processing a second set of i-th values of the i-th layer via the i-th composite filter.
2. The method of claim 1 , wherein the first set of i-th values comprise pixel values of an image.
3. The method of claim 1 , wherein the first set of i-th values comprise values of a feature map generated by a previous layer of the neural network.
4. The method of claim 1 , wherein the processing of the first set of i-th values of the i-th layer comprises:
applying the i-th composite filter to the first set of i-th values of the i-th layer, such that the first i-th filter is offset from the second i-th filter by a stride length of the first i-th filter.
5. The method of claim 4 , wherein the first i-th filter is a sliding convolutional filter in a form of a p×q matrix, where p and q are integers greater than zero.
6. The method of claim 4 , wherein applying the i-th composite filter comprises:
performing a matrix multiplication operation between the i-th composite filter and ones of the first set of i-th values overlapping the i-th composite filter.
7. The method of claim 1 , wherein the determining the quantity of the (i+1)-th values as being sufficient for processing is based on a clock cycle determined by at least one of a size of the i-th composite filter, a size of an (i+1)-th filter associated with the (i+1)-th layer, a stride of the (i+1)-th filter, and a size of an image or feature map corresponding to the first and second set of i-th values of the i-th layer.
8. The method of claim 1 , wherein the processing of the first set of i-th values of the i-th layer comprises:
applying the first i-th filter to a first part of the first set of i-th values to generate a first one of the (i+1)-th values for the (i+1)-th layer; and
concurrently with applying the first i-th filter, applying the second i-th filter associated with the i-th layer to a second part of the first set of i-th values to generate a second one of the (i+1)-th values for the (i+1)-th layer.
9. The method of claim 8 , wherein the first and second parts of the first set of i-th values have overlapping values.
10. The method of claim 8 , wherein the first and second parts of the first set of i-th values do not have overlapping values.
11. The method of claim 8 , wherein the first and second i-th filters are sliding convolutional filters having a same stride and a same dimension.
12. The method of claim 8 , wherein the first and second i-th filters comprise the same weight values.
13. A system for pipelining inference of a neural network comprising a plurality of layers comprising an i-th layer (i being an integer greater than zero), an (i+1)-th layer, and an (i+2)-th layer, the system comprising:
a processor; and
a processor memory local to the processor, wherein the processor memory has stored thereon instructions that, when executed by the processor, cause the processor to perform:
processing a first set of i-th values of the i-th layer via an i-th composite filter to generate (i+1)-th values for the (i+1)-th layer;
determining a quantity of the (i+1)-th values as being sufficient for processing by an (i+1)-th filter associated with the (i+1)-th layer, based on generating the (i+1)-th values to correspond to each unit being operated on by the (i+1)-th filter, the i-th composite filter comprising a first i-th filter and a second i-th filter, and generating more than one (i+1)-th value per clock cycle based on the second i-th filter being offset from the first i-th filter on the i-th layer; and
in response to the determining, processing the (i+1)-th values via the (i+1)-th filter to generate an output value for the (i+2)-th layer while concurrently processing a second set of i-th values of the i-th layer via the i-th composite filter.
14. The system of claim 13 , wherein the processing of the first set of i-th values of the i-th layer comprises:
applying the i-th composite filter to the first set of i-th values of the i-th layer to generate the (i+1)-th values for the (i+1)-th layer, such that the first i-th filter is offset from the second i-th filter by a stride length of the first i-th filter.
15. The system of claim 13 , wherein the determining the quantity of the (i+1)-th values as being sufficient for processing is based on a clock cycle determined by at least one of a size of the i-th composite filter, a size of an (i+1)-th filter associated with the (i+1)-th layer, a stride of the (i+1)-th filter, and a size of an image or feature map corresponding to the first and second set of i-th values of the i-th layer.
16. The system of claim 13 , wherein the processing of the first set of i-th values of the i-th layer comprises:
applying the first i-th filter to a first part of the first set of i-th values to generate a first one of the (i+1)-th values for the (i+1)-th layer; and
concurrently with applying the first i-th filter, applying the second i-th filter associated with the i-th layer to a second part of the first set of i-th values to generate a second one of the (i+1)-th values for the (i+1)-th layer.
17. A configurable processing-in-memory (PIM) system configured to implement a neural network, the system comprising:
a first at least one PIM subarray configured to perform a filtering operation of an i-th composite filter of an i-th layer of the neural network (i being an integer greater than zero);
a second at least one PIM subarray configured to perform a filtering operation of an (i+1)-th filter of an (i+1)-th layer of the neural network; and
a controller configured to control the first and second at least one PIM subarrays, the controller being configured to perform:
supplying a first set of i-th values of the i-th layer to the first at least one PIM subarray and processing the first set of i-th values via an i-th composite filter to generate (i+1)-th values for the (i+1)-th layer;
determining a quantity of the (i+1)-th values as being a sufficient quantity for processing by the second at least one PIM subarray, based on generating the (i+1)-th values to correspond to each unit being operated on by the second at least one PIM subarray, the i-th composite filter comprising a first i-th filter and a second i-th filter, and generating more than one (i+1)-th value per clock cycle based on the second i-th filter being offset from the first i-th filter on the i-th layer; and
in response to the determining, supplying the (i+1)-th values to the second at least one PIM subarray to generate an output value for a subsequent layer of the neural network while concurrently supplying a second set of i-th values of the i-th layer to the first at least one PIM subarray for processing.
18. The system of claim 17 , wherein a PIM subarray of the first and second at least one PIM subarrays comprises:
a plurality of bitcells for storing a plurality of weights corresponding to a respective one of the i-th composite filter or the (i+1)-th filter.
19. The system of claim 17 , wherein the neural network comprises a convolutional neural network, and
wherein the first i-th filter is a sliding convolutional filter.