Configurable and programmable multi-core architecture with a specialized instruction set for embedded application based on neural networks
A programmable architecture specialized for convolutional neural networks (CNNs) processing such that different applications of CNNs may be supported by the presently disclosed method and apparatus by reprogramming the processing elements therein. The architecture may include an optimized architecture that provides a low-area or footprint and low-power solution desired for embedded applications while still providing the computational capabilities required for CNN applications that may be computationally intensive, requiring a huge number of convolution operations per second to process inputs such as video streams in real time.
1. A convolutional neural network (CNN) architecture including:
a) a plurality of processing elements (PEs), each processing element including a plurality of CNN specific processing instructions; and
b) a streaming data interconnect module, the module including:
i) a plurality of input ports;
ii) a plurality of output ports, each coupled to one PE of the plurality of PEs;
iii) a plurality of first in first out registers (FIFO)s;
iv) an input streaming interconnect module coupled to the plurality of input ports and the plurality of FIFOs, the input streaming interconnect module capable of changeably directing data from at least one input port of the plurality of input ports to least one FIFO of the plurality of FIFOs based on CNN layer of the plurality of CNN layers being processed by the plurality of PEs; and
v) an output streaming interconnect module coupled to a plurality of output ports and the plurality of FIFOs, the output streaming interconnect module capable of changeably directing data from at least one FIFO of the plurality of FIFOs to least one output port of the plurality of output ports based on CNN layer of the plurality of CNN layers being processed by the plurality of PEs.
2. The CNN architecture of claim 1 , wherein the number of PEs in the plurality of PEs, the number of FIFOs of the plurality of FIFOs is selected during design based on CNN graph topologies to be processed by the CNN architecture.
3. The CNN architecture of claim 2 , wherein each FIFOs of the plurality of FIFOs has a depth, and its depth is selected during design based on the CNN graph topologies to be processed by the CNN architecture.
4. The CNN architecture of claim 1 , wherein each PE of the plurality of PEs is an application specific instruction processor.
5. The CNN architecture of claim 4 , wherein each PE of the plurality of PEs includes a tightly coupled data memory.
6. The CNN architecture of claim 5 , wherein each PE of the plurality of PEs includes a tightly coupled program memory.
7. The CNN architecture of claim 4 , wherein each PE of the plurality of PEs includes a plurality of pipelines, at least one pipeline for processing CNN specific instructions only.
8. The CNN architecture of claim 4 , wherein each PE of the plurality of PEs includes at least 3 pipelines including a general instruction pipeline, a convolution calculating pipeline, and a non-linear activation function pipeline.
9. The CNN architecture of claim 4 , wherein each PE of the plurality of PEs includes at least 4 pipelines including a general instruction pipeline, a convolution calculating pipeline, a non-linear activation function pipeline, and a data movement pipeline.
10. The CNN architecture of claim 1 , wherein the CNN specific processing functions include convolution functions and non-linear activation functions.
11. The CNN architecture of claim 1 , wherein the non-linear activation function employs a look up table function.
12. The CNN architecture of claim 2 , wherein the number of FIFOs of the plurality of FIFOs is greater than the number of PEs in the plurality of PEs.
13. The CNN architecture of claim 1 , wherein the input streaming interconnect module includes a plurality of demultiplexers individually coupled to one of the plurality of input ports and changeably coupled to a plurality of multiplexers, the plurality of multiplexers individually coupled to one of the plurality of FIFOs.
14. The CNN architecture of claim 13 , wherein the output streaming interconnect module includes a plurality of demultiplexers individually coupled to one of the plurality of FIFOs and changeably coupled to a plurality of multiplexers, the plurality of multiplexers individually coupled to one of the plurality of output ports.
15. The CNN architecture of claim 12 , wherein the input streaming interconnect module includes a plurality of demultiplexers individually coupled to one of the plurality of input ports and changeably coupled to a plurality of multiplexers, the plurality of multiplexers individually coupled to one of the plurality of FIFOs, the number of demultiplexers of the plurality of demultiplexers less than the number of multiplexers in the plurality of multiplexers.
16. The CNN architecture of claim 15 , wherein the output streaming interconnect module includes a plurality of demultiplexers individually coupled to one of the plurality of FIFOs and changeably coupled to a plurality of multiplexers, the plurality of multiplexers individually coupled to one of the plurality of output ports, the number of demultiplexers of the plurality of demultiplexers greater than the number of multiplexers in the plurality of multiplexers.
17. The CNN architecture of claim 1 , wherein the CNN architecture is runtime configurable to process different CNN graph topologies.
18. The CNN architecture of claim 17 , wherein the input streaming interconnect module is runtime configurable to process the different CNN graph topologies.
19. The CNN architecture of claim 18 , wherein the output streaming interconnect module is runtime configurable to process the different CNN graph topologies.
20. The CNN architecture of claim 1 , wherein the non-linear activation functions includes one of sigmoid functions and tan h functions.
21. The CNN architecture of claim 2 , wherein the number of FIFOs of the plurality of FIFOs is less than the number of PEs in the plurality of PEs.