Configurable processor for implementing convolution neural networks
Configurable processors for implementing CNNs are provided. One such configurable CNN processor includes a plurality of core compute circuitry elements, each configured to perform a CNN function in accordance with a preselected dataflow graph, an active memory buffer, a plurality of connections between the active memory buffer and the plurality of core compute circuitry elements, each established in accordance with the preselected dataflow graph, a plurality of connections between the plurality of core compute circuitry elements, each established in accordance with the preselected dataflow graph, wherein the active memory buffer is configured to move data between the plurality of core compute circuitry elements via the active memory buffer in accordance with the preselected dataflow graph.
1. A configurable processor dedicated to implementing convolution neural networks (CNNs) and implemented in a single integrated circuit die, comprising:
a plurality of core compute circuits on the die, each configured to perform a CNN function in accordance with a preselected dataflow graph;
an active memory buffer on the die;
a plurality of connections, on the die, between the active memory buffer and the plurality of core compute circuits, each established in accordance with the preselected dataflow graph; and
a plurality of connections, on the die, between the plurality of core compute circuits, each established in accordance with the preselected dataflow graph,
wherein the active memory buffer on the die is configured to store data from, and move data between, the plurality of core compute circuits in accordance with the preselected dataflow graph,
wherein each of the plurality of core compute circuits is configured to perform the CNN function in accordance with the preselected dataflow graph and without using an instruction set, and
wherein the active memory buffer is further configured to apply backpressure on a data generation source.
2. The configurable processor of claim 1 , wherein the preselected dataflow graph is based on a preselected CNN.
3. The configurable processor of claim 1 , wherein at least two of the plurality of core compute circuits are configured to operate asynchronously from one another.
4. The configurable processor of claim 1 , wherein the active memory buffer and each of the plurality of core compute circuits are configured to operate asynchronously from one another.
5. The configurable processor of claim 1 , wherein each of the plurality of core compute circuits is dedicated to performing the CNN function.
6. The configurable processor of claim 1 , wherein each of the plurality of core compute circuits is configured, prior to a runtime of the configurable processor, to perform the CNN function.
7. The configurable processor of claim 1 , wherein each of the plurality of core compute circuits is configured to compute a layer of the CNN function.
8. The configurable processor of claim 1 , wherein each of the plurality of core compute circuits is configured to compute an entire CNN.
9. The configurable processor of claim 1 , wherein each of the plurality of core compute circuits is configured to perform the CNN function for both inference and training.
10. The configurable processor of claim 1 , wherein each of the plurality of core compute circuits comprises a memory configured to store a weight used to perform the CNN function.
11. The configurable processor of claim 1 :
wherein the plurality of connections between the active memory buffer and the plurality of core compute circuits are established during a compile time and fixed during a runtime of the configurable processor; and
wherein the plurality of connections between the plurality of core compute circuits are established during the compile time and fixed during the runtime.
12. A processor array, comprising:
a plurality of the configurable processors of claim 1 ;
an interconnect circuitry; and
a plurality of connections between the plurality of configurable processors and/or the interconnect circuitry, each established in accordance with the preselected dataflow graph.
13. A system comprising:
a mobile industry processor interface camera serial interface (MIPI-CSI) source;
a MIPI-CSI sink;
a MIPI-CSI bus coupled between the MIPI-CSI source and the MIPI-CSI sink; and
the configurable processor of claim 1 disposed serially along the MIPI-CSI bus such that all data on the MIPI-CSI bus passes through the configurable processor.
14. The system of claim 13 , further comprising:
a non-MIPI-CSI output interface comprising at least one of a SPI, an I2C interface, or a UART interface; and
wherein the configurable processor is configured to send information to an external device using either the non-MIPI-CSI output interface or the MIPI-CSI bus.
15. A system comprising:
a sensor configured to generate sensor data;
the configurable processor of claim 1 directly coupled to the sensor and configured to generate processed data based on the sensor data; and
a wireless transmitter directly coupled to the configurable processor and configured to transmit at least a portion of the processed data.
16. The system of claim 15 :
wherein the sensor data comprises image data;
wherein the processed data comprises classification data generated based on the image data; and
wherein the wireless transmitter is configured to transmit the classification data.
17. A method for configuring a configurable processor dedicated to implementing convolution neural networks (CNNs), comprising:
receiving a preselected dataflow graph;
programming, prior to a runtime of the configurable processor, each of a plurality of core compute circuits of the configurable processor to perform a CNN function in accordance with the preselected dataflow graph;
programming, prior to the runtime, an active memory buffer of the configurable processor in accordance with the preselected dataflow graph;
programming a plurality of connections, of the configurable processor prior to the runtime, between the active memory buffer and the plurality of core compute circuits in accordance with the preselected dataflow graph;
programming a plurality of connections, of the configurable processor prior to the runtime, between the plurality of core compute circuits in accordance with the preselected dataflow graph;
programming, prior to the runtime, the active memory buffer to move data between the plurality of core compute circuits via the memory buffer in accordance with the preselected dataflow graph and to apply backpressure on a data generation source; and
operating the plurality of core compute circuits, at the runtime, to perform the CNN function without using an instruction set.
18. The method of claim 17 , further comprising operating the active memory buffer, at the runtime, without using an instruction set.
19. The method of claim 17 , wherein the preselected dataflow graph is based on a preselected CNN.
20. The method of claim 17 , further comprising operating at least two of the plurality of core compute circuits asynchronously from one another.
21. The method of claim 17 , further comprising operating the active memory buffer and each of the plurality of core compute circuits asynchronously from one another.
22. The method of claim 17 , wherein each of the plurality of core compute circuits is dedicated to performing the CNN function.
23. The method of claim 17 , further comprising:
performing, during the runtime, the CNN function at each of a respective one of the plurality of core compute circuits.
24. The method of claim 17 , further comprising:
computing, during the runtime, a layer of the CNN function at each of a respective one of the plurality of core compute circuits.
25. The method of claim 17 , further comprising:
computing, during the runtime, an entire CNN at least one of the plurality of core compute circuits.
26. The method of claim 17 :
wherein the plurality of connections between the active memory buffer and the plurality of core compute circuits are programmed during a compile time and fixed during the runtime; and
wherein the plurality of connections between the plurality of core compute circuits are programmed during the compile time and fixed during the runtime.
27. The method of claim 17 , wherein each of the plurality of core compute circuits is configured to perform the CNN function for both inference and training.
28. The method of claim 17 , wherein each of the plurality of core compute circuits comprises a memory configured to store a weight used to perform the CNN function.
29. A configurable processor dedicated to implementing convolution neural networks (CNNs) and implemented in a single integrated circuit die, comprising:
a plurality of means, on the die, for performing a CNN function in accordance with a preselected dataflow graph;
a means, on the die, for storing data;
a means, on the die, for establishing connections between the means for storing data and the plurality of means for performing the CNN function, in accordance with the preselected dataflow graph; and
a means, on the die, for establishing connections between the plurality of means for performing the CNN function, in accordance with the preselected dataflow graph,
wherein the means for storing data comprises a means for moving data between the plurality of means for performing the CNN function via the means for storing data in accordance with the preselected dataflow graph,
wherein each of the plurality of means for performing the CNN function is configured to perform the CNN function in accordance with the preselected dataflow graph and without using an instruction set, and
wherein the means for storing data is configured to apply backpressure on a data generation source.
30. A configurable processor dedicated to implementing convolution neural networks (CNNs), comprising:
a mobile industry processor interface camera serial interface (MIPI-CSI) input circuitry configured to be directly coupled to a MIPI-CSI source circuitry;
a MIPI-CSI output circuitry configured to be directly coupled to an application processor;
a MIPI-CSI bus coupled between the MIPI-CSI input circuitry and the MIPI-CSI output circuitry; and
a configurable CNN sub-processor implemented in a single integrated circuit die and disposed serially along the MIPI-CSI bus such that all data on the MIPI-CSI bus passes through the configurable processor, the configurable CNN sub-processor configured to:
receive image data from the MIPI-CSI source;
generate processed data based on the image data and without using an instruction set; and
provide the processed data to the application processor,
wherein the configurable CNN sub-processor comprises:
a plurality of core compute circuits on the die, each configured to perform a CNN function in accordance with a preselected dataflow graph; and
an active memory buffer on the die, wherein the active memory buffer on the die is configured to:
store data from, and move data on the die between the plurality of core compute circuits via the active memory buffer in accordance with the preselected dataflow graph; and
apply backpressure on a data generation source.
31. The configurable processor of claim 30 , wherein the configurable CNN sub-processor is further configured to generate the processed data based on the image data using a preselected CNN.
32. The configurable processor of claim 30 , wherein the configurable CNN sub-processor comprises a plurality of the configurable CNN sub-processors in a cascade configuration.
33. The configurable processor of claim 30 , wherein the configurable CNN sub-processor is configured to provide the processed data to the application processor via the MIPI-CSI bus.
34. The configurable processor of claim 30 , wherein the configurable CNN sub-processor further comprises:
a plurality of connections, on the die, between the active memory buffer and the plurality of core compute circuits, each established in accordance with the preselected dataflow graph; and
a plurality of connections, on the die, between the plurality of core compute circuits, each established in accordance with the preselected dataflow graph.
35. The configurable processor of claim 30 , further comprising:
a non-MIPI-CSI output interface comprising at least one of a SPI, an I2C interface, or a UART interface; and
wherein the configurable processor is configured to send information to the application processor using either the non-MIPI-CSI output interface or the MIPI-CSI bus.
36. The configurable processor of claim 1 :
wherein each of the plurality of core compute circuits configured to perform the CNN function in accordance with the preselected dataflow graph is further configured to generate intermediate results associated with performance of the CNN function; and
wherein the intermediate results are stored on the die and not in an external memory.
37. The configurable processor of claim 1 , wherein each of the plurality of core compute circuits are configured with a static configuration during a compile time of the configurable processor such that the static configuration does not change during a runtime of the configurable processor.
38. The configurable processor of claim 30 , wherein the active memory buffer comprises two or more ports configured to operate asynchronously from one another.
39. The configurable processor of claim 1 , wherein the active memory buffer comprises two or more ports configured to operate asynchronously from one another.