Hierarchical networks on chip (NoC) for neural network accelerator
This application describes a hardware accelerator and a device for accelerating neural network computations. An example accelerator may include multiple cores and a central processing unit (CPU) respectively associated with DDRs, a data exchange interface connecting a host device to the accelerator, and a three-layer NoC architecture. The three-layer NoC architecture includes an outer-layer NoC configured to transfer data between the host device and the DDRs, a middle-layer NoC configured to transfer data among the plurality of cores; and an inner-layer NoC within each core and including a cross-bar network for broadcasting weights and activations of neural networks from a global buffer of the core to a plurality of processing entity (PE) clusters within the core.
1. A neural network (NN) accelerator with a multi-layer networks-on-chip (NoCs) architecture, comprising:
a plurality of cores and a central processing unit (CPU), wherein each core comprises a plurality of processing entity (PE) clusters, and the plurality of cores and the CPU are respectively coupled to memories, and
a data exchange interface connecting a host device to the NN accelerator,
an outer-layer NoC,
a middle-layer NoC, and
an inner-layer NoC,
wherein:
the outer-layer NoC is configured to transfer data between the host device and the memories, and comprises a plurality of routers forming a bi-directional ring-shape data link, wherein the plurality of routers comprise a first router connected to the data exchange interface, a second router connected to the CPU and the corresponding memory, and multiple third routers respectively connected to the plurality of cores and the corresponding memories,
the middle-layer NoC is configured to transfer data among the plurality of cores; and
the inner-layer NoC is within each core and configured to perform data casting among the plurality of PE clusters within the core for implementing matrix operations.
2. The NN accelerator of claim 1 , wherein the data exchange interface comprises a Peripheral Component Interconnect Express (PCIe) interface.
3. The NN accelerator of claim 1 , wherein the bi-directional ring-shape data link connected to the data exchange interface and the memories.
4. The NN accelerator of claim 1 , wherein the outer-layer NoC has a transaction mode, in which a transaction-initiating device initiates a data transfer transaction against a target device.
5. The NN accelerator of claim 1 , wherein the middle-layer NoC comprises a pair of uni-directional ring-shape data links, each uni-directional ring-shape data link comprising a subset of the plurality of cores.
6. The NN accelerator of claim 5 , wherein the pair of uni-directional ring-shape data links in the middle-layer NoC have opposite data transferring directions.
7. The NN accelerator of claim 5 , wherein the pair of uni-directional ring-shape data links comprise a first ring and a second ring, the cores in the first ring respectively correspond to the cores in the second ring, and
each core in the first ring is connected to a corresponding core in the second ring with a bi-directional data link.
8. The NN accelerator of claim 1 , wherein the middle-layer NoC has a streaming mode, in which data is transferred from a sender to a receiver.
9. The NN accelerator of claim 1 , wherein the middle-layer NoC moves weights or activations of neural networks among the plurality of cores.
10. The NN accelerator of claim 1 , wherein the inner-layer NoC comprises a cross-bar network connecting a global buffer of the core to the plurality of PE clusters within the core, and the global buffer of the core is divided into a plurality of banks respectively corresponding to the plurality of PE clusters within the core.
11. The NN accelerator of claim 10 , wherein the cross-bar network in the inner-layer NoC comprises a first butterfly network for casting weights of a neural network from the plurality of banks to the plurality of PE clusters within the core, and a second butterfly network for casting activations of the neural network from the plurality of banks to the plurality of PE clusters within the core.
12. The NN accelerator of claim 11 , wherein the first butterfly network is configured to broadcast or unicast the weights of the neural network from the plurality of banks to all of the plurality of PE clusters within the core.
13. The NN accelerator of claim 11 , wherein the second butterfly network is configured to broadcast or unicast the activations of the neural network from the plurality of banks to a subset of the plurality of PE clusters within the core.
14. A Network-on-Chip (NoC) device, comprising:
a plurality of cores and a central processing unit (CPU), wherein each core comprises a plurality of processing entity (PE) clusters, and the plurality of cores and the CPU are respectively coupled to memories,
a data exchange interface for connecting to a host device,
an outer-layer NoC,
a middle-layer NoC, and
an inner-layer NoC,
wherein:
the outer-layer NoC comprises a bi-directional ring-shape data link connected to the data exchange interface and the memories,
the middle-layer NoC comprises a pair of uni-directional ring-shape data links connecting the plurality of cores; and
the inner-layer NoC is within each core and configured to perform data casting among the plurality of PE clusters within the core for implementing matrix operations.
15. The NoC device of claim 14 , wherein the outer-layer NoC comprises a plurality of routers forming a bi-directional ring-shape data link connected to the data exchange interface and the memories, wherein the plurality of routers comprise a first router connected to the data exchange interface, a second router connected to the CPU and the corresponding memory, and multiple third routers respectively connected to the plurality of cores and the corresponding memories.
16. The NoC device of claim 14 , wherein the middle-layer NoC comprises a pair of uni-directional ring-shape data links, each uni-directional ring-shape data link comprising a subset of the plurality of cores.
17. The NoC device of claim 16 , wherein the pair of uni-directional ring-shape data links in the middle-layer NoC have opposite data transferring directions, and each of the pair of uni-directional ring-shape data links comprises a subset of the plurality of cores.
18. The NoC device of claim 14 , wherein the pair of uni-directional ring-shape data links comprises a first ring and a second ring, the cores in the first ring respectively correspond to the cores in the second ring, and
each core in the first ring is connected to a corresponding core in the second ring with a bi-directional data link.
19. The NoC device of claim 14 , wherein the inner-layer NoC comprises a cross-bar network connecting a global buffer of the core to the plurality of PE clusters within the core, and the global buffer of the core is divided into a plurality of banks respectively corresponding to the plurality of PE clusters within the core.
20. The NoC device of claim 14 , wherein a data transfer bandwidth of the outer-layer NoC is lower than a data transfer bandwidth of the middle-layer NoC, and the data transfer bandwidth of the middle-layer NoC is lower than a data transfer bandwidth of the inner-layer NoC.