AI accelerator apparatus using full mesh connectivity chiplet devices for transformer workloads
An AI accelerator apparatus using in-memory compute chiplet devices. The apparatus includes a first semiconductor substrate having a plurality of chiplets, each of which includes a plurality of tiles. Each tile includes a plurality of slices, a central processing unit (CPU), and a hardware dispatch device. Each slice can include a digital in-memory compute (DIMC) device configured to perform high throughput computations. In particular, the DIMC device can be configured to accelerate the computations of attention functions for transformer-based models (a.k.a. transformers) applied to machine learning applications. The chiplets are in a full mesh connectivity configuration such that at least one of the die-to-die (D2D) interconnects of each chiplet is coupled to one of the D2D interconnects of each other chiplet using a non-diagonal link. The chiplets can also include other interfaces to facilitate communication between the chiplets, memory and a server or host system.
1. An AI accelerator apparatus configured with in-memory compute, the apparatus comprising:
a plurality of chiplets coupled together in a full mesh connectivity configuration, each of the chiplets comprising a plurality of tiles, and each of the tiles comprising:
a plurality of slices,
a central processing unit (CPU) coupled to the plurality of slices, and
a hardware dispatch device coupled to the CPU;
a first clock configured to output a clock signal of 0.5 GHz to 4 GHz;
a plurality of die-to-die (D2D) interconnects coupled to the each of CPUs in each of the tiles, wherein at least one of the D2D interconnects of each chiplet is coupled to one of the D2D interconnects of each other chiplet using a non-diagonal link;
a peripheral component interconnect express (PCIe) bus coupled to the CPUs in each of the tiles;
a dynamic random access memory (DRAM) interface coupled to the CPUs in each of the tiles;
a global reduced instruction set computer (RISC) interface coupled to each of the CPUs in each of the tiles;
wherein each of the slices includes a digital in memory compute (DIMC) device coupled to a second clock and configured to allow for a throughput of one or more matrix computations provided in the DIMC device such that the throughput is characterized by 512 multiply accumulates per a clock cycle;
wherein the DIMC device is coupled to the second clock configured at an output rate of one half of the rate of the first clock; and
a substrate member configured to provide mechanical support and having a surface region, the surface region being coupled to support the plurality of chiplets.
2. The apparatus of claim 1 further comprising one or more double data rate (DDR) DRAM devices, the one or more DDR DRAM devices being coupled to one or more chiplets using the DRAM interface.
3. The apparatus of claim 1 further comprising a main bus device, the main bus device being coupled to each PCIe bus in each chiplet using a master chiplet device, the master chiplet device being coupled to each of the other chiplet devices using at least the plurality of D2D interconnects.
4. The apparatus of claim 3 further comprising a server apparatus, the apparatus being configured and operable to the server apparatus using the main bus device.
5. The apparatus of claim 4 wherein the server apparatus is one of a plurality of server apparatuses configured for a server farm within a data center.
6. The apparatus of claim 5 further comprising a power source coupled to the apparatus.
7. The apparatus of claim 1 further comprising an aggregate of transformer devices, the transformer devices comprising a plurality of transformers each of which is stacked in a layer by layer ranging from three (3) to M, where M is an integer up to 128.
8. The apparatus of claim 7 wherein each of the plurality of transformers is configured within one or more DIMC devices such that each of the transformers comprises a plurality of matrix multipliers including a query key value (QKV) matrices configured for an attention layer of a transformer followed by three fully connected (FC) matrices.
9. The apparatus of claim 8 wherein the DIMC device is configured to accelerate the transformer and further comprises a dot product of QK T followed by a softmax (QK T /square root (d k ))V.
10. The apparatus of claim 9 wherein each of the slices includes a single input multiple data (SIMD) device configured to accelerate a computing process of the softmax.
11. The apparatus of claim 1 wherein each of the chiplets comprises four tiles arranged symmetrical to each other, each of the tiles comprises four slices.
12. The apparatus of claim 1 wherein the DIMC device is configured to support one or more block floating point data types using a shared exponent.
13. The apparatus of claim 12 wherein the DIMC device is configured to support a block structured sparsity.
14. The apparatus of claim 1 further comprising a network on chip (NoC) device configured for a multicast process and coupled to each of the plurality of slices.
15. The apparatus of claim 1 wherein the plurality of chiplets are configured to process a workload of a transformer;
wherein the transformer includes a plurality of transformer layers, each of the transformer layers having an attention layer associated with a portion of the workload; and
wherein each attention layer is mapped on to one of the plurality of slices using the global RISC interface to communicate with the CPU associated with the tile of the slice to process the portion of the workload associated with the attention layer.
16. The apparatus of claim 1 wherein the substrate member includes an interposer, and wherein the plurality of chiplets is coupled to each other using the interposer.
17. The apparatus of claim 1 further comprising
a first semiconductor substrate including the plurality of chiplets; and
a second semiconductor substrate comprising a plurality of DRAM memory cells, one of more of the plurality of DRAM memory cells being coupled to the DRAM interface such that the first semiconductor substrate and the second semiconductor substrate are bonded through a mechanical interface.
18. The apparatus of claim 1 wherein the plurality of D2D interconnects comprises a plurality of universal chiplet interconnect express (UCIe) interconnects.
19. The apparatus of claim 1 wherein the plurality of D2D interconnects of each chiplet are configured in at least four input/output (I/O) regions in a symmetrical configuration enabling scalable chiplet array configurations in any die orientation.
20. An AI accelerator apparatus configured with in-memory compute, the apparatus comprising:
at least four chiplets, each of the chiplets comprising a plurality of tiles, and each of the tiles comprising:
a plurality of slices,
a central processing unit (CPU) coupled to the plurality of slices, and
a hardware dispatch device coupled to the CPU;
a first clock configured to output a clock signal of 0.5 GHz to 4 GHz;
a plurality of die-to-die (D2D) interconnects coupled to the each of CPUs in each of the tiles, wherein each of the chiplets is coupled to each other chiplet in a full mesh connectivity configuration using a plurality of intra-chiplet non-diagonal links and a plurality of inter-chiplet non-diagonal links;
a peripheral component interconnect express (PCIe) bus coupled to the CPUs in each of the tiles;
a dynamic random access memory (DRAM) interface coupled to the CPUs in each of the tiles;
a global reduced instruction set computer (RISC) interface coupled to each of the CPUs in each of the tiles;
wherein each of the slices includes a digital in memory compute (DIMC) device coupled to a second clock and configured to allow for a throughput of one or more matrix computations provided in the DIMC device such that the throughput is characterized by 512 multiply accumulates per a clock cycle;
wherein the DIMC device is coupled to the second clock configured at an output rate of one half of the rate of the first clock; and
a substrate member configured to provide mechanical support and having a surface region, the surface region being coupled to support the plurality of chiplets.
21. The apparatus of claim 20 further comprising one or more double data rate (DDR) DRAM devices, the one or more DDR DRAM devices being coupled to one or more chiplets using the DRAM interface.
22. The apparatus of claim 20 further comprising a main bus device, the main bus device being coupled to each PCIe bus in each chiplet using a master chiplet device, the master chiplet device being coupled to each of the other chiplet devices using at least the plurality of D2D interconnects.
23. The apparatus of claim 22 further comprising a server apparatus, the apparatus being configured and operable to the server apparatus using the main bus device.
24. The apparatus of claim 23 wherein the server apparatus is one of a plurality of server apparatuses configured for a server farm within a data center.
25. The apparatus of claim 24 further comprising a power source coupled to the apparatus.
26. The apparatus of claim 20 further comprising an aggregate of transformer devices, the transformer devices comprising a plurality of transformers each of which is stacked in a layer by layer ranging from three (3) to M, where M is an integer up to 128.
27. The apparatus of claim 26 wherein each of the plurality of transformers is configured within one or more DIMC devices such that each of the transformers comprises a plurality of matrix multipliers including a query key value (QKV) matrices configured for an attention layer of a transformer followed by three fully connected (FC) matrices.
28. The apparatus of claim 27 wherein the DIMC device is configured to accelerate the transformer and further comprises a dot product of QK T followed by a softmax (QK T /square root (d k ))V.
29. The apparatus of claim 28 wherein each of the slices includes a single input multiple data (SIMD) device configured to accelerate a computing process of the softmax.
30. The apparatus of claim 20 wherein each of the chiplets comprises four tiles arranged symmetrical to each other, each of the tiles comprises four slices.
31. The apparatus of claim 20 wherein the DIMC device is configured to support one or more block floating point data types using a shared exponent.
32. The apparatus of claim 20 wherein the DIMC device is configured to support a block structured sparsity.
33. The apparatus of claim 20 further comprising a network on chip (NoC) device configured for a multicast process and coupled to each of the plurality of slices.
34. The apparatus of claim 20 wherein the chiplets are configured to process a workload of a transformer;
wherein the transformer includes a plurality of transformer layers, each of the transformer layers having an attention layer associated with a portion of the workload; and
wherein each attention layer is mapped on to one of the plurality of slices using the global RISC interface to communicate with the CPU associated with the tile of the slice to process the portion of the workload associated with the attention layer.
35. The apparatus of claim 20 wherein the substrate member includes an interposer, and wherein the chiplets are coupled to each other using the interposer.
36. The apparatus of claim 20 further comprising
a first semiconductor substrate including the chiplets; and
a second semiconductor substrate comprising a plurality of DRAM memory cells, one of more of the plurality of DRAM memory cells being coupled to the DRAM interface such that the first semiconductor substrate and the second semiconductor substrate are bonded through a mechanical interface.
37. The apparatus of claim 20 wherein the plurality of D2D interconnects comprises a plurality of universal chiplet interconnect express (UCIe) interconnects.
38. The apparatus of claim 20 wherein the plurality of D2D interconnects of each chiplet are configured in at least four input/output (I/O) regions in a symmetrical configuration enabling scalable chiplet array configurations in any die orientation.
39. An AI accelerator apparatus configured with in-memory compute, the apparatus comprising:
a plurality of chiplets coupled together in a full mesh connectivity configuration, each of the chiplets comprising a plurality of tiles, and each of the tiles comprising:
a plurality of slices, and
a central processing unit (CPU) coupled to the plurality of slices;
a plurality of die-to-die (D2D) interconnects coupled to the each of CPUs in each of the tiles, wherein at least one of the D2D interconnects of each chiplet is coupled to one of the D2D interconnects of each other chiplet using a non-diagonal link;
wherein each of the slices includes a digital in memory compute (DIMC) device; and
a substrate member configured to provide mechanical support and having a surface region, the surface region being coupled to support the plurality of chiplets.
40. The apparatus of claim 39 wherein the plurality of chiplets includes at least four chiplets; and wherein each of the chiplets is coupled to each other chiplet in a full mesh connectivity configuration using a plurality of intra-chiplet non-diagonal links and a plurality of inter-chiplet non-diagonal links.