SPATIAL ARCHITECTURE FOR ATTENTION MECHANISMS
The present specification discloses a computing device architecture and method for executing computationally intensive attention mechanisms, as commonly utilized in transformer models for artificial intelligence. In an embodiment, the device includes a row of interconnected processing elements (PEs), each linked to dedicated coefficient memory units (CRAMs) and controlled by a central controller. By employing a diagonal-offset and transposition technique for matrix coefficients, the architecture enables efficient execution of Generalized Matrix-Vector (GEMV) operations.
1 . A method of operating a computing device having a plurality of processing elements connected to a controller, the method comprising:
receiving, by the controller, at least a portion of an input matrix into the processing elements; the input matrix having undergone a pre-scaling operation;
loading, by the controller, a coefficient into a memory unit associated with each of the processing elements in a diagonally-offset configuration;
transposing the coefficients by shifting each coefficient from its respective memory unit in a first processing element to another memory location in an adjacent processing element;
executing, within the processing elements, a computational operation based on the transposed coefficients within the processing elements; and,
controlling, by the controller, an output device based on the executed computational operation.
2 . The method of claim 1 wherein the computational operation is a multi-step computational operation based on one or more of a GEMV operation, a Softmax operation, and/or an attention operation.
3 . The method of claim 1 wherein the pre-scaling operation approximates e{circumflex over ( )}x as 2{circumflex over ( )}x by applying a pre-scaling factor.
4 . The method of claim 3 , wherein the approximation is achieved by applying a hardware bit-shifting operation within each processing element.
5 . The method of claim 1 , wherein the diagonally-offset configuration of coefficients is achieved using a mask-controlled selective shifting operation across a subset of processing elements.
6 . The method of claim 1 , wherein the transposing of coefficients includes a rotation operation performed by a rotator link connecting adjacent processing elements, the rotation configured to align coefficients for matrix-vector multiplication.
7 . The method of claim 1 , wherein the pre-scaling operation includes subtracting the maximum value from all elements in the input matrix to enhance numerical stability during subsequent computations.
8 . The method of claim 1 , wherein the computational operation comprises a GEMV-Broadcast operation in which a single input element is broadcast across all processing elements.
9 . The method of claim 1 , wherein the computational operation comprises a GEMV-Reduce operation that aggregates intermediate results from multiple processing elements using a reduce-sum operation.
10 . The method of claim 1 , wherein the transposition of coefficients includes a skewing operation that aligns diagonally-offset coefficients for parallel processing.
11 . The method of claim 1 , further comprising performing a normalization operation on the results of the computational operation by broadcasting an inverse sum value across the processing elements.
12 . The method of claim 1 , wherein the memory unit associated with each processing element is a coefficient random access memory (CRAM) configured to store at least 64 coefficients.
13 . The method of claim 1 , further comprising executing a computational operation in which two separate computational operations are performed in parallel by distinct groups of processing elements.
14 . The method of claim 1 , wherein the diagonally-offset configuration is achieved by sequentially writing coefficients into memory locations in a staggered pattern across the processing elements.
15 . The method of claim 1 , wherein the transposing of coefficients utilizes four parallel groups of rotator links to perform concurrent rotation operations across subsets of the processing elements.
16 . The method of claim 1 , wherein the computational operation includes a Softmax operation that utilizes the pre-scaled coefficients for exponentiation and normalization.
17 . The method of claim 1 , further comprising performing an activation operation, including one of pooling, summation, or averaging, on the output of the computational operation.
18 . The method of claim 1 , wherein the input matrix is divided into smaller submatrices, each submatrix being processed independently across the processing elements.
19 . The method of claim 1 , wherein the computational operation involves a bi-directional GEMV operation, wherein input data is processed simultaneously in both forward and reverse directions across the processing elements.