Hardware accelerator for compressed RNN on FPGA

1. A device for implementing compressed RNN (recurrent neural network), said device comprising:

a receiving unit, which is used to receive a plurality of input vectors and distribute them to a plurality of processing elements (PE);

the plurality of processing elements (PE), each of which comprising:

a reading unit configured to read weight matrices W, said W indicates weights of said RNN;

an Arithmetic Logic Unit (ALU) configured to perform multiplication and addition calculation of said weight matrices W; and

a calculation buffer configured to store intermediate results of matrix-vector multiplication and output the results to an assembling unit;

the assembling unit configured to receive the results from the PEs and assemble the results into a complete result vector; and

a controller unit configured for implementing a state machine to control said plurality of processing elements and the receiving unit for the plurality of processing elements and the receiving unit to work in a parallelized pipeline, wherein the state machine has three states, an initial state in which an input vector and a corresponding weight matrix are prepared, a state 1 and a state 2, wherein, during each of the state 1 and state 2, calculation of matrix-vector multiplication of a current layer is performed in parallel with accumulation and activation function operation of a preceding layer to achieve pipeline computation, wherein the state 1 and state 2 are conducted alternatively.

2. The device of claim 1 , further comprising:

a hidden layer unit, configured to perform a hidden layer's activation function of said RNN.

3. The device of claim 2 , further comprising:

a vector buffer configured to receive an activation vector of the hidden layer from said hidden layer unit and store the activation vector and a bias vector.

4. The device of claim 3 , further comprising:

an adder tree unit configured to perform vector accumulation on vectors output from the assembling unit and the vector buffer.

5. The device of claim 1 , wherein said receiving unit further comprises:

a plurality of first-in-first-out buffer, each of which corresponds to a PE.

6. The device of claim 1 , wherein said reading unit further comprises:

a location unit configured to read and store location information of non-zero elements in the weight matrices W;

a decoding unit configured to decode the weight matrix to obtain the weight value of the weight matrices W.

7. The device of claim 1 , wherein said ALU comprises:

a plurality of multipliers configured to perform multiplication operation on matrix elements and input vector elements;

a plurality of adders configured to perform addition operation on results of the multiplication operation.

8. The device of claim 1 , wherein said calculation buffer comprises:

a first output buffer and a second output buffer, said first output buffer and second output buffer receive and output a computation result alternatively, wherein while one buffer receives a present computation result, the other buffer outputs a previous computation result.

9. The device of claim 3 , wherein said vector buffer further comprises:

a plurality of sub-buffers configured to store computation results of respective hidden layers.

10. A method for implementing a compressed Recurrent Neural Network (RNN) based on a device, comprising:

a) receiving data from an off-chip memory and storing the data into anon-chip memory of the device, wherein said data are related to RNN computation, said data including an input vector, a bias vector, and weight matrices;

b) decoding the data received in step a) using anon-chip processor of the device in order to obtain real weights, and storing the real weights into the on-chip memory of the device;

c) matrix computing by performing matrix-vector multiplication using on-chip processing elements of the device and storing a result of the matrix computation into the on-chip memory of the device;

d) vector accumulating by performing vector accumulation on vectors using the on-chip processing elements of the device and storing a result of the vector accumulation into the on-chip memory of the device, said vectors including both a resultant vector obtained in step c) and said bias vector;

e) activation function computing by performing activation function on the result of the vector accumulation obtained in the above step d), and storing a result of the activation function into the on-chip memory of the device; and

iterating the above steps a), b), c), d), e) to transition between three different states of a state machine to obtain the RNN's activation sequences and computing the RNN's output sequence according to the activation sequences in a parallelized pipeline, wherein the three different states of the state machine include: an initial state, a state 1 and a state 2, wherein during the initial state steps a) and b) are performed, wherein during each of the state 1 and the state 2 steps c), d) and e) are performed to carry out calculation of matrix-vector multiplication of a current layer in parallel with accumulation and activation function operation of a preceding layer to achieve pipeline computation, wherein the state 1 and state 2 are conducted alternatively.

11. The method of claim 10 , further comprising:

distributing the received data to the on-chip processing elements (PE) after receiving data in step a); and

assembling results from each PE to obtain a complete result vector of matrix-vector multiplication after computation in step c).

12. The method of claim 11 , further comprising:

in each PE, providing a plurality of on-chip adders and multipliers for parallel processing.

13. The method of claim 10 , in at least one of said steps a), b), c), providing a pair of ping-pong buffers in the on-chip memory.

14. A method for implementing a Recurrent Neural Network (RNN), wherein weights of said RNN being characterized by W hh and W hx , where W hh is a weight matrix of hidden layers of said RNN, W hx is a weight matrix being applied to input of the hidden layers, where an activation to be applied to an input vector by said hidden layers is h t , the input of the hidden layers of said RNN is a series of input vectors x=(x 1 , x 2 , . . . , x T ), said method comprising:

an initialization step of reading a first piece of data for computing W hxx into anon-chip memory of a device, said first piece of data including an input vector of the series of input vectors x and W hx ;

step 1 of computing W hx x by processing elements of said device and reading a second piece of data for computing W hh x into the on chip memory of the device, the second piece of data including W hh ;

step 2 of computing W hh h t−1 by the processing elements of said device, where h t−1 is an activation to be applied to a previous input vector by the hidden layer, and reading a third piece of data for computing a next W hx x into the on-chip memory of the device, the third piece of data including a next input vector; and

iteratively repeating said step 1 and step 2, wherein the initialization step, step 1 and step 2 are performed during three states of a state machine corresponding to an initial state, a state 1 and a state 2, wherein, during each of the state 1 and state 2, calculation of matrix-vector multiplication of a current layer is performed in parallel with accumulation and activation function operation of a preceding layer to achieve pipeline computation.

15. The method of claim 14 , wherein each of said Step 1 and Step 2 further comprises:

while computing matrix-vector multiplication for a present input vector, computing the activation h t of the hidden layers and the RNN's output y t .

16. The method of claim 14 , wherein:

said initial step, step 1 and step 2 are processed sequentially; and

said step 1 and step 2 are processed periodically.