Parallel execution of gated activation unit operations

1. A method for implementing a neural network on a parallel processing device, the method comprising:

receiving, by a computer-implemented system having a processor, a plurality of weight matrices of a gated activation unit of the neural network, the gated activation unit having two or more layers, each layer defining operations comprising: (i) a matrix operation between a weight matrix for the layer and concatenated input vectors and (ii) a nonlinear activation operation using a result of the matrix operation;

interleaving, by the computer-implemented system, rows of the plurality of weight matrices by assigning groups of corresponding rows to respective thread blocks of the parallel processing device, each thread block being a computation unit for execution by an independent processing unit of a plurality of independent processing units of the parallel processing device, wherein each thread block is assigned with a respective interleaved group of rows that include one or more rows from each of the plurality of weight matrices; and

processing, by each thread block in parallel without inter-block synchronization, the respective interleaved group of rows that include the one or more rows from each of the plurality of weight matrices assigned to the thread block to generate a respective output of the gated activation unit.

2. The method of claim 1 , wherein processing, by each thread block in parallel without inter-block synchronization, the respective interleaved group of rows to generate the respective output of the gated activation unit comprises:

receiving, by each thread block, respective input vectors; and

generating, by each thread block in parallel without inter-block synchronization, a respective portion of a current state vector including:

performing a plurality of partial matrix operations using one or more groups of corresponding rows of the plurality of weight matrices assigned to the thread block and a concatenation of the respective input vectors, and

performing a plurality of nonlinear activation operations using respective results of the plurality of partial matrix operations.

3. The method of claim 1 , wherein interleaving the rows comprises assigning a same number of rows to all thread blocks.

4. The method of claim 1 , wherein the gated activation unit is a gated recurrent unit or a long short term memory unit of the neural network.

5. The method of claim 1 , wherein the operations of a layer comprise a bias addition operation between a result of a matrix operation for the layer and a bias vector for the layer, and further comprising:

distributing portions of the bias vector to each thread block of the thread blocks; and

adding, by each thread block, a portion of the bias vector to a result of a matrix operation.

6. The method of claim 1 , further comprising interleaving rows of the plurality of weight matrices by warps such that all warps receive some rows from every weight matrix and corresponding rows from every weight matrix are assigned to the same warp.

7. The method of claim 1 , further comprising interleaving values of the plurality of weight matrices by threads such that all threads receive some values from every weight matrix and corresponding values from every weight matrix are assigned to the same thread.

8. A system configured to implement a neural network on a parallel processing device, the neural network comprising:

at least one gated activation unit having a plurality of weight matrices, the gated activation unit having two or more layers, each layer defining operations comprising: (i) a matrix operation between a weight matrix for the layer and concatenated input vectors and (ii) a nonlinear activation operation using a result of the matrix operation;

wherein the system is configured to:

interleave rows of the plurality of weight matrices by assigning groups of corresponding rows to respective thread blocks of the parallel processing device, each thread block being a computation unit for execution by an independent processing unit of a plurality of independent processing units of the parallel processing device, wherein each thread block is assigned with a respective interleaved group of rows that include one or more rows from each of the plurality of weight matrices; and

process, by each thread block in parallel without inter-block synchronization, the respective interleaved group of rows that include the one or more rows from each of the plurality of weight matrices assigned to the thread block to generate a respective output of the gated activation unit.

9. The system of claim 8 , further configured to:

receive, by each thread block, respective input vectors; and

generate, by each thread block in parallel without inter-block synchronization, a respective portion of a current state vector; wherein each thread block is configured to:

perform a plurality of partial matrix operations using one or more groups of corresponding rows of the plurality of weight matrices assigned to the thread block and a concatenation of the respective input vectors, and

perform a plurality of nonlinear activation operations using respective results of the plurality of partial matrix operations.

10. One or more non-transitory computer storage media encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving a plurality of weight matrices of a gated activation unit of a neural network to be executed on a parallel processing device, the gated activation unit having two or more layers, each layer defining operations comprising: (i) a matrix operation between a weight matrix for the layer and concatenated input vectors and (ii) a nonlinear activation operation using a result of the matrix operation;

interleaving rows of the plurality of weight matrices by assigning groups of corresponding rows to respective thread blocks of the parallel processing device, each thread block being a computation unit for execution by an independent processing unit of a plurality of independent processing units of the parallel processing device, wherein each thread block is assigned with a respective interleaved group of rows that include one or more rows from each of the plurality of weight matrices; and

processing, by each thread block in parallel without inter-block synchronization, the respective interleaved group of rows that include the one or more rows from each of the plurality of weight matrices assigned to the thread block to generate a respective output of the gated activation unit.

11. The one or more non-transitory computer storage media of claim 10 , wherein the computer program instructions further cause the one or more computers to perform:

receiving, by each thread block, respective input vectors; and

generating, by each thread block in parallel without inter-block synchronization, a respective portion of a current state vector including:

performing a plurality of partial matrix operations using one or more groups of corresponding rows of the plurality of weight matrices assigned to the thread block and a concatenation of the respective input vectors, and

performing a plurality of nonlinear activation operations using respective results of the plurality of partial matrix operations.

12. The one or more non-transitory computer storage media of claim 10 , wherein interleaving the rows comprises assigning a same number of rows to all thread blocks.

13. The one or more non-transitory computer storage media of claim 10 , wherein the gated activation unit is a gated recurrent unit or a long short term memory unit of the neural network.

14. The one or more non-transitory computer storage media of claim 10 , wherein:

the operations of a layer comprise a bias addition operation between a result of a matrix operation for the layer and a bias vector for the layer; and

the computer program instructions further cause the one or more computers to further perform:

distributing portions of the bias vector to each thread block of the plurality of thread blocks; and

adding, by each thread block, a portion of the bias vector to a result of a matrix operation.

15. The one or more non-transitory computer storage media of claim 10 , wherein the computer program instructions further cause the one or more computers to perform:

interleaving rows of the plurality of weight matrices by warps such that all warps receive some rows from every weight matrix and corresponding rows from every weight matrix are assigned to the same warp.