Compressed recurrent neural network models
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a compressed recurrent neural network (RNN). One of the systems includes a compressed RNN, the compressed RNN comprising a plurality of recurrent layers, wherein each of the recurrent layers has a respective recurrent weight matrix and a respective inter-layer weight matrix, and wherein at least one of recurrent layers is compressed such that a respective recurrent weight matrix of the compressed layer is defined by a first compressed weight matrix and a projection matrix and a respective inter-layer weight matrix of the compressed layer is defined by a second compressed weight matrix and the projection matrix.
1. A computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations comprising:
re-configuring an uncompressed version of a trained recurrent neural network (RNN) to generate a compressed version of the trained RNN by replacing a particular uncompressed trained recurrent layer of a plurality of uncompressed recurrent layers of the uncompressed version of the trained RNN with a corresponding compressed trained recurrent layer by:
generating a first compressed weight matrix, Z h l , and a projection matrix, P l , based on an uncompressed recurrent weight matrix, W h , containing trained recurrent weights for the particular uncompressed trained recurrent layer of the trained RNN;
generating a second compressed weight matrix, Z x l , based on the first compressed weight matrix, Z h l , and the projection matrix, P l ; and
generating, based on a product of the first compressed weight matrix, Z h l , and the projection matrix, P l , a compressed trained recurrent weight matrix that replaces the uncompressed recurrent weight matrix, W h , in the corresponding compressed trained recurrent layer, the compressed trained recurrent weight matrix comprising fewer parameters than the uncompressed recurrent weight matrix, W h ; and
transmitting the compressed version of the trained RNN having the the corresponding compressed trained recurrent layer to a device, the compressed version of the trained RNN having the corresponding compressed trained recurrent layer configured to receive a respective neural network input at each of multiple time steps and generate a respective neural network output at each of the multiple time steps,
wherein, after compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers, the compressed version of the trained RNN comprises fewer parameters than the uncompressed version of the trained RNN.
2. The computer-implemented method of claim 1 , wherein respective ranks of the first compressed weight matrix, Z h l , and the projection matrix, P l , are less than a respective rank of the uncompressed recurrent weight matrix, W h .
3. The computer-implemented method of claim 1 , wherein re-configuring the trained RNN further comprises re-configuring the trained RNN by compressing the particular uncompressed trained recurrent layer by replacing a respective inter-layer weight matrix, W x , for the particular uncompressed trained recurrent layer of the uncompressed version of the trained RNN with the product of the second compressed weight matrix, Z x l , and the projection matrix, P l .
4. The computer-implemented method of claim 3 , wherein respective ranks of the second compressed weight matrix, Z x l , and the projection matrix, P l , are less than a respective rank of the respective inter-layer weight matrix, W x , for the recurrent layer.
5. The computer-implemented method of claim 1 , wherein generating the first compressed weight matrix, Z h l , and the projection matrix, P l , comprises:
determining a respective singular value decomposition (SVD) of the uncompressed recurrent weight matrix, W h ; and
generating the first compressed weight matrix, Z h l , and the projection matrix, P l , based on the respective SVD of the respective recurrent weight matrix, W h .
6. The computer-implemented method of claim 1 , wherein each recurrent layer of the plurality of uncompressed recurrent layers is configured to, for each of a plurality of time steps, generate a respective layer output for the time step by applying an inter-layer weight matrix, W x l−l , for a previous recurrent layer to a current input to the recurrent layer and applying the respective recurrent weight matrix, W h l , for the recurrent layer to a respective recurrent input to the recurrent layer.
7. The computer-implemented method of claim 6 , further comprising, after compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers, generating the respective layer output, for each time step, by applying the first compressed weight matrix, Z h l , and the projection matrix, P l , to the respective recurrent input to the recurrent layer.
8. The computer-implemented method of claim 1 , wherein the RNN comprises an acoustic model.
9. The computer-implemented method of claim 1 , wherein the RNN comprises a speech recognition model.
10. The computer-implemented method of claim 1 , wherein, after compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers, the compressed version of the trained RNN comprises at least a 68% compression of the uncompressed version of the trained RNN prior to compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers.
11. The computer-implemented method of claim 1 , wherein, after compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers, a word error rate of the compressed version of the trained RNN is within 5% of a word error rate of the uncompressed version of the trained RNN prior to compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers.
12. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising:
re-configuring an uncompressed version of a trained recurrent neural network (RNN) to generate a compressed version of the trained RNN by replacing a particular uncompressed trained recurrent layer of a plurality of uncompressed recurrent layers of the uncompressed version of the trained RNN with a corresponding compressed trained recurrent layer by:
generating a first compressed weight matrix, Z h l , and a projection matrix, P l , based on an uncompressed recurrent weight matrix, W h , containing trained recurrent weights for the particular uncompressed trained recurrent layer of the trained RNN;
generating a second compressed weight matrix, Z x l , based on the first compressed weight matrix, Z h l , and the projection matrix, P l ; and
generating, based on a product of the first compressed weight matrix, Z h l , and the projection matrix, P l , a compressed trained recurrent weight matrix that replaces the uncompressed recurrent weight matrix, W h , in the corresponding compressed trained recurrent layer, the compressed trained recurrent weight matrix comprising fewer parameters than the uncompressed recurrent weight matrix, W h ; and
transmitting the compressed version of the trained RNN having the the corresponding compressed trained recurrent layer to a device, the compressed version of the trained RNN having the corresponding compressed trained recurrent layer configured to receive a respective neural network input at each of multiple time steps and generate a respective neural network output at each of the multiple time steps,
wherein, after compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers, the compressed version of the trained RNN comprises fewer parameters than the uncompressed version of the trained RNN.
13. The system of claim 12 , wherein respective ranks of the first compressed weight matrix, Z h l , and the projection matrix, P l , are less than a respective rank of the uncompressed recurrent weight matrix, W h .
14. The system of claim 12 , wherein re-configuring the trained RNN further comprises re-configuring the trained RNN by compressing the particular uncompressed trained recurrent layer by replacing a respective inter-layer weight matrix, W x , for the particular uncompressed trained recurrent layer of the uncompressed version of the trained RNN with the product of the second compressed weight matrix, Z x l , and the projection matrix, P l .
15. The system of claim 14 , wherein respective ranks of the second compressed weight matrix, Z x l , and the projection matrix, P l , are less than a respective rank of the respective inter-layer weight matrix, W x , for the recurrent layer.
16. The system of claim 12 , wherein generating the first compressed weight matrix, Z h l , and the projection matrix, P l , comprises:
determining a respective singular value decomposition (SVD) of the uncompressed recurrent weight matrix, W h ; and
generating the first compressed weight matrix, Z h l , and the projection matrix, P l , based on the respective SVD of the respective recurrent weight matrix, W h .
17. The system of claim 12 , wherein each recurrent layer of the plurality of uncompressed recurrent layers is configured to, for each of a plurality of time steps, generate a respective layer output for the time step by applying an inter-layer weight matrix, W x l−1 , for a previous recurrent layer to a current input to the recurrent layer and applying the respective recurrent weight matrix, W h l , for the recurrent layer to a respective recurrent input to the recurrent layer.
18. The system of claim 17 , wherein the operations further comprising, after compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers, generating the respective layer output, for each time step, by applying the first compressed weight matrix, Z h l , and the projection matrix, P l , to the respective recurrent input to the recurrent layer.
19. The system of claim 12 , wherein the RNN comprises an acoustic model.
20. The system of claim 12 , wherein the RNN comprises a speech recognition model.
21. The system of claim 12 , wherein, after compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers, the compressed version of the trained RNN comprises at least a 68% compression of the uncompressed version of the trained RNN prior to compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers.
22. The system of claim 12 , wherein, after compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers, a word error rate of the compressed version of the trained RNN is within 5% of a word error rate of the uncompressed version of the trained RNN prior to compressing the particular uncompressed trained recurrent layer of the plurality of uncompressed recurrent layers.