System and computer-executable program code for accelerated rescoring with recurrent neural net language models on hybrid CPU/GPU machines using a frame-wise, delayed dispatch of RNNLM score computation tasks to the GPU(s)
A system and computer-executable program code for accelerated rescoring with recurrent neural net language models (RNNLMs) on hybrid CPU/GPU computer systems utilizes delayed, frame-wise (or layer-wise) batch dispatch of RNNLM scoring tasks to the GPU(s), while performing substantially all other tasks on the CPU(s).
1. A system for rescoring a weighted finite-state transducer (WFST) lattice, using a recurrent neural network language model (RNNLM), said system implemented on a hybrid computing platform that includes a central processing unit (CPU) and a graphics processing unit (GPU), said system further comprising at least the following:
computer executable code for storing a RNNLM; and,
a module for performing a RNNLM rescoring of a WFST lattice, said module for performing a RNNLM rescoring comprising:
computer executable code for obtaining a maximum order for the RNNLM;
computer executable code for specifying a RNNLM search beam;
computer executable code for allocating a hash table;
computer executable code for adding a first RNNLM node, corresponding to a first frame, onto an RNNLM nodelist;
computer executable code for performing n-gram rescoring on the CPU;
computer executable code for adding additional RNNLM nodes onto the RNNLM nodelist;
first computer executable code for dispatching the RNNLM nodelist for batch score computation on the GPU;
a module for processing additional frame(s), said module including:
computer executable code for obtaining a cutoff look-ahead value, based on said RNNLM search beam and other parameters;
computer executable code for adding RNNLM nodes to an additional RNNLM nodelist based on words in the WFST that end in a next frame of interest; and,
second computer executable code for dispatching the additional RNNLM nodelist for batch score computation on the GPU.
2. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , wherein the computer executable code for storing the RNNLM includes a linked list of RNNLM nodes.
3. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , wherein the module for performing the RNNLM rescoring further includes computer executable code for excluding filler words from the RNNLM rescoring process.
4. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , wherein the module for performing the RNNLM rescoring further includes computer executable code for deallocating the list of RNNLM nodes after they are processed by the GPU batch score computation process.
5. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , wherein the module for performing the RNNLM rescoring further includes computer executable code for copying an input layer of the RNNLM to the RNNLM nodelist.
6. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 5 , wherein the module for performing the RNNLM rescoring further includes computer executable code for copying a hidden layer of the RNNLM to the RNNLM nodelist.
7. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , further including computer executable code for selecting a best path through the WFST lattice using RNNLM scores.
8. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 7 , wherein the computer executable code for selecting a best path through the WFST lattice using RNNLM scores includes computer executable code for hashing computed partial paths through portions of the WFST lattice.
9. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , further including computer executable code for storing vocabulary words associated with the RNNLM.
10. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 9 , wherein the computer executable code for storing vocabulary words associated with the RNNLM separates the vocabulary words into multiple classes.
11. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , wherein the computer executable code for dispatching the RNNLM nodelist for batch score computation on the GPU receives a linked list of RNNLM nodes, where each node contains index, current word, history, hidden layer, etc., and the computed RNNLM score(s) are returned through this linked list structure.
12. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 11 , wherein the computer executable code for dispatching the RNNLM nodelist for batch score computation on the GPU receives one or more scaling constants.
13. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 11 , wherein the computer executable code for dispatching the RNNLM nodelist for batch score computation on the GPU receives a score to associate with word(s) that are out of the RNNLM vocabulary.
14. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 11 , wherein the computer executable code for dispatching the RNNLM nodelist for batch score computation on the GPU receives an index to the nodes in the RNNLM nodelist.
15. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 11 , wherein the computer executable code for dispatching the RNNLM nodelist for batch score computation on the GPU receives an indication of the RNNLM hidden layer size.
16. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 11 , wherein the computer executable code for dispatching the RNNLM nodelist for batch score computation on the GPU receives a value indicative of a number of vocabulary classes.
17. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 16 , wherein the computer executable code for dispatching the RNNLM nodelist for batch score computation on the GPU receives lists of words for each vocabulary class.
18. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , further including computer executable code for loading relevant portion(s) of the RNNLM into CPU RAM.
19. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , further including computer executable code for clearing irrelevant portion(s) of the RNNLM from CPU RAM.
20. A system for rescoring the WFST lattice, using the RNNLM, as defined in claim 1 , further including computer executable code for transferring relevant portion(s) of the RNNLM from CPU RAM to GPU RAM.