Connectionist temporal classification using segmented labeled sequence data
Classification training systems and methods include a neural network for classification of input data, a training dataset providing segmented labeled training data, and a classification training module operable to train the neural network using the training data. A forward pass processing module is operable to generate neural network outputs for the training data using weights and bias for the neural network, and a backward pass processing module is operable to update the weights and biases in a backward pass, including obtaining Region of Target (ROT) information from the training data, generate a forward-backward masking based on the ROT information, the forward-backward masking placing at least one restriction on a neural network output path, compute modified forward and backward variables based on the neural network outputs and the forward-backward masking, and update the weights and biases.
1. A computer-implemented method for training a neural network for phoneme recognition comprising:
receiving, at a computing device, a stream of segmented, labeled training data having a sequence of frames;
generating, by the computing device, neural network outputs for the sequence of frames in a forward pass through the training data and in accordance with weights and biases; and
updating, by the computing device, the weights and biases through a backward pass through the training data, the backward pass comprising:
obtaining Region of Target (ROT) information from the stream of segmented, labeled training data;
generating a forward-backward masking based on the ROT information, the forward-backward masking placing at least one restriction on a neural network output path;
computing modified forward and backward variables based on the neural network outputs and the forward-backward masking; and
updating the weights and biases of the neural network.
2. The computer-implemented method of claim 1 , wherein updating, by the computing device, the weights and biases through the backward pass further comprises computing a soft target using the forward-backward masking and the neural network outputs.
3. The computer-implemented method of claim 2 , wherein updating, by the computing device, weights and biases through the backward pass further comprises computing a signal error based on the neural network outputs and the soft target.
4. The computer-implemented method of claim 3 , further comprising identifying target and shared regions in the stream of segmented, labeled training data and wherein the soft target is computed only for identified target and shared regions.
5. The computer-implemented method of claim 1 , wherein updating the weights and biases of the neural network further comprises adaptively learning to improve a convergence rate of the neural network.
6. The computer-implemented method of claim 5 , wherein adaptively learning to improve a convergence rate of the neural network comprises a first algorithm based on momentum for the weights and biases of an output layer of the neural network.
7. The computer-implemented method of claim 6 , wherein adaptively learning to improve a convergence rate of the neural network further comprises a second algorithm based on a Root Mean Square (RMS) of a signal error for other weights and biases.
8. The computer-implemented method of claim 1 , wherein computing the forward-backward masking further comprises, computing modified forward and backward variables in a logarithm domain, thereby avoiding underflow.
9. The computer-implemented method of claim 1 , wherein the neural network is a recurrent neural network.
10. A classification training system comprising:
a neural network for use in a classification of input data;
a training dataset providing segmented labeled training data; and
a classification training module operable to train the neural network using the segmented labeled training data, the classification training module comprising a forward pass processing module, and a backward pass processing module;
wherein the forward pass processing module is operable to train the neural network by generating neural network outputs for the training data using weights and bias for the neural network; and
wherein the backward pass processing module is operable to train the neural network by updating the weights and biases in a backward pass, wherein the backward pass module is operable to:
obtain Region of Target (ROT) information from the segmented, labeled training data;
generate a forward-backward masking based on the ROT information, the forward-backward masking placing at least one restriction on a neural network output path;
compute modified forward and backward variables based on the neural network outputs and the forward-backward masking; and
update the weights and biases of the neural network.
11. The classification training system of claim 10 , wherein the backward pass processing module is further operable to update the weights and biases by computing a soft target using the forward-backward masking and the neural network outputs.
12. The classification training system of claim 11 , wherein the backward pass processing module is further operable to update the weights and biases by computing a signal error based on the neural network outputs and the soft target.
13. The classification training system of claim 12 , wherein the backward pass processing module is further operable to identify target and shared regions in the segmented, labeled training data and wherein the soft target is computed only for identified target and shared regions.
14. The classification training system of claim 11 , wherein the backward pass processing module is further operable to update the weights and biases of the neural network by adaptively learning to improve a convergence rate of the neural network.
15. The classification training system of claim 14 , wherein adaptively learning to improve a convergence rate of the neural network comprises a first algorithm based on momentum for the weights and biases of an output layer of the neural network.
16. The classification training system of claim 15 , wherein adaptively learning to improve a convergence rate of the neural network further comprises a second algorithm based on a Root Mean Square (RMS) of a signal error for other weights and biases.
17. The classification training system of claim 11 , wherein the backward pass processing module is further operable to compute the forward-backward masking by computing modified forward and backward variables in a logarithm domain, thereby avoiding underflow.
18. The classification training system of claim 11 , wherein the neural network is a recurrent neural network.
19. The classification training system of claim 11 , wherein the neural network is a long short-term memory network.
20. The classification training system of claim 11 , wherein the classification training system is a phoneme recognition system, and wherein the backward pass processing module is further operable to update the weights and biases by computing a signal error for output nodes of the neural network, thereby allowing the phoneme recognition system to operate without latency.