End-to-end neural network based automated speech scoring

1. A system for end-to-end automated scoring, the system comprising: a word embedding layer for converting a plurality of automatic speech recognition (ASR) outputs into input tensors; a neural network lexical model encoder receiving the input tensors; a neural network acoustic model (AM) encoder implementing AM posterior probability, word duration, mean value of pitch and mean value of intensity based on a plurality of cues; and a linear regression module, for receiving concatenated encoded features from the neural network lexical model encoder and the neural network acoustic model encoder.

2. The system of claim 1 , wherein the neural network lexical model encoder and the neural network acoustic model encoder each implements one of a 1D Convolutional Neural Network, a Bi-Directional Recurrent Neural Network (RNN) using Long Short Time Memory (LSTM) cells (BD-LSTM), and BD-LSTM RNN with an attention weighting scheme.

3. The system of claim 2 , wherein the 1D Convolutional Neural Network further comprises:

a layer for receiving input for ASR outputs using an embedding layer;

a dropout layer for improving over-fit;

a layer of one dimensional convolution with multiple varied-sized filters;

a max overtime pooling layer; and

a second dropout layer for improving over-fit.

4. The system of claim 2 , wherein the Bi-Directional Recurrent Neural Network using Long Short Time Memory cells further comprises:

a layer for receiving input for ASR outputs using an embedding layer;

a dropout layer for improving over-fit;

a layer of Bi-Directional Recurrent Neural Network using Long Short Time Memory cells; and

a second dropout layer for improving over-fit.

5. The system of claim 2 , wherein the Bi-Directional Recurrent Neural Network using Long Short Time Memory cells with an attention weighting scheme further comprises:

a layer for receiving input for ASR outputs using an embedding layer;

a dropout layer for improving over-fit;

a layer of Bi-Directional Recurrent Neural Network using Long Short Time Memory cells;

an attention layer; and

a second dropout layer for improving over-fit.

6. The system of claim 4 , wherein the Bi-Directional Recurrent Neural Network processes a sequence of input data by recursively applying a transitional function to a hidden state vector h t .

7. The system of claim 6 , wherein the activation of h t at a time-step t depends on both a current input x t and a previous hidden state h t-1 by h t =f(h t-1 , x t ).

8. The system of claim 7 , wherein the Bi-Directional Recurrent Neural Network encodes an input sequence to a fixed-sized vector h T on a last time step T and uses the fixed-sized vector h T as a input for a following prediction step.

9. The system of claim 8 , wherein the LSTM cells are defined by a plurality of vectors, wherein the plurality of vectors further comprises: an input gate i t , a forget gate f t , an output gate o t , a memory cell c t and a hidden state h t .

10. The system of claim 9 , wherein at each time step, the LSTM maintains a hidden vector h and a memory vector c for controlling state updates and outputs.

11. The system of claim 10 , wherein the computation at time step t is defined as:

i t =σ( W i x t +U i h t-1 +V i c t-1 )

f t =σ( W f x t +U f h t-1 +V f c t-1 )

o t =σ( W o x t +U o h t-1 +V o c t )

c t =f t ⊙c t-1 +i t ⊙ tan h ( W c x t +U c h t-1 )

h t =o t ⊙ tan h ( c t )

wherein x t is an input at the current time step t, σ is tahe logistic sigmoid function.

12. A method for end-to-end automated scoring, the method comprising the steps of: deploying a word embedding layer for converting a plurality of automatic speech recognition (ASR) outputs into input tensors; deploying a neural network lexical model encoder receiving the input tensors; deploying a neural network acoustic model (AM) encoder implementing AM posterior probability, word duration, mean value of pitch and mean value of intensity based on a plurality of cues; and deploying a linear regression module, for receiving concatenated encoded features from the neural network lexical model encoder and the neural network acoustic model encoder.

13. The method of claim 12 , wherein the neural network lexical model encoder and the neural network acoustic model encoder is each constructed by implementing one of a 1D Convolutional Neural Network, a Bi-Directional Recurrent Neural Network (RNN) using Long Short Time Memory (LSTM) cells (BD-LSTM), and BD-LSTM RNN with an attention weighting scheme.

14. The method of claim 13 , wherein the 1D Convolutional Neural Network is constructed by:

deploying a layer for receiving input for ASR outputs using an embedding layer;

deploying a dropout layer for improving over-fit;

deploying a layer of one dimensional convolution with multiple varied-sized filters;

deploying a max overtime pooling layer; and

deploying a second dropout layer for improving over-fit.

15. The method of claim 14 , wherein the Bi-Directional Recurrent Neural Network using Long Short Time Memory cells is constructed by:

deploying a layer for receiving input for ASR outputs using an embedding layer;

deploying a dropout layer for improving over-fit;

deploying a layer of Bi-Directional Recurrent Neural Network using Long Short Time Memory cells; and

deploying a second dropout layer for improving over-fit.

16. The method of claim 15 , wherein the Bi-Directional Recurrent Neural Network using Long Short Time Memory cells with an attention weighting scheme is constructed by:

deploying a layer for receiving input for ASR outputs using an embedding layer;

deploying a dropout layer for improving over-fit;

deploying a layer of Bi-Directional Recurrent Neural Network using Long Short Time Memory cells;

deploying an attention layer; and

deploying a second dropout layer for improving over-fit.

17. The method of claim 16 , wherein the Bi-Directional Recurrent Neural Network processes a sequence of input data by recursively applying a transitional function to a hidden state vector h t .

18. The method of claim 17 , wherein the activation of h t at a time-step t depends on both a current input x t and a previous hidden state h t-1 by h t =f(h t-1 , x t ).

19. The method of claim 18 , wherein the Bi-Directional Recurrent Neural Network encodes an input sequence to a fixed-sized vector h T on a last time step T and uses the fixed-sized vector h T as a input for a following prediction step.

20. The method of claim 19 , wherein the LSTM cells are defined by a plurality of vectors, wherein the plurality of vectors further comprises: an input gate i t , a forget gate f t , an output gate o t , a memory cell c t and a hidden state h t .