Exploring ASR-free end-to-end modeling to improve spoken language understanding in a cloud-based dialog system

1. A computer-implemented method for conducting a simulated conversation with a language learner, the method comprising:

determining, by one or more data processors, a first dialog state of the simulated conversation;

transmitting, by the one or more data processors, first audio data corresponding to simulated speech based on the first dialog state;

receiving, by the one or more data processors, second audio data corresponding to a variable length utterance spoken in response to the simulated speech, metrics associated with the variable length utterance being stored in a variable length data structure whose length is based on a length of the variable length utterance;

generating, by the one or more data processors, a fixed dimension vector based on the variable length utterance, a length of the fixed dimension vector being fixed regardless of the length of the variable length utterance, the fixed dimension vector being generated by a factor analysis that samples a t-th frame of the variable length utterance from a distribution

x t ˜Σ j γ jt N ( m j +T j V,Σ j )

where x t represents the variable length utterance, m j represents a mean, Σ j represents a covariance of a j-th Gaussian component, γ jt represents a statistical alignment result of the frame t, T j represents a total variability, N is a constant, and V represents an utterance-specific standard normal distributed latent vector obtained by maximum a posterior (MAP) estimation;

predicting, by the one or more data processors, a semantic label for the variable-length utterance based on the fixed dimension vector using both of (i) a recurrent neural network (RNN) in which an output of the factor analysis is an input layer to the RNN and an output layer of the RNN is a softmax layer with N one-hot vectors that each represent a different one of N dialog states that does not use automated speech recognition (ASR); and (ii) a convolutional neural network using ASR output as an input;

determining, by one or more data processors, a second dialog state of the simulated conversation based on the semantic label; and

transmitting, by the one or more data processors, third audio data corresponding to simulated speech based on the second dialog state.

2. The computer-implemented method of claim 1 , wherein the fixed dimension vector is generated further based on a recurrent neural network (RNN)-based acoustic auto-encoder.

3. The computer-implemented method of claim 1 , wherein the factor analysis uses multi-dimensional Mel-frequency cepstral coefficients (MFCCs) and first and second derivatives of the MFCCs.

4. The computer-implemented method of claim 1 , wherein the semantic label is based on the first dialog state and comprises one of M semantic classes for the first dialog state.

5. The computer-implemented method of claim 1 , wherein the simulated conversation corresponds to multi-task learning, wherein training data is shared among each task in a neural network.

6. The computer-implemented method of claim 1 , wherein the language learner is a non-native English speaker.

7. The computer-implemented method of claim 1 , wherein the third audio is transmitted across a computer network for playing through a speaker.

8. A computer-implemented system for conducting a simulated conversation with a language learner, the system comprising:

one or more data processors;

a non-transitory computer-readable medium encoded with instructions for commanding the one or more data processors to execute steps of a process that include:

determining, by one or more data processors, a first dialog state of the simulated conversation;

transmitting, by the one or more data processors, first audio data corresponding to simulated speech based on the first dialog state;

receiving, by the one or more data processors, second audio data corresponding to a variable length utterance spoken in response to the simulated speech, metrics associated with the variable length utterance being stored in a variable length data structure whose length is based on a length of the variable length utterance;

generating, by the one or more data processors, a fixed dimension vector based on the variable length utterance, a length of the fixed dimension vector being fixed regardless of the length of the variable length utterance, the fixed dimension vector being generated by a factor analysis that samples a t-th frame of the variable length utterance from a distribution

x t ˜Σ j γ jt N ( m j +T j V,Σ j )

where x t represents the variable length utterance, m j represents a mean, Σ j represents a covariance of a j-th Gaussian component, γ jt represents a statistical alignment result of the frame t, T j represents a total variability, N is a constant, and V represents an utterance-specific standard normal distributed latent vector obtained by maximum a posterior (MAP) estimation;

predicting, by the one or more data processors, a semantic label for the variable-length utterance based on the fixed dimension vector using both of (i) a recurrent neural network (RNN) in which an output of the factor analysis is an input layer to the RNN and an output layer of the RNN is a softmax layer with N one-hot vectors that each represent a different one of N dialog states that does not use automated speech recognition (ASR); and (ii) a convolutional neural network using ASR output as an input;

determining, by one or more data processors, a second dialog state of the simulated conversation based on the semantic label; and

transmitting, by the one or more data processors, third audio data corresponding to simulated speech based on the second dialog state.

9. The computer-implemented system of claim 8 , wherein the fixed dimension vector is generated further based on a recurrent neural network (RNN)-based acoustic auto-encoder.

10. The computer-implemented system of claim 8 , wherein the factor analysis uses multi-dimensional Mel-frequency cepstral coefficients (MFCCs) and first and second derivatives of the MFCCs.

11. The computer-implemented system of claim 8 , wherein the semantic label is based on the first dialog state and comprises one of M semantic classes for the first dialog state.

12. The computer-implemented system of claim 8 , wherein the simulated conversation corresponds to multi-task learning, wherein training data is shared among each task in a neural network.

13. The computer-implemented system of claim 8 , wherein the language learner is a non-native English speaker.

14. A non-transitory computer-readable medium encoded with instructions for commanding one or more data processors to execute steps of a method for conducting a simulated conversation with a language learner, the method comprising:

determining, by one or more data processors, a first dialog state of the simulated conversation;

transmitting, by the one or more data processors, first audio data corresponding to simulated speech based on the first dialog state;

receiving, by the one or more data processors, second audio data corresponding to a variable length utterance spoken in response to the simulated speech, metrics associated with the variable length utterance being stored in a variable length data structure whose length is based on a length of the variable length utterance;

generating, by the one or more data processors, a fixed dimension vector based on the variable length utterance, a length of the fixed dimension vector being fixed regardless of the length of the variable length utterance, the fixed dimension vector being generated by a factor analysis that samples a t-th frame of the variable length utterance from a distribution

x t ˜Σ j γ jt N ( m j +T j V,Σ j )

where x t represents the variable length utterance, m j represents a mean, Σ j represents a covariance of a j-th Gaussian component, γ jt represents a statistical alignment result of the frame t, T j represents a total variability, N is a constant, and V represents an utterance-specific standard normal distributed latent vector obtained by maximum a posterior (MAP) estimation;

predicting, by the one or more data processors, a semantic label for the variable-length utterance based on the fixed dimension vector using both of (i) a RNN decoder in which an output of the factor analysis is an input layer to the RNN decoder and an output layer of the RNN decoder is a softmax layer with N one-hot vectors that each represent a different one of N dialog states that does not use automated speech recognition (ASR); and (ii) a convolutional neural network using ASR output as an input;

determining, by one or more data processors, a second dialog state of the simulated conversation based on the semantic label; and

transmitting, by the one or more data processors, third audio data corresponding to simulated speech based on the second dialog state.