Dialog device with dialog support generated using a mixture of language models combined using a recurrent neural network
A dialog device comprises a natural language interfacing device (chat interface or a telephonic device), and a natural language output device (the chat interface, a display device, or a speech synthesizer outputting to the telephonic device). A computer stores natural language dialog conducted via the interfacing device and constructs a current utterance word-by-word. Each word is chosen by applying a plurality of language models to a context comprising concatenation of the stored dialog and the current utterance thus far. Each language model outputs a distribution over the words of a vocabulary. A recurrent neural network (RNN) is applied to the distributions to generate a mixture distribution. The next word is chosen using the mixture distribution. The output device outputs the current natural language utterance after it has been constructed by the computer.
1. A dialog device comprising:
a natural language interfacing device comprising a chat interface or a telephonic device;
a natural language output device comprising a display device; and
a computer programmed to store natural language dialog conducted via the natural language interfacing device and to construct a current natural language utterance word-by-word with each word of the current natural language utterance being chosen by operations including:
applying K independent language models F 1 , . . . , F k to a context x comprising a concatenation of the stored natural language dialog and the current natural language utterance up to but not including the word being chosen where K is a positive integer having a value of at least two,
normalizing each word distribution output by each of the applied K independent language models to generate corresponding normalized word distributions and applying a recurrent neural network (RNN) to the normalized word distributions output by the K applied independent language models to generate a mixture distribution y comprising a weighted sum of the normalized word distributions output by the applied K independent language models according to:
y
=
∑
k
=
1
K
p
(
k
|
x
)
F
k
(
x
)
where the p(k|x) terms are computed by a gating network which is a neural network whose output is a normalized vector; and
choosing the next word by sampling the mixture distribution;
wherein the computer is further programmed to repeat construction of a current natural language utterance a plurality of times to construct a plurality of current utterance candidates, and
the display device is configured to output a list of the current utterance candidates on the display device; and
wherein the K independent language models include: (i) a neural chat model for capturing natural language content, and (ii) at least one question-answer (QA) language model for capturing domain-specific content.
2. The dialog device of claim 1 wherein:
the natural language output device further comprises the speech synthesizer.
3. The dialog device of claim 1 wherein the RNN is a Long Short-Term Memory (LSTM) model or a Gated Recurrent Unit (GRU).
4. The dialog device of claim 1 wherein the normalizing comprises applying a softmax function to generate the normalized distributions.
5. The dialog device of claim 1 wherein K=2 and the normalizing comprises applying a sigmoid function to generate the normalized distributions.
6. The dialog device of claim 1 wherein the at least one QA language model is configured to provide a distribution of answers obtained from a knowledge base (KB) over questions directed to the KB.
7. The dialog device of claim 6 wherein the K independent language models further includes at least one language model that is not configured to provide answers obtained from a KB.
8. The dialog device of claim 6 wherein the computer is programmed to apply the at least one QA language model to the context by operations including:
computing a first probability distribution over a set of first KB question parameters for the context;
computing a second probability distribution over a set of second KB question parameters for the context;
computing a question probability distribution as a product of the first probability distribution and the second probability distribution;
identifying a question posed by the context using the question probability distribution; and
outputting, for the at least one QA language model, a distribution of answers obtained from the KB over the question probability distribution.
9. The dialog device of claim 1 , wherein each independent language model computes the respective word distribution over an associated vocabulary.
10. A dialog method comprising:
conducting a natural language dialog using a chat interface or a telephonic device;
while conducting the natural language dialog, constructing a current natural language utterance word-by-word using a computer programmed to choose each word of the current natural language utterance by operations including:
applying K language models F 1 , . . . , F K to a context x comprising a concatenation of the natural language dialog and the current natural language utterance up to but not including the word being chosen where K is a positive integer having a value of at least two,
applying a recurrent neural network (RNN) to word distributions output by the K applied language models to generate a mixture distribution y comprising a weighted sum of the word distributions output by the K applied language models wherein the RNN is applied according to:
y
=
∑
k
=
1
K
p
(
k
|
x
)
F
k
(
x
)
where the p(k|x) terms are computed by a gating network which is a neural network whose output is a normalized vector, and
choosing the next word using the mixture distribution y; and
outputting the constructed current natural language utterance via one of the chat interface, a display device, and a speech synthesizer outputting speech to the telephonic device;
wherein the K applied language models include: (i) a neural chat model for capturing natural language content, and (ii) at least one question-answer (QA) language model for capturing domain-specific content.
11. The dialog method of claim 10 wherein:
the choosing of the next word is by sampling the mixture distribution y,
the constructing is repeated a plurality of times to construct a plurality of current natural language utterance candidates, and
the outputting comprises outputting a list of the current natural language utterance candidates via a display device.
12. The dialog method of claim 10 wherein the RNN is a Long Short-Term Memory (LSTM) model.
13. The dialog method of claim 10 wherein the computer is programmed to choose each word of the current natural language utterance by operations further including:
normalizing the word distributions output by the K applied language models using a softmax or sigmoid function.
14. The dialog method of claim 10 wherein the at least one QA language model is configured to provide a distribution of answers obtained from a knowledge base (KB) over questions directed to the KB.
15. The dialog method of claim 14 wherein the K applied language models further includes at least one language model not configured to provide answers obtained from a KB.
16. The dialog method of claim 14 wherein the applying of the at least one QA language model to the context includes:
computing a first probability distribution over a set of first KB question parameters for the context;
computing a second probability distribution over a set of second KB question parameters for the context;
computing a question probability distribution as a product of the first probability distribution and the second probability distribution;
identifying a question posed by the context using the question probability distribution; and
outputting, for the at least one QA language model, a distribution of answers obtained from the KB over the question probability distribution.
17. A non-transitory storage medium storing instructions readable and executable by a computer to construct a current natural language utterance for continuing a natural language dialog by a method in which each word of the current natural language utterance is chosen by operations including:
applying K language models F 1 , . . . , F K to a context comprising a concatenation of the natural language dialog and the current natural language utterance up to but not including the word being chosen where K is a positive integer having a value of at least two, wherein the K language models include a neural chat model for capturing natural language content, and at least one question-answer (QA) language model for capturing domain-specific content;
normalizing word distributions output by the K language models to generate corresponding normalized word distributions;
applying a recurrent neural network (RNN) to the normalized distributions to generate a mixture distribution comprising a weighted sum of the normalized word distributions output by the K language models according to:
y
=
∑
k
=
1
K
p
(
k
❘
x
)
F
k
(
x
)
where the p(k|x) terms are computed by a gating network which is a neural network whose output is a normalized vector; and
choosing the next word using the mixture distribution;
wherein the RNN is a Gated Recurrent Unit (GRU).
18. The non-transitory storage medium of claim 17 wherein: the at least one QA language model is configured to provide a distribution of answers contained in a knowledge base (KB).