Memory-based neural network for question answering
The present disclosure provides a memory-based neural network for question answering. Embodiments of the disclosure identify meta-evidence nodes in an embedding space, where the meta-evidence nodes represent salient features of a training set. Each element of the training set may include a questions appended to a ground truth answer. The training set may also include questions with wrong answers that are indicated as such. In some examples, a neural Turing machine (NTM) reads a dataset and summarizes the dataset into a few meta-evidence nodes. A subsequent question may be appended to multiple candidate answers to form an input phrase, which may also be embedded in the embedding space. Then, corresponding weights may be identified for each of the meta-evidence nodes. The embedded input phrase and the weighted meta-evidence nodes may be used to identify the most appropriate answer.
1. A method comprising:
embedding an input phrase comprising a natural language query and a candidate answer from a set of candidate answers into an embedding space to obtain an embedded input phrase;
identifying a plurality of meta-evidence nodes in the embedding space based on the embedded input phrase, wherein each of the plurality of meta-evidence nodes represents a cluster of points in the embedding space, and wherein each point in the cluster of points is an embedding of a training example that includes a training query and a ground truth answer;
generating a support vector for the input phrase, wherein the support vector comprises a weighted sum of the plurality of meta-evidence nodes in the embedding space; and
selecting an answer to the natural language query from the set of candidate answers based on the support vector.
2. The method of claim 1 , further comprising:
receiving the natural language query from a user; and
concatenating the natural language query with the candidate answer to form the input phrase.
3. The method of claim 1 , wherein:
the natural language query is received from a user during an automated dialogue with the user in a chat application.
4. The method of claim 1 , further comprising:
embedding the input phrase into the embedding space using an embedding model.
5. The method of claim 1 , further comprising:
generating a response to the natural language query based on the selected answer.
6. The method of claim 1 , further comprising:
generating weights of the support vector for each of the plurality of meta-evidence nodes using a neural network.
7. The method of claim 1 , further comprising:
summarizing training data using a neural Turing machine to produce the plurality of meta-evidence nodes.
8. The method of claim 1 , further comprising:
combining an input vector representing of the input phrase in the embedding space with a corresponding support vector to produce a combined vector;
adding a bias vector to the combined vector to produce a biased combined vector; and
applying a softmax function to the biased combined vector to produce a probability value, wherein the answer is selected based on the probability value.
9. The method of claim 1 , further comprising:
updating the plurality of meta-evidence nodes based on the selected answer.
10. The method of claim 1 , further comprising:
determining that the selected answer is incorrect;
selecting one or more meta-evidence nodes that contributed to selecting the answer; and
identifying a data point in a set of training data that contributed to selecting the answer.
11. The method of claim 10 , further comprising:
removing the data point from the set of training data; and
updating the plurality of meta-evidence nodes after removing the data point.
12. The method of claim 1 , wherein:
a number of the plurality of meta-evidence nodes is greater than a number of the set of candidate answers.
13. An apparatus comprising:
an embedding network configured to embed an input phrase comprising a natural language query and a candidate answer from a set of candidate answers into an embedding space to obtain an embedded input phrase;
a memory controller configured to identify a plurality of meta-evidence nodes in the embedding space based on the embedded input phrase and to generate a support vector for the input phrase, wherein each of the plurality of meta-evidence nodes represents a cluster of points in the embedding space, wherein each point in the cluster of points is an embedding of a training example that includes a training query and a ground truth answer, and wherein the support vector comprises a weighted sum of the plurality of meta-evidence nodes in the embedding space; and
an answer selection module configured to select an answer to the natural language query from the set of candidate answers based on the support vector for the input phrase.
14. The apparatus of claim 13 , further comprising:
a neural Turing machine configured to select the plurality of meta-evidence nodes based on a set of training data.
15. The apparatus of claim 13 , wherein:
a neural Turing machine is trained together with the memory controller.
16. The apparatus of claim 13 , further comprising:
a dialog generator configured to generate a response to the natural language query based on the selected answer.
17. A method of training a neural network for question answering, the method comprising:
receiving a training set comprising a plurality of training phrases, wherein each of the plurality of training phrases comprises a natural language query combined with a ground truth answer;
generating an input phrase for at least one of the plurality of training phrases by combining the natural language query with a plurality of candidate answers;
embedding the input phrase into an embedding space to obtain an embedded input phrase;
identifying a plurality of meta-evidence nodes in the embedding space based on the embedded input phrase, wherein each of the plurality of meta-evidence nodes represents a cluster of points in the embedding space, and wherein each point in the cluster of points is an embedding of a training phrase of the plurality of training phrases;
generating a support vector for the input phrase using a memory controller, wherein the support vector comprises a weighted sum of the plurality of meta-evidence nodes in the embedding space;
selecting a predicted answer to the natural language query from the plurality of candidate answers based on the support vector;
computing a loss function by comparing the predicted answer to the ground truth answer; and
updating parameters of a neural Turing machine and the memory controller based on the loss function.
18. The method of claim 17 , further comprising:
embedding the plurality of training phrases and the input phrase into the embedding space using a pre-trained embedding network.
19. The method of claim 17 , further comprising:
iteratively updating the parameters of the neural Turing machine and the memory controller to train the neural Turing machine and the memory controller.