Grounded dialogue generation with cross-encoding re-ranker, grounding span prediction, and passage dropout
An information-seeking dialogue system can be trained using a pipeline process having stages, or components, of passage retrieval (selecting passages relevant to a query from a corpus or knowledge base), re-ranking, and generating a response to the query based on one or more of the re-ranked passages. Each stage, or component, of the pipeline can be individually optimized based on ground truth data.
1. A computer-implemented method for responding to user queries, the method comprising:
obtaining a training set, the training set including a plurality of passages, a plurality of queries, and a plurality of responses, the training set further including association data associating particular queries of the plurality of queries with particular passages of the plurality of passages and particular responses of the plurality of responses;
performing a training process using the training set, wherein the training process includes:
training a retriever model to identify similarities between a query and a passage;
training a re-ranking model to rank a plurality of passages based on a distance metric defining a distance between a passage and a query, wherein the re-ranking model uses an ensemble of transformer language models that each output a distance function between a passage and a query, and wherein the re-ranking model ranks the plurality of passages based on a weighted sum of the distance functions of the transformer language models; and
training a generator model to generate a response to a query based on a passage; and
performing a testing process, wherein the testing process includes:
receiving a testing query;
using the retriever model to retrieve a plurality of passages responsive to the testing query;
using the re-ranking model to rank the retrieved passages based on the testing query; and
using the generator model to generate a response based on a highest-ranking one of the retrieved passages.
2. The method of claim 1 wherein the retriever model is a Dense Passage Retriever (DPR) model having a first encoder for a query and a second encoder for a passage.
3. The method of claim 2 wherein the retriever model defines similarity based on a dot product of an encoding of the query using the first encoder and an encoding of the passage using the second encoder.
4. The method of claim 1 wherein the ensemble of transformer language models includes at least three transformer language models.
5. The method of claim 1 wherein training the re-ranking model includes training one or more weights for the weighted sum.
6. The method of claim 1 wherein training the generator model includes training grounding span prediction as an auxiliary task.
7. The method of claim 1 wherein training the generator model includes applying passage dropout.
8. The method of claim 1 wherein each query in the training set includes a query history and wherein the testing query also includes a query history.
9. A system comprising:
a memory; and
a processor coupled to the memory and configured to:
obtain a training set, the training set including a plurality of passages, a plurality of queries, and a plurality of responses, the training set further including association data associating particular queries of the plurality of queries with particular passages of the plurality of passages and particular responses of the plurality of responses;
perform a training process using the training set, wherein performing the training process includes:
training a retriever model to identify similarities between a query and a passage;
training a re-ranking model to rank a plurality of passages based on a distance metric defining a distance between a passage and a query, wherein the re-ranking model uses an ensemble of transformer language models that each output a distance function between a passage and a query, and wherein the re-ranking model ranks the plurality of passages based on a weighted sum of the distance functions of the transformer language models; and
training a generator model to generate a response to a query based on a passage; and
perform a testing process, wherein the testing process includes:
receiving a testing query;
using the retriever model to retrieve a plurality of passages responsive to the testing query;
using the re-ranking model to rank the retrieved passages based on the testing query; and
using the generator model to generate a response based on a highest-ranking one of the retrieved passages.
10. The system of claim 9 wherein the retriever model is a Dense Passage Retriever (DPR) model having a first encoder for a query and a second encoder for a passage and wherein the retriever model defines similarity based on a dot product of an encoding of the query using the first encoder and an encoding of the passage using the second encoder.
11. The system of claim 10 wherein:
training the re-ranking model includes training one or more weights for the weighted sum.
12. The system of claim 9 wherein training the generator model includes training grounding span prediction as an auxiliary task.
13. The system of claim 9 wherein training the generator model includes applying passage dropout.
14. A non-transitory computer readable storage medium having stored therein program code instructions that, when executed by a processor of a computer system, cause the processor to perform a method comprising:
obtaining a training set, the training set including a plurality of passages, a plurality of queries, and a plurality of responses, the training set further including association data associating particular queries of the plurality of queries with particular passages of the plurality of passages and particular responses of the plurality of responses;
performing a training process using the training set, wherein the training process includes:
training a retriever model to identify similarities between a query and a passage;
training a re-ranking model to rank a plurality of passages based on a distance metric defining a distance between a passage and a query, wherein the re-ranking model uses an ensemble of transformer language models that each output a distance function between a passage and a query, and wherein the re-ranking model ranks the plurality of passages based on a weighted sum of the distance functions of the transformer language models; and
training a generator model to generate a response to a query based on a passage; and
performing a testing process, wherein the testing process includes:
receiving a testing query;
using the retriever model to retrieve a plurality of passages responsive to the testing query;
using the re-ranking model to rank the retrieved passages based on the testing query; and
using the generator model to generate a response based on a highest-ranking one of the retrieved passages.
15. The computer readable storage medium of claim 14 wherein the retriever model is a Dense Passage Retriever (DPR) model having a first encoder for a query and a second encoder for a passage.
16. The computer readable storage medium of claim 15 wherein the retriever model defines similarity based on a dot product of an encoding of the query using the first encoder and an encoding of the passage using the second encoder.
17. The computer readable storage medium of claim 14 wherein the ensemble of transformer language models includes at least three transformer language models.
18. The computer readable storage medium of claim 14 wherein training the re-ranking model includes training one or more weights for the weighted sum.
19. The computer readable storage medium of claim 14 wherein training the generator model includes training grounding span prediction as an auxiliary task.
20. The computer readable storage medium of claim 14 wherein training the generator model includes applying passage dropout.
21. The computer readable storage medium of claim 14 wherein each query in the training set includes a query history and wherein the testing query also includes a query history.