Medical question answering system
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating answers to medical questions using neural networks and other components. In one aspect, a method includes: receiving, from a user and by way of a user interface presented to the user on a display of a user device, a query for medical information; generating multiple responses to the query from the user by automatically retrieving and parsing data from a corpus of documents; and presenting, by way of the user interface and on the display of the user device: a first user interface element that presents a first response generated based only on clinical practice guideline documents, and a second user interface element that presents a second response generated based at least in part on documents that are not clinical practice guideline documents.
1 . A method performed by one or more computers for reducing risk of hallucination and reducing latency when generating responses to medical questions using a generative neural network, the method comprising:
receiving, from a user and by way of a user interface presented to the user on a display of a user device, a medical question;
generating multiple answers to the medical question from the user by automatically retrieving and parsing data from a corpus of documents, comprising:
generating, by using an embedding model, a medical question embedding for the medical question;
performing an automated search to identify, from the corpus of documents, document snippets that include information that is relevant to the medical question, wherein performing the automated search comprises:
prior to receiving the medical question:
generating, by using the embedding model, a document snippet embedding for each document snippet of at least a subset of the document snippets, wherein the embedding model has been trained based on generating more training medical question embeddings than training document snippet embeddings during training, and
storing the document snippet embedding for each document snippet of at least the subset of the document snippets, and
after receiving the medical question, performing an embedding-based search that uses the medical question embedding and the document snippet embedding for each document snippet of at least the subset of the document snippets;
partitioning the identified document snippets into first document snippets that are obtained from one or more clinical practice guideline documents and second document snippets that are obtained from one or more other documents that are not clinical practice guideline documents;
generating a first prompt by including (i) the medical question and (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents, excluding the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents;
making a first call to a generative neural network to cause the generative neural network to generate a first answer to the medical question based on processing the first prompt that includes (i) the medical question and (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents but excludes the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents, such that the generative neural network generates the first answer without being conditioned on the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents; and
generating a second prompt by including (i) the medical question, (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents, and (iii) the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents;
making a second call to the generative neural network to cause the generative neural network to generate a second answer to the medical question based on processing a second prompt that includes (i) the medical question, (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents, and (iii) the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents, such that the generative neural network generates the second answer being conditioned at least on the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents; and
presenting, by way of the user interface and on the display of the user device:
a first user interface element that presents the first answer generated by the generative neural network in response to the first call, wherein the first user interface element visually highlights that the first answer is derived only from clinical practice guideline documents and identifies the one or more clinical practice guideline document that were processed by the generative neural network to generate the first answer; and
a second user interface element that presents the second answer generated by the generative neural network in response to the second call.
2 . The method of claim 1 , wherein the first user interface element is presented on top of and temporally before the second user interface element within the user interface.
3 . The method of claim 1 , wherein the first user interface element comprises a walled garden environment presented within the user interface, and wherein the first response generated based only on the clinical practice guideline documents is presented within the walled garden environment.
4 . The method of claim 1 , wherein the first user interface element comprises a header that indicates the first response is generated based only on clinical practice guideline documents.
5 . The method of claim 1 , wherein making the second call to the generative neural network comprises making the second call after the generative neural network has generated the first response in response to the first call.
6 . The method of claim 1 , wherein the second prompt comprises the first response generated by the generative neural network in response to the first call.
7 . A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for reducing risk of hallucination and reducing latency when generating responses to medical questions using a generative neural network, the operations comprising:
receiving, from a user and by way of a user interface presented to the user on a display of a user device, a medical question;
generating multiple answers to the medical question from the user by automatically retrieving and parsing data from a corpus of documents, comprising:
generating, by using an embedding model, a medical question embedding for the medical question;
performing an automated search to identify, from the corpus of documents, document snippets that include information that is relevant to the medical question, wherein performing the automated search comprises:
prior to receiving the medical question:
generating, by using the embedding model, a document snippet embedding for each document snippet of at least a subset of the document snippets, wherein the embedding model has been trained based on generating more training medical question embeddings than training document snippet embeddings during training, and
storing the document snippet embedding for each document snippet of at least the subset of the document snippets, and
after receiving the medical question, performing an embedding-based search that uses the medical question embedding and the document snippet embedding for each document snippet of at least the subset of the document snippets;
partitioning the identified document snippets into first document snippets that are obtained from one or more clinical practice guideline documents and second document snippets that are obtained from one or more other documents that are not clinical practice guideline documents;
generating a first prompt by including (i) the medical question and (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents, excluding the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents;
making a first call to a generative neural network to cause the generative neural network to generate a first answer to the medical question based on processing the first prompt that includes (i) the medical question and (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents but excludes the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents, such that the generative neural network generates the first answer without being conditioned on the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents; and
generating a second prompt by including (i) the medical question, (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents, and (iii) the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents;
making a second call to the generative neural network to cause the generative neural network to generate a second answer to the medical question based on processing a second prompt that includes (i) the medical question, (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents, and (iii) the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents, such that the generative neural network generates the second answer being conditioned at least on the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents; and
presenting, by way of the user interface and on the display of the user device:
a first user interface element that presents the first answer generated by the generative neural network in response to the first call, wherein the first user interface element visually highlights that the first answer is derived only from clinical practice guideline documents and identifies the one or more clinical practice guideline document that were processed by the generative neural network to generate the first answer; and
a second user interface element that presents the second answer generated by the generative neural network in response to the second call.
8 . The system of claim 7 , wherein the first user interface element is presented on top of and temporally before the second user interface element within the user interface.
9 . The system of claim 7 , wherein the first user interface element comprises a walled garden environment presented within the user interface, and wherein the first response generated based only on the clinical practice guideline documents is presented within the walled garden environment.
10 . The system of claim 7 , wherein the first user interface element comprises a header that indicates the first response is generated based only on clinical practice guideline documents.
11 . The system of claim 7 , wherein making the second call to the generative neural network comprises making the second call after the generative neural network has generated the first response in response to the first call.
12 . The system of claim 7 , wherein the second prompt comprises the first response generated by the generative neural network in response to the first call.
13 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for reducing risk of hallucination and reducing latency when generating responses to medical questions using a generative neural network, the operations comprising:
receiving, from a user and by way of a user interface presented to the user on a display of a user device, a medical question;
generating multiple answers to the medical question from the user by automatically retrieving and parsing data from a corpus of documents, comprising:
generating, by using an embedding model, a medical question embedding for the medical question;
performing an automated search to identify, from the corpus of documents, document snippets that include information that is relevant to the medical question, wherein performing the automated search comprises:
prior to receiving the medical question:
generating, by using the embedding model, a document snippet embedding for each document snippet of at least a subset of the document snippets, wherein the embedding model has been trained based on generating more training medical question embeddings than training document snippet embeddings during training, and
storing the document snippet embedding for each document snippet of at least the subset of the document snippets, and
after receiving the medical question, performing an embedding-based search that uses the medical question embedding and the document snippet embedding for each document snippet of at least the subset of the document snippets;
partitioning the identified document snippets into first document snippets that are obtained from one or more clinical practice guideline documents and second document snippets that are obtained from one or more other documents that are not clinical practice guideline documents;
generating a first prompt by including (i) the medical question and (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents, excluding the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents;
making a first call to a generative neural network to cause the generative neural network to generate a first answer to the medical question based on processing the first prompt that includes (i) the medical question and (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents but excludes the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents, such that the generative neural network generates the first answer without being conditioned on the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents; and
generating a second prompt by including (i) the medical question, (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents, and (iii) the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents;
making a second call to the generative neural network to cause the generative neural network to generate a second answer to the medical question based on processing a second prompt that includes (i) the medical question, (ii) the first document snippets that are obtained from the one or more clinical practice guideline documents, and (iii) the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents, such that the generative neural network generates the second answer being conditioned at least on the second document snippets that are obtained from the one or more other documents that are not clinical practice guideline documents; and
presenting, by way of the user interface and on the display of the user device:
a first user interface element that presents the first answer generated by the generative neural network in response to the first call, wherein the first user interface element visually highlights that the first answer is derived only from clinical practice guideline documents and identifies the one or more clinical practice guideline document that were processed by the generative neural network to generate the first answer; and
a second user interface element that presents the second answer generated by the generative neural network in response to the second call.
14 . The computer storage media of claim 13 , wherein the first user interface element is presented on top of and temporally before the second user interface element within the user interface.
15 . The computer storage media of claim 13 , wherein the first user interface element comprises a walled garden environment presented within the user interface, and wherein the first response generated based only on the clinical practice guideline documents is presented within the walled garden environment.
16 . The computer storage media of claim 13 , wherein making the second call to the generative neural network comprises making the second call after the generative neural network has generated the first response in response to the first call.
17 . The computer storage media of claim 13 , wherein the second prompt comprises the first response generated by the generative neural network in response to the first call.