Artificial intelligence (AI) to provide decision support insights including while a doctor is engaged in conversation with a patient
In some aspects, orchestration logic is configured to receive, from a multi-modal interface, an upstream conversation between a doctor and a patient, provide, to at least one large language model (LLM), the upstream conversation and the patient's medical history, and cause the LLM to generate raw decision support insights. The LLM generates the raw decision support insights based at least in part on the upstream conversation and the patient's medical history. Real-time decision support logic is configured to: transform the raw decision support insights into prioritized, conversation-responsive decision support insights. The prioritized, conversation-responsive decision support insights are prioritized based on medical urgency. Presentation of the prioritized, conversation-responsive decision support insights to the doctor is responsive to downstream conversation between the patient and the doctor. The prioritized, conversation-responsive decision support insights are delivered to the multi-modal interface for presentation to the doctor.
1. A system, comprising:
orchestration logic configured to:
continually receive, from a multi-modal interface, audio data comprising at least a portion of an upstream conversation between a doctor and a patient;
continually provide, to at least one large language model, the audio data and the patient's medical history, wherein the at least one large language is trained using training data that includes multiple audio conversations between doctors and patients to create at least one trained large language model; and
continually cause the at least one trained large language model to generate raw decision support insights based at least in part on audio data and the patient's medical history; and
real-time decision support logic, in communication with the orchestration logic, and configured to:
transform the raw decision support insights into conversation-responsive decision support insights;
prioritize the conversation-responsive decision support insights based on medical urgency to create prioritized, conversation-responsive decision support insights;
deliver a text-based presentation of the prioritized, conversation-responsive decision support insights to the multi-modal interface for presentation to the doctor via a continually updated graphical user interface;
determine, based on the continually received audio data, that a condition has been met by a particular insight of the conversation-responsive decision support insights;
based on determining that the condition has been met by a particular insight of the conversation-responsive decision support insights, modify a graphical characteristic of the text-based presentation of the particular insight being presented in the graphical user interface; and
re-training the at least one trained large language using additional training data comprising the upstream conversation between the doctor and the patient that includes the continually received audio data.
2. The system of claim 1 , wherein the prioritized, conversation-responsive decision support insights are presented to the doctor in conjunction with criticality scores, wherein the criticality scores are determined based on the medical urgency.
3. The system of claim 1 , wherein the prioritized, conversation-responsive decision support insights include a subject decision support insight with a high criticality score, wherein the subject decision support insight is persisted despite the doctor not expressly accounting for the subject decision support insight.
4. The system of claim 1 , wherein the prioritized, conversation-responsive decision support insights include a particular decision support insight, wherein the downstream conversation establishes that the doctor already accounted for the particular decision support insight.
5. The system of claim 4 , wherein the presentation of the prioritized, conversation-responsive decision support insights to the doctor is updated to specify that the doctor already accounted for the particular decision support insight.
6. The system of claim 1 , wherein the prioritized, conversation-responsive decision support insights include suggestions for the doctor to make to the patient.
7. The system of claim 1 , wherein the prioritized, conversation-responsive decision support insights include follow-up actions for the doctor to make.
8. The system of claim 7 , wherein the real-time decision support logic, in communication with the orchestration logic is further configured to execute one or more of the follow-up actions in response to confirmation from the doctor.
9. The system of claim 7 , wherein the at least one trained large language model generates the raw decision support insights based at least in part on medical knowledge.
10. The system of claim 9 , wherein the medical knowledge includes historical doctor-patient conversations of a particular doctor.
11. The system of claim 1 , wherein the patient's medical history includes electronic medical records.
12. The system of claim 1 , wherein the electronic medical records span multiple medical providers.
13. The system of claim 1 , wherein the upstream conversation is supplemented with upstream biometrics of the patients.
14. The system of claim 1 , wherein the downstream conversation is supplemented with downstream biometrics of the patients.
15. A computer-implemented method, including:
continually receiving, from a multi-modal interface, audio data comprising at least a portion of an upstream conversation between a doctor and a patient;
continually providing, to at least one large language model, the audio data and the patient's medical history, wherein the at least one large language is trained using training data that includes multiple audio conversations between doctors and patients to create at least one trained large language model;
continually causing the at least one trained large language model to generate raw decision support insights based at least in part on the audio data and the patient's medical history;
transforming the raw decision support insights into prioritized, conversation-responsive decision support insights;
prioritizing the conversation-responsive decision support insights based on medical urgency to create prioritized conversation-responsive decision support insights;
delivering a text-based presentation of the prioritized, conversation-responsive decision support insights to the multi-modal interface for presentation to the doctor via a continually updated graphical user interface;
determining, based on the continually received audio data, that a condition has been met by a particular insight of the conversation-responsive decision support insights;
based on determining that the condition has been met by a particular insight of the conversation-responsive decision support insights, modifying a graphical characteristic of the text-based presentation of the particular insight being presented in the graphical user interface; and
re-training the at least one trained large language using additional training data comprising the upstream conversation between the doctor and the patient that includes the continually received audio data.
16. The computer-implemented method of claim 15 , wherein the prioritized, conversation-responsive decision support insights are presented to the doctor in conjunction with criticality scores, wherein the criticality scores are determined based on the medical urgency.
17. The computer-implemented method of claim 15 , wherein the prioritized, conversation-responsive decision support insights include a subject decision support insight with a high criticality score, wherein the subject decision support insight is persisted despite the doctor not expressly accounting for the subject decision support insight.
18. The computer-implemented method of claim 15 , wherein the prioritized, conversation-responsive decision support insights include a particular decision support insight, wherein the downstream conversation establishes that the doctor already accounted for the particular decision support insight.
19. The computer-implemented method of claim 18 , wherein the presentation of the prioritized, conversation-responsive decision support insights to the doctor is updated to specify that the doctor already accounted for the particular decision support insight.
20. A non-transitory computer readable storage medium to store computer program instructions that are executable by a processor to perform operations comprising:
continually receiving, from a multi-modal interface, audio data comprising at least a portion of an upstream conversation between a doctor and a patient;
continually providing, to at least one large language model, the audio data and the patient's medical history, wherein the at least one large language is trained using training data that includes multiple audio conversations between doctors and patients to create at least one trained large language model;
continually causing the at least one trained large language model to generate raw decision support insights based at least in part on the audio data and the patient's medical history;
transforming the raw decision support insights into conversation-responsive decision support insights;
prioritizing a text-based presentation of the conversation-responsive decision support insights based on medical urgency to create prioritized, conversation-responsive decision support insights;
delivering the prioritized, conversation-responsive decision support insights to the multi-modal interface for presentation to the doctor via a continually updated graphical user interface;
determining, based on the continually received audio data, that a condition has been met by a particular insight of the conversation-responsive decision support insights;
based on determining that the condition has been met by a particular insight of the conversation-responsive decision support insights, modifying a graphical characteristic of the text-based presentation of the particular insight being presented in the graphical user interface; and
re-training the at least one trained large language using additional training data comprising the upstream conversation between the doctor and the patient that includes the continually received audio data.