Drawing conclusions from free form texts with deep reinforcement learning
Techniques are described herein for drawing conclusions using free form texts and external resources. In various embodiments, free form input data ( 202 ) may be segmented ( 504 ) into a plurality of input data segments. A first input data segment may be compared ( 510 ) with an external resource ( 304 ) to identify a first candidate conclusion. A reinforcement learning trained agent ( 310 ) may be applied ( 512 ) to make a first determination of whether to accept or reject the first candidate conclusion. Similar actions may be performed with a second input data segment to make a second determination of whether to accept or reject a second candidate conclusion. A final conclusion may be presented ( 522 ) based on the first and second determinations of the reinforcement learning trained agent with respect to at least the first candidate conclusion and the second candidate conclusion.
1. A method implemented by one or more processors and comprising:
segmenting a free form clinical notes into a plurality of segments;
for each segment of the plurality of segments:
identifying a candidate diagnosis, comprising identifying a document in an evidence pool that most closely matches said segment, the evidence pool comprising clinical reference documents and associated diagnoses, and selecting a diagnosis associated with the identified document as the candidate diagnosis;
applying a reinforcement learning trained agent to a state vector encoding said segment and said candidate diagnosis to make a determination of whether to accept or reject the candidate diagnosis, the reinforcement learning trained agent being trained to accept or reject candidate diagnoses based on a reward function measuring an accuracy of candidate diagnoses, and wherein the reinforcement learning trained agent is a deep Q-network (“DQN”) agent trained via Q-learning to accept or reject candidate diagnoses based on a reward function using a training set of clinical narratives and associated ground truths; and
presenting the one or more accepted candidate diagnoses;
wherein the reward function includes an instant reward and a global reward, the instant reward based on a match of the candidate diagnosis with a gold standard diagnosis, and the global reward comprising a number of correct diagnoses accepted so far minus a number of incorrect diagnoses accepted so far.
2. The method of claim 1 , wherein identifying said document comprises formulating a query for the evidence pool based on said segment and searching the evidence pool using said query to identify said document.
3. The method of claim 1 , wherein the reinforcement learning trained agent further determines whether to accept or reject said segment.
4. The method of claim 1 , wherein the state vector comprises a vector representation of words of said segment produced using a Long Short Term Memory Network and mean pooling.
5. The method of claim 3 , wherein the state vector comprises one or more of a similarity between said segment and segments accepted so far, a similarity between said segment and a context of the free form clinical note, a similarity between said segment and a context of a clinical reference document associated with the candidate diagnosis, a similarity between segments accepted so far and the context of the clinical reference document associated with the candidate diagnosis, and a similarity between the candidate diagnosis and candidate diagnoses accepted so far.
6. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
segmenting a free form clinical note into a plurality of segments;
for each segment of the plurality of segments:
identifying a candidate diagnosis, comprising identifying a document in an evidence pool that most closely matches said segment, the evidence pool comprising clinical reference documents and associated diagnoses, and selecting a diagnosis associated with the identified document as the candidate diagnosis;
applying a reinforcement learning trained agent to a state vector encoding said segment and said candidate diagnosis to make a determination of whether to accept or reject the candidate conclusion diagnosis, the reinforcement learning trained agent being trained to accept or reject candidate diagnoses based on a reward function measuring an accuracy of candidate diagnoses, and wherein the reinforcement learning trained agent is a deep Q-network (“DQN”) agent trained via Q-learning to accept or reject candidate diagnoses based on a reward function using a training set of clinical narratives and associated ground truths; and
presenting the one or more accepted candidate diagnoses;
wherein the reward function includes an instant reward and a global reward, the instant reward based on a match of the candidate diagnosis with a gold standard diagnosis, and the global reward comprising a number of correct diagnoses accepted so far minus a number of incorrect diagnoses accepted so far.
7. A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
segmenting a free form clinical note into a plurality of segments;
for each segment of the plurality of segments:
identifying a candidate diagnosis, comprising identifying a document in an evidence pool that most closely matches said segment, the evidence pool comprising clinical reference documents and associated diagnoses, and selecting a diagnosis associated with the identified document as the candidate diagnosis;
applying a reinforcement learning trained agent to a state vector encoding said segment and said candidate diagnosis to make a determination of whether to accept or reject the candidate diagnosis, the reinforcement learning trained agent being trained to accept or reject candidate diagnoses based on a reward function measuring an accuracy of candidate diagnoses, and wherein the reinforcement learning trained agent is a deep Q-network (“DQN”) agent trained via Q-learning to accept or reject candidate diagnoses based on a reward function using a training set of clinical narratives and associated ground truths; and
presenting the one or more accepted candidate diagnoses;
wherein the reward function includes an instant reward and a global reward, the instant reward based on a match of the candidate diagnosis with a gold standard diagnosis, and the global reward comprising a number of correct diagnoses accepted so far minus a number of incorrect diagnoses accepted so far.