AI-driven natural language co-pilot for pathology
Systems and methods are provided for providing natural language decision support for pathology. A lower-dimensionality representation of each of a set of received pathology image is generated and a first set of tokens is generated from the representations of the set of pathology images by projecting the lower-dimensionality representations of the received pathology images to a same dimension as an embedding space of a large language model for text tokens or through multimodal blocks added to the large language model such as cross-attention. The large language model is trained on an instruction dataset complied from a plurality of pathology-related sources. A second set of tokens associated with a natural language prompt is received at the large language model. A response is determined from the first set of tokens and the second set of tokens at the large language model.
1 . A system for providing natural language decision support for pathology, the system comprising:
a processor; and
a non-transitory computer readable medium storing instructions executable by the processor, the machine-executable instructions comprising:
a vision encoder that receives a pathology image and generates a representation of the pathology image;
a multimodal projector that generates a set of first set of tokens from the representation of the pathology image;
a user interface that receives a prompt from a user as natural language text;
a tokenizer that generates a second set of tokens from the prompt; and
a large language model that is trained on an instruction dataset complied from a plurality of pathology-related sources, each training sample within the instruction dataset comprising a set of pathology images and text describing or answering specific queries pertaining to the images, the large language model receiving the second set of tokens and determining a response from the first set of tokens and the second set of tokens, the instruction dataset being selected as to exclude any text determined to be a generic description of an image;
wherein the user interface displays the response to the user at an associated display.
2 . The system of claim 1 , wherein the instruction dataset is selected as to exclude text associated with pathology images of animals, wherein text associated with pathology images of animals is identified via a regular expression pattern matching process using a set of key phrases.
3 . The system of claim 1 , wherein the response is a first response of a plurality of responses and the user interface allows the user to select a response from the plurality of responses, the image and the selected response being added to the instruction data set as a training sample.
4 . The system of claim 1 , wherein the response is a first response of a plurality of responses and the user interface allows the user to rate a response as one of helpful and unhelpful, the image and the selected response being added to the instruction data set as a training sample when the response is rated as helpful.
5 . The system of claim 1 , wherein the user interface allows the user to provide a task prompt and example data that are stored for training-free adaption to the user's task.
6 . A system for providing natural language decision support for pathology, the system comprising:
a processor; and
a non-transitory computer readable medium storing instructions executable by the processor, the machine-executable instructions comprising:
a vision encoder that receives a pathology image and generates a representation of the pathology image;
a multimodal projector that generates a set of first set of tokens from the representation of the pathology image;
a user interface that receives a prompt from a user as natural language text;
a tokenizer that generates a second set of tokens from the prompt; and
a large language model that is trained on an instruction dataset complied from a plurality of pathology-related sources, each training sample within the instruction dataset comprising a set of pathology images and text describing or answering specific queries pertaining to the images, the large language model receiving the second set of tokens and determining a response from the first set of tokens and the second set of tokens, the instruction dataset being selected as to exclude text associated with pathology images of animals, wherein text associated with pathology images of animals are identified via a regular expression pattern matching process using a set of key phrases;
wherein the user interface displays the response to the user at an associated display.
7 . A system for providing natural language decision support for pathology, the system comprising:
a processor; and
a non-transitory computer readable medium storing instructions executable by the processor, the machine-executable instructions comprising:
a vision encoder that receives a pathology image and generates a representation of the pathology image;
a multimodal projector that generates a first set of tokens from the representation of the pathology image; and
a large language model that is trained on an instruction dataset complied from a plurality of pathology-related sources, a given training sample within the instruction dataset comprising a set of pathology images and text describing or answering specific queries pertaining to the images, the large language model receiving a second set of tokens associated with a prompt and determining a response from the first set of tokens and the second set of tokens, the instruction dataset being selected as to exclude experimental studies, wherein experimental studies are identified via a regular expression pattern matching process using a set of key phrases.
8 . The system of claim 7 , wherein the response is a first response of a plurality of responses and the user interface allows the user to select a response from the plurality of responses, the image and the selected response being added to the instruction data set as a training sample.
9 . The system of claim 7 , wherein the response is a first response of a plurality of responses and the user interface allows the user to rate a response as one of helpful and unhelpful, the image and the selected response being added to the instruction data set as a training sample when the response is rated as helpful.
10 . The system of claim 7 , wherein one of the second set of tokens and the prompt are stored on the non-transitory computer readable medium, and the second set of tokens is provided to the large language model without input by a user.
11 . The system of claim 7 , further comprising an image segmenter that selects a region of interest within an image, the pathology images being a plurality of tiles generated from the region of interest and the first set of tokens representing the content and position of the pathology images.
12 . The system of claim 7 , wherein the plurality of pathology-related sources includes at least two of captions of medical images, educational articles, pathology case reports, and extracted regions from whole slide imaging.
13 . The system of claim 1 , wherein one of the second set of tokens and the prompt are stored on the non-transitory computer readable medium, and the second set of tokens is provided to the large language model without input by a user.
14 . The system of claim 1 , further comprising an image segmenter that selects a region of interest within an image, the pathology images being a plurality of tiles generated from the region of interest and the first set of tokens representing the content and position of the pathology images.
15 . The system of claim 1 , wherein the plurality of pathology-related sources includes at least two of captions of medical images, educational articles, pathology case reports, and extracted regions from whole slide imaging.
16 . The system of claim 6 , wherein the response is a first response of a plurality of responses and the user interface allows the user to select a response from the plurality of responses, the image and the selected response being added to the instruction data set as a training sample.
17 . The system of claim 6 , wherein the response is a first response of a plurality of responses and the user interface allows the user to rate a response as one of helpful and unhelpful, the image and the selected response being added to the instruction data set as a training sample when the response is rated as helpful.
18 . The system of claim 6 , wherein one of the second set of tokens and the prompt are stored on the non-transitory computer readable medium, and the second set of tokens is provided to the large language model without input by a user.
19 . The system of claim 6 , further comprising an image segmenter that selects a region of interest within an image, the pathology images being a plurality of tiles generated from the region of interest and the first set of tokens representing the content and position of the pathology images.
20 . The system of claim 6 , wherein the plurality of pathology-related sources includes at least two of captions of medical images, educational articles, pathology case reports, and extracted regions from whole slide imaging.