SYSTEMS AND METHODS FOR PROCESSING ELECTRONIC IMAGES USING DEEP FOUNDATION MODELS

1 . A computer-implemented method for processing digital medical images to infer metadata from those images, the method comprising:

receiving a plurality of digital medical images;

receiving a prompt, the prompt being a request for a specific type of metadata to be inferred from the plurality of digital medical images;

determining, using a trained foundation model, at least one feature descriptor from the plurality of digital medical images based on the prompt; and

providing for output the at least one feature descriptor for each of the plurality of digital medical images.

2 . The method of claim 1 , wherein the plurality of digital medical images include at least one of whole slide image (WSI), hematoxylin and eosin (H&E) stains, immunohistochemistry (IHC) slides, immunofluorescent slides, or CT scans.

3 . The computer-implemented method of claim 1 , wherein the metadata comprises any combination of supplemental medical images, structured diagnostic reports, unstructured free text reports, genomic data, proteomic data, treatment data, responses, or diagnoses.

4 . The computer-implemented method of claim 1 , further comprising:

receiving at least one query constraint, the query constraints including judgments or hypotheses from a clinician or expert; and

providing for output, from the trained foundation model, metadata estimations that are consistent with the at least one query constraint.

5 . The computer-implemented method of claim 1 , further comprising:

receiving free text; and

providing for output, from the trained foundation model, a structured synoptic diagnostic report based on the plurality of digital medical images and free text.

6 . The computer-implemented method of claim 1 , further comprising:

determining, using a content-based retrieval system, a collection of related digital medical images or cases based on the metadata associated with each of the digital medical images or cases.

7 . The computer-implemented method of claim 6 , further comprising:

receiving content-based constraints, the content-based constraints comprising instructions to include or exclude specific types of metadata, or attributes of that metadata, to a query for content retrieval.

8 . The computer-implemented method of claim 1 , further comprising:

determining, using a downstream task model, output targets that were not contained within metadata types based on the at least one feature descriptor for each of the plurality of digital medical images.

9 . The computer-implemented method of claim 8 , wherein the output targets include any combination of learning markers for drug response, building a model to replicate an existing biomarker, learning a novel biomarker from test data or other ground-truth indicators, or predicting additional disease states or diagnostics.

10 . A method of training a foundation model to process digital medical images to infer metadata from those images, the method comprising:

receiving a plurality of digital medical images;

generating a plurality of image tokens from the digital medical images, the image tokens being fixed-sized patches;

removing a subset of the plurality of image tokens from each of the digital medical images to generate a remaining plurality of image tokens from each of the digital medical images;

encoding, using an encoder, the remaining plurality of image tokens from each of the digital medical images;

adding a classification token to the encoded image tokens;

appending masked tokens with position encodings to each respective encoded image token; and

reconstructing, using a decoder, the image tokens, such that the image tokens align with original image pixel values.

11 . The method of claim 10 , wherein the encoder is a Vision Transformer (ViT) encoder.

12 . The method of claim 10 , wherein the classification token is a network-specific vector of numbers that summarizes an image tile representation.

13 . The method of claim 10 , wherein the decoder is a ViT decoder.

14 . The method of claim 13 , wherein the ViT decoder may be optimized using L2 image reconstruction loss applied on the masked tokens.

15 . The method of claim 10 , further comprising training the encoder and decoder to align the image tokens and the reconstructed masked tokens.

16 . A system for processing digital medical images to infer metadata from those images, the system comprising:

at least one memory storing instructions; and

at least one processor configured to execute the instructions to perform operations comprising:

receiving a plurality of digital medical images;

receiving a prompt, the prompt being a request for a specific type of metadata to be inferred from the plurality of digital medical images;

determining, using a trained foundation model, at least one feature descriptor from the plurality of digital medical images based on the prompt; and

providing for output the at least one feature descriptor for each of the plurality of digital medical images.

17 . The system of claim 16 , wherein the plurality of digital medical images include at least one of whole slide images (WSI), hematoxylin and eosin (H&E) stains, immunohistochemistry (IHC) slides, immunofluorescent slides, or CT scans.

18 . The system of claim 16 , wherein the metadata comprises any combination of supplemental medical images, structured diagnostic reports, unstructured free text reports, genomic data, proteomic data, treatment data, responses, or diagnoses.

19 . The system of claim 16 , the operations further comprising:

determining, using a downstream task model, output targets that were not contained within metadata types based on the at least one feature descriptor for each of the plurality of digital medical images.

20 . The system of claim 19 , wherein the output targets include any combination of learning markers for drug response, building a model to replicate an existing biomarker, learning a novel biomarker from test data or other ground-truth indicators, or predicting additional disease states or diagnostics.