Machine learning based medical data checker
A method of verifying multi-modal medical data is proposed. The method comprises: accessing multi-modal medical data of a subject, the multi-modal medical data comprising a medical image of a specimen slide, wherein a specimen in the specimen slide was collected from the subject; generating a prediction pertaining to a biological attribute of the medical image based on the medical image; determining a degree of consistency between the biological attribute of the medical image and other modalities of the multi-modal medical data; and outputting, based on the degree of consistency, an indication of whether the multi-modal medical data contain inconsistency.
1 . A computer implemented method for verifying the consistency of electronic medical records, the method comprising:
accessing electronic multi-modal medical data records of a patient, the multi-modal medical data comprising a medical image of a specimen slide;
generating a prediction pertaining to a biological attribute of the specimen based on the medical image;
determining a degree of consistency between the prediction and other modalities of the multi-modal medical data of the patient; and
outputting, based on the degree of consistency, an indication of whether the electronic multi-modal medical data records contain inconsistency.
2 . The method of claim 1 , wherein the biological attribute is associated with one or more biological attribute types, the one or more biological attribute types comprising at least one of: a source organ of the specimen, an extraction method of the specimen, a protein expression of the specimen, or a type of tumor cell in the specimen.
3 . The method of claim 1 further comprising:
defining a set of tiles, wherein each of the set of tiles includes a different portion of the medical image; and
generating, for each tile of the set of tiles, a tile-level prediction pertaining to a biological attribute of the tile,
wherein the prediction pertaining to the biological attribute of the specimen is generated based on aggregating the tile-level predictions of the set of tiles.
4 . The method of claim 3 , wherein the tile-level predictions of the set of tiles comprise tile-level predictions of a first biological attribute for a first subset of the set of tiles and a second biological attribute for a second subset of the set of tiles; and
wherein generating the prediction pertaining to the biological attribute of the specimen based on aggregating the tile-level predictions comprises:
determining a first count for the first subset of the set of tiles associated with the first biological attribute;
determining a second count for the second subset of the set of tiles attributed with the second biological attribute, the second count based on a count of the tiles in the second subset of the set of tiles and/or a scaling factor for one or more of the second subset of tiles, the scaling factor based on at least one of: a confidence level or a degree of relevance of the tile-level prediction of the second biological attribute for the tile; and
selecting the first biological attribute as the biological attribute of the specimen based on the first count being larger than the second count.
5 . The method of claim 4 , wherein generating, for each tile of the set of tiles, the tile-level prediction pertaining to the biological attribute of the tile comprises:
generating, for each tile, a probability of the tile having a candidate biological attribute for each of a set of candidate biological attributes; and
selecting the candidate biological attribute from the set of candidate biological attributes having the highest probability;
wherein the confidence level of the tile-level prediction for each tile is based on a difference between a first probability value of the candidate biological attribute selected for the tile and a second probability value of another candidate biological attribute not selected for the tile.
6 . The method of claim 3 comprising:
assigning, for each tile of the set of tiles, the tile to a particular cluster of a set of clusters based on the tile-level prediction of the tile, the assignment of a tile to a particular cluster based on a relationship between a tile-level representation and clusters of reference tile-level representations;
determining a distribution of the cluster assignments across the set of clusters based on the tile assignments;
generating the prediction pertaining to the biological attribute of the medical image based on the distribution;
generating a slide-level representation of the medical image based on the distribution; and
generating the prediction pertaining to a biological attribute of the specimen based on the slide-level representation.
7 . The method of claim 6 , wherein the particular cluster is represented by a centroid of the reference tile-level representations and a radius of the cluster; and
wherein the assignment of the tile is based on a distance between the tile-level representation and the centroid is within the radius.
8 . The method of claim 6 , wherein the slide-level representation comprises a multi-dimensional vector of multiple dimensions;
wherein each dimension of the multi-dimensional vector is associated with a cluster of the clusters of reference tile-level representations; and
wherein a value of the dimension is based on a count of the tiles assigned to the cluster associated with the dimension.
9 . The method of claim 8 , wherein the value of the dimension is based on a ratio between the count of the tiles of the set of tiles being in the cluster and a total count of the tiles in the set of tiles.
10 . The method of claim 8 wherein:
the medical image is a first medical image of the specimen slide at a first magnification;
the multi-dimensional vector is a first multi-dimensional vector generated from tiles of the first medical image;
the set of clusters is a first set of clusters associated with the first magnification; and
the slide-level representation further comprises a second multi-dimensional vector generated based on comparing tile-level representations of tiles of a second medical image of the specimen slide at a second magnification with reference tile-level representations of a second set of clusters associated with a second magnification.
11 . The method of claim 10 , wherein the tile-level representations and the reference tile-level representations are generated using a machine learning model that maps pixels of a tile to a latent space having a reduced dimension compared with a number of pixels in a tile; and
wherein the tile-level representations and the reference tile-level representations comprise embedding vectors.
12 . The method of claim 6 , wherein the prediction is generated based on inputting the slide-level representation to a machine learning model comprising one or more decision trees, the one or more decision trees being trained using gradient boosting techniques.
13 . The method of claim 6 , wherein the reference tile representations are associated with reference medical images of other subjects and different biological attributes of the same biological attribute type within the medical data of the other subjects; and
wherein the method further comprises:
performing, using the slide-level representation, a similarity search for one or more of the reference medical images that are similar to the medical image,
wherein the tile-level prediction pertaining to the biological attribute of the tile is based on the similarity search.
14 . The method of claim 13 , wherein the same biological attribute type is based on at least one of: a history of treatments of the other subjects, or results of diagnosis of the other subjects.
15 . The method of claim 1 , wherein the prediction comprises a prediction of a diagnosis of the subject.
16 . The method of claim 15 , wherein the prediction of the diagnosis of the subject comprises at least one of: a type of tumor of the subject, or a severity of the tumor of the subject.
17 . A system for verifying the consistency of electronic medical records, the system comprising:
a database configured to store medical records comprising multi-modal medical data of a plurality of patients, the multi-modal medical data comprising a medical image associated with each of the plurality of patients; and
one or more processors programmed and configured to:
access the electronic multi-modal medical data records of a patient of the plurality of patients;
based on a respective medical image, generate a prediction pertaining to a biological attribute of the patient;
determine a degree of consistency between the predicted biological attribute of the patient and other modalities of the multi-modal medical data for the patient; and
output, based on the degree of consistency, an indication of whether the electronic multi-modal medical data records for the patient contain inconsistency.
18 . The system of claim 17 , wherein the one or more processors are programmed and configured to:
based on the respective medical image, generate a plurality of predictions pertaining to a plurality of biological attributes of the patient; and
select the biological attribute among the plurality of biological attributes for determining the degree of consistency, the selection based on determining the relevancy of the selected biological attribute to the other multi-modal medical data of the patient.
19 . The system of claim 17 , wherein the one or more processors are programmed and configured to:
input the respective medical image associated with the patient into a machine learning model, the machine learning model trained to compare the respective medical image with reference medical images of other patients associated with particular biological attributes to predict the biological attribute of the patient; and
based on the comparison, determine the degree of consistency between the predicted biological attribute of the patient and other modalities of the multi-modal medical data for the patient.
20 . The system of claim 17 , wherein the one or more processors are programmed and configured to:
define a set of tiles, wherein each of the set of tiles includes a different portion of the medical image; and
generate, for each tile of the set of tiles, a tile-level prediction pertaining to the biological attribute of the tile,
wherein the prediction pertaining to the biological attribute of the specimen is generated based on aggregating the tile-level predictions of the set of tiles.