Tissue staining and sequential imaging of biological samples for deep learning image analysis and virtual staining

1 . A computer-implemented method for training a ground truth generator machine learning model, comprising:

creating a ground truth multi-record training dataset wherein a record comprises:

a first image of a sample of tissue of a subject depicting a first group of biological objects, presenting at least one biomarker,

a second image, comprising at least a portion of the same sample of tissue, wherein the second image comprises one or more biological objects in the first group of biological objects, sequentially stained with a stain configured to stain a second group of biological objects presenting at least one second biomarker, and

ground truth labels indicating a respective biological object category, selected from a plurality of biological object categories, for a subgroup of biological objects in the first and second images;

optimizing the ground truth generator machine learning model based on the ground truth labels in the ground truth multi-record training dataset, for automatically generating ground truth labels for biological objects depicted in an input set of images of a first type corresponding to the first image;

feeding unlabeled sets of images of samples of tissue of a plurality of sample individuals into the ground truth generator machine learning model to obtain automatically generated ground truth labels, wherein the unlabeled sets of images depict biological objects respectively presenting the at least one first biomarker and at least one second biomarker; and

creating a synthetic multi-record training dataset, wherein a synthetic record comprises images depicting the at least one first biomarker, labelled with the automatically generated ground truth labels obtained from the ground truth generator machine learning model:

wherein ground truth labels of biological objects presenting the at least one second biomarker depicted in one respective image of a respective set are mapped to corresponding non-labeled biological objects presenting the at least one first biomarker depicted in another respective image of the set, and

wherein synthetic records of biological objects depicting the at least one first biomarker are labelled with ground truth labels mapped from biological objects depicting the at least one second biomarker.

2 . The computer-implemented method of claim 1 , wherein the first image comprises a brightfield image, and the second images comprises at least one of: (i) a fluorescent image with fluorescent markers indicating the at least one second biomarker, (ii) spectral imaging image indicating the at least one second biomarker, and (iii) a non-labelled image depicting the at least one second biomarker.

3 . The computer-implemented method of claim 1 , further comprising:

training a biological object machine learning model on the synthetic multi-record training dataset for generating an outcome of at least one of the plurality of biological object categories for respective target biological objects depicted in a target image depicting biological objects presenting at least one first biomarker.

4 . The computer-implemented method of claim 3 , further comprising:

training a diagnosis machine learning model on a biological object category multi-record training dataset comprising images depicting biological objects presenting at least one first biomarker, wherein biological objects depicted in the images are labelled with at least one of the plurality of biological object categories obtained as an outcome of the biological object machine learning model in response to input of the images, wherein images are labelled with ground truth labels indicative of a diagnosis.

5 . The computer-implemented method of claim 1 , wherein the biological objects comprise cells.

6 . A computer-implemented method of generating ground truth images from a ground truth multi-record dataset comprising:

segmenting biological visual features of a first image of a first record using an automated segmentation process;

mapping the segmentation of the features of the first image to a second image of the first record;

for a respective segmentation of the second image:

computing intensity values of pixels within and/or in proximity to a surrounding of the respective segmentation indicating a visual depiction of at least one biomarker in the second image; and

classifying the respective segmentation with one or more ground truth labels by mapping the computed intensity value using a set of rules to a classification category;

wherein the ground truth multi-record dataset comprises multiple records, and

wherein a second record comprises:

a first image of a sample depicting a first group of cells,

a second image of the same sample comprising the same group of cells, sequentially stained with a stain configured to stain a group of cells presenting at least one biomarker, and

one or more ground truth labels indicating a respective category for at least one sub-group of cells in the second images.

7 . A computer-implemented method of automatically generating ground truth labels for biological objects, comprising:

feeding unlabeled sets of images of samples of tissue of a plurality of sample individuals into a ground truth generator machine learning model, wherein the unlabeled sets of images depict biological objects respectively presenting at least one first biomarker and at least one second biomarker different than the at least one first biomarker; and

obtaining automatically generated ground truth labels as an outcome of the ground truth generator machine learning model,

wherein the ground truth generator machine learning model is trained on a ground truth multi-record training dataset wherein a record comprises:

a first image of a sample of tissue of a subject depicting a first group of biological objects presenting at least one first biomarker,

a second image, comprising at least a portion of the same sample of tissue comprising one or more biological objects in the first group of biological objects, sequentially stained with a stain configured to stain a second group of biological objects presenting at least one second biomarker different from the at least one first biomarker, and

ground truth labels indicating a respective biological object category for at least one sub-group of biological objects in the first and second images.

8 . The computer-implemented method of claim 7 , wherein in response to receiving a respective image of a respective sample of tissue of a respective sample individual depicting the first group of biological objects presenting the at least one first biomarker, the respective image is fed into a virtual stainer machine learning model; and

obtaining as an outcome of the virtual stainer machine learning model at least one of: (i) a synthetic image used as a second image of the unlabeled sets of images, and (ii) a synthetic image used as the second image of the sample of the record of the ground truth multi-record training dataset,

wherein the virtual stainer machine learning model is trained on a virtual imaging multi-record, wherein a record comprises a first image of a sample of tissue of a subject depicting a first group of biological objects presenting the at the least one first biomarker, and a ground truth indicated by a corresponding second image of the same sample of tissue depicting a second group of biological objects presenting the at least one second biomarker.

9 . The computer-implemented method of claim 7 , further comprising

feeding a target image depicting a plurality of biological objects presenting the at least one first biomarker into a biological object machine learning model; and

obtaining at least one of the plurality of biological object categories for respective target biological objects depicted in the target image as an outcome of the biological object machine learning model,

wherein the biological object machine learning model is trained on a synthetic multi-record training dataset, wherein a synthetic record comprises images depicting the at least one first biomarker and excludes images depicting the at least one second biomarker, labelled with the automatically generated ground truth labels obtained from the ground truth generator machine learning model.

10 . The computer-implemented method of claim 7 , wherein the biological objects comprise cells, wherein the first biomarker comprises PD-L1 and the second biomarker comprises a biomarker expressed by at least one type of mammalian immune cell, and the ground truth labels are cell-level annotations indicating immune cell type.

11 . The computer-implemented method of claim 7 , wherein the biological objects comprise cells, wherein the ground truth labels comprise cell-level annotations for biological objects in the first and second images.

12 . The computer-implemented method of claim 6 , wherein the biomarker in the second image is P40.