IP Library › Granted Patent US 12,561,993

Granted Patent B2

US 12,561,993 · App. 18/289,299 · Granted Feb 24, 2026

Analysis of histopathology samples

Inventors: Hanyun Zhang (London, GB); Yinyin Yuan (London, GB)

Assignee: The Institute of Cancer Research: Royal Cancer Hospital

G06V20/698 G06V10/761 G06V10/772 G06V10/774 G06V10/82 G06V2201/03

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,561,993
App. No.: 18/289,299
Granted: Feb 24, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Methods and systems for analysing the cellular composition of a sample are described, comprising: providing an image of the sample in which a plurality of cellular populations are associated with respective signals and classifying a plurality of query cells in the image between a plurality of classes corresponding to respective cellular populations in the plurality of cellular populations. This is performed by providing a query single cell image to an encoder module of a machine learning model to produce a feature vector for the query image, and assigning the query cell to one of the plurality of classes based on the feature vector for the query image and feature vectors produced by the encoder module for each of a plurality of reference single cell images. The machine leaning model comprises: the encoder module, configured to take as input a single cell image and to produce as output a feature vector the single cell image, and a similarity module configured to take as input a pair of feature vectors for a pair of single cell images and to produce as output a score indicative of the similarity between the single cell images. Thus, the machine learning model can be obtained without the need for an extensively annotated dataset. The methods find use in the analysis of multiplex immunohistochemistry/immunofluorescence in a variety of clinical contexts.

Claims (63)

1 . A method of analysing the cellular composition of a sample, the method comprising:

providing, to a processor, an image of the sample, wherein the sample is a stained pathology sample comprising a plurality of cellular populations;

classifying, by said processor, a plurality of query cells in the image between a plurality of classes comprising one or more classes corresponding to respective cellular populations in the plurality of cellular populations, by, for each query cell:

providing a query single cell image to an encoder module of a machine learning model to produce a feature vector for the query image, wherein the machine leaning model comprises:

the encoder module, wherein the encoder module is configured to take as input a single cell image and to produce as output a feature vector the single cell image, and

a similarity module configured to take as input a pair of feature vectors for a pair of single cell images and to produce as output a score indicative of the similarity between the single cell images,

assigning the query cell to one of the plurality of classes based on the feature vector for the query single cell image and feature vectors produced by the encoder module for each of a plurality of reference single cell images comprising at least one reference single cell image for each of the plurality of cellular populations.

2 . The method of claim 1 , wherein the machine learning model has been trained to take as input a pair of sub-patches of single cell images and to produce as output a score indicative of the similarity between the sub-patches of single cell images, and to associate a first label and/or a high score to positive pairs of sub-patches of single cell images and a second label and/or a low score to negative pairs of sub-patches of single cell images, using a set of training single cell images comprising:

a plurality of positive pairs of single cell images that each show a sub-patch of the same cell, and

a plurality of negative pairs of single cell images that each show a subpatch of a different cell,

wherein the two images in a pair are different from each other.

3 . The method of claim 2 , wherein the set of training single cell images have been obtained by:

providing a plurality of single cell images;

obtaining a plurality of sub-patches from each single cell image, wherein a sub-patch is an image that comprises a portion of the single cell image; and

obtaining a plurality of positive pairs of single cell images by pairing sub-patches from the same single cell image, and a plurality of negative pairs of single cell images by pairing sub-patches from different single cell images.

4 . The method of claim 2 , wherein the machine learning model has been trained using a loss function that weighs positive pairs more than negative pairs.

5 . The method of claim 1 , wherein assigning the query cell to one of the plurality of classes based on the feature vector for the query single cell image and feature vectors produced by the encoder module for each of a plurality of reference single cell images comprises:

providing the feature vector for the query image and the feature vectors for each the plurality of reference single cell images in turn to the similarity module, thereby obtaining a plurality of scores indicative of the similarity between the query single cell image and each of the respective reference single cell image; and

assigning the query cell to one of the plurality of classes based on the plurality of scores, optionally wherein assigning the query cell to one of the plurality of classes based on the plurality of scores comprises assigning the query cell to the class that is associated with the reference single cell image that is predicted to be most similar to the query single cell image.

6 . The method of claim 5 , wherein assigning the query cell to one of the plurality of classes based on the plurality of scores comprises assigning the query cell to the class that is associated with the reference single cell image that is predicted to be most similar to the query single cell image, by quantifying a similarity metric quantified between the plurality of scores and each of corresponding pluralities of scores obtained for pairs of reference cell images and assigning the query cell to the class that is associated with the similarity metric indicative of highest similarity.

7 . The method of claim 1 , wherein classifying a query cell between a plurality of classes comprises obtaining a plurality of sub-patches from the query single cell image,

wherein providing a query single cell image to the encoder module of the machine learning model comprises providing a one or more of the plurality of sub-patches from the query single cell image to the encoder module of the machine learning model,

wherein the encoder module is configured to take as input a sub-patch of a single cell image and to produce as output a feature vector for the sub-patch of the single cell image, and the similarity module is configured to take as input a pair of feature vectors for a pair of sub-patches of single cell images and to produce as output a score indicative of the similarity between the sub-patches of single cell images; and

wherein the feature vector for the query single cell image comprises feature vectors for the one or more sub-patches of the query single cell image and the feature vectors produced by the encoder module for each of a plurality of reference single cell images comprise feature vectors produced for one or more of a plurality of sub-patches from each reference single cell image.

8 . The method of claim 7 , wherein assigning the query cell to one of the plurality of classes based on the feature vector for the query image and feature vectors produced by the encoder module for each of a plurality of reference single cell images comprises:

for each distinct pair comprising a sub-patch from the query single cell image and a sub-patch from the reference single cell image, providing the feature vector for the sub-patch for the query image and the sub-patch from the reference single cell image to the similarity module;

wherein the score indicative of the similarity between the query single cell image and the respective reference single cell image comprises a score predicted by the machine learning model for each distinct pair,

wherein assigning the query cell to one of the plurality of classes based on the plurality of scores comprises assigning the query cell to the class that is associated with the reference single cell image that comprises the sub-patch that was predicted to have the highest similarity with a sub-patch from the query single cell image.

9 . The method of claim 7 , wherein:

classifying a query cell between a plurality of classes comprises obtaining a representative sub-patch from the query single cell image,

providing a sub-patch from the query single cell image to the encoder module of the machine learning model comprises providing the representative sub-patch from the query single cell image to the encoder module,

the feature vectors produced by the encoder module for one or more sub-patches of each of a plurality of reference single cell images comprise feature vectors produced for a representative sub-patch from each reference single cell image, and

assigning the query cell to one of the plurality of classes based on the plurality of scores comprises assigning the query cell to the class that is associated with the reference single cell image whose representative sub-patch was predicted to have the highest similarity with the representative sub-patch from the query single cell image, optionally wherein the respective representative sub-patches are the central sub-patches of the respective single cell images.

10 . The method of claim 7 , wherein assigning the query cell to one of the plurality of classes based on the feature vector for the query image and feature vectors produced by the encoder module for each of a plurality of reference single cell images comprises:

assigning the query cell to one of the plurality of classes using a classifier model trained to classify the feature vectors produced by the encoder module for sub-patches of each of the plurality of reference single cell images between classes corresponding to the plurality of cellular populations, wherein the feature vectors used by the classifier for each single cell image comprise feature vectors produced by the encoder module for each of the plurality of sub-patches for the single cell image.

11 . The method of claim 1 , wherein classifying at least a subset of the cells in the image comprises obtaining a plurality of query single cell images using a machine learning model trained to locate single cells in an image.

12 . The method of claim 1 , wherein the reference set of single cell images comprises between 1 and 5 reference single cell images for each of the plurality of cellular populations.

13 . The method of claim 1 , wherein the reference set of single cell images further comprises at least one image corresponding to background signal.

14 . The method of claim 1 , wherein the method comprises providing, to the processor, a reference set of single cell images and expanding the reference set of single cell images by:

providing an initial reference set of single cell images

using the initial reference set of single cell images and the trained machine learning model to classify a plurality of query single cell images;

for each class of cells, selecting one or more query single cell images that are most dissimilar to any of the reference single cell images from the same class; and

obtaining a label for each selected query single cell image including them in an updated reference set of single cell images.

15 . The method of claim 1 , wherein the machine learning model comprises a deep learning model, preferably a Siamese neural network.

16 . The method of claim 1 , wherein the image of the sample and any training images are immunohistochemistry or immunofluorescence images, or wherein each of the plurality of cellular population is associated with a respective label or combination of labels that is represented in the image.

17 . The method of claim 1 , further comprising determining one or more cellular composition metrics selected from: the amounts, ratios, proportions or spatial distribution of cells in the sample belonging to one or more of the plurality of cellular populations using the number of query cells assigned to the respective classes corresponding to the one or more of the plurality of cellular populations, and optionally determining a diagnostic and/or prognostic for a subject from whom the sample has been obtained using said one or more cellular composition metrics.

18 . The method of claim 1 , further comprising:

providing, to the processor, a plurality of training single cell images;

providing, to the processor, a machine learning model configured to take as input a pair of images and to produce as output a score indicative of the similarity between the images; and

training, by the processor, the machine learning model to take as input a pair of single cell images or sub-patches thereof and to produce as output a score indicative of the similarity between the single cell images or sub-patches thereof, by training the machine learning model to associate a first label and/or a high score to positive pairs of training single cell images and a second label and/or a low score to negative pairs of training single cell images, wherein positive pairs comprise pairs of distinct single cell images or sub-patches thereof that each show at least a portion of the same cell (positive pairs), and

negative pairs comprises pairs of distinct single cell images or sub-patches thereof that each show at least a portion of a different cell.

19 . A system for analysing the cellular composition of a sample, the system comprising one or more processor and one or more computer readable medium/media comprising instructions that, when executed by the processor, cause the processor to perform a method comprising:

receiving, by the one or more processors, an image of the sample, wherein the sample is a stained pathology sample comprising a plurality of cellular populations;

providing a query single cell image to an encoder module of a machine learning model to produce a feature vector for the query image, wherein the machine leaning model comprises:

the encoder module, wherein the encoder module is configured to take as input a single cell image and to produce as output a feature vector the single cell image, and

20 . One or more non-transitory computer readable media comprising instructions that, when executed by at least one processor, cause the at least one processor to perform a method comprising:

obtaining, by the at least one processor, a plurality of training single cell images from stained pathology samples;

obtaining, by the at least one processor a machine learning model configured to take as input a pair of images and to produce as output a score indicative of the similarity between the images; and

training the machine learning model to take as input a pair of sub-patches of single cell images and to produce as output a score indicative of the similarity between the sub-patches of single cell images, by training the machine learning model to associate a first label and/or a high score to positive pairs of training single cell images and a second label and/or a low score to negative pairs of training single cell images, wherein positive pairs comprise pairs of distinct sub-patches of single cell images that each show at least a portion of the same cell (positive pairs), and

negative pairs comprise pairs of distinct sub-patches of single cell images that each show at least a portion of a different cell.

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 2, 2023

From: ZHANG, HANYUN; YUAN, YINYIN

To: THE INSTITUTE OF CANCER RESEARCH: ROYAL CANCER HOSPITAL

Reel/Frame 065437/0961 →

Priority Claims (1)

GB 2106397 · May 5, 2021 · national

Continuity (1)

Related Publication 20240233416A1 · Jul 11, 2024

References Cited (32)

US 20210118136A1 · Hassan-Shafique et al. · 2021 [cited by applicant]

US 20220139072A1 · Klaiman · 2022 [cited by examiner]

Hradel et al., “Interpretable diagnosis of breast cancer from histological images using Siamese neural networks” (Year: 2020). [cited by examiner]

Yarlagadda et al. “A system for one-shot learning of cervical cancer cell classification in histopathology images” (Year: 2019). [cited by examiner]

Abduljabbar et al., “Geospatial Immune Variability Illuminates Differential Evolution of Lung Adenocarcinoma,” [cited by applicant]

Abousamra et al., “Weakly-Supervised Deep Stain Decomposition for Multiple IHC Images,” [cited by applicant]

Bankhead et al., “QuPath: Open Source Software for Digital Pathology Image Analysis,” [cited by applicant]

Bindea et al., “Spatiotemporal Dynamics of Intratumoral Immune Cells Reveal the Immune Landscape in Human Cancer,” [cited by applicant]

Bromley et al., “Signature Verification Using a “Siamese” Time Delay Neural Network,” [cited by applicant]

Chang et al., “LIBSVM: A Library for Support Vector Machines,” [cited by applicant]

Chen et al., “A Simple Frankwork for Contrastive Learning of Visual Representations,” [cited by applicant]

Fassler et al., “Deep Learning-based Image Analysis Methods for Brightfield-Acquired Multiplex Immunohistochemistry Images,” [cited by applicant]

Galon et al., “Type, Density, and Location of Immune Cells Within Human Colorectal Tumors Predict Clinical Outcome,” [cited by applicant]

Gerdes et al., “Highly Multiplexed Single-Cell Analysis of Formalin-Fixed, Paraffin-Embedded Cancer Tissue,” [cited by applicant]

Giesen et al., “Highly Multiplexed Imaging of Tumor Tissues with Subcellular Resolution by Mass Cytometry,” [cited by applicant]

Hagos et al., “ConCORDe-Net: Cell Count Regularized Convolutional Neural Network for Cell Detection in Multiplex Immunohistochemistry Images,” [cited by applicant]

Ma et al., “Data Integration from Pathology Slides for Quantitative Imaging of Multiple Cell Types Within the Tumor Immune Cell Infiltrate,” [cited by applicant]

Nalepa et al., “Selecting Training Sets for Support Vector Machines: A Review,” [cited by applicant]

Narayanan et al., “Unmasking the Immune Microecology of Ductal Carcinoma in Situ with Deep Learning,” [cited by applicant]

Perkin Elmer, “User's Manual for Nuance 3.0.2,” 2010 (132 pages). [cited by applicant]

Raza et al., “Deconvolving Convolutional Neural Network for Cell Detection,” [cited by applicant]

Sirinukunwattana et al., “Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images”, [cited by applicant]

Tan et al., “Overview of Multiplex Immunohistochemistry/Immunofluorescence Techniques in the Era of Cancer Immunotherapy,” [cited by applicant]

Tamborero et al., “A Pan-cancer Landscape of Interactions between Solid Tumors and Infiltrating Immune Cell Populations,” [cited by applicant]

Tirosh et al., “Dissecting the multicellular ecosystem of metastatic melanoma by single-cell RNA-seq,” [cited by applicant]

Tsyurmasto et al., “Value-at-Risk Support Vector Machine: Stability to Outliers,” [cited by applicant]

Gildenblat et al., “Self-Supervised Similarity Learning for Digital Pathology,” [cited by applicant]

Intellectual Property Office Search Report dated Feb. 9, 2022, issued for GB 2106397.9 (2 pages). [cited by applicant]

International Search Report and Written Opinion dated Aug. 5, 2022, issued for PCT/EP2022/061941 (18 pages). [cited by applicant]

Medela et al., “Few Shot Learning in Histopathological Images: Reducing the Need of Labeled Data on Biological Datasets,” [cited by applicant]

Yang et al., “Liver Histopathological Image Retrieval Based on Deep Metric Learning,” [cited by applicant]

Yarlagadda et al., “A System for One-Shot Learning of Cervical Cancer Cell Classification in Histopathology Images,” [cited by applicant]