Granted Patent B2

US 12,293,295 · App. 17/753,931 · Granted May 6, 2025

Inventors: Sepp De Raedt (Oslo, NO); Ole-Johan Skrede (Oslo, NO); Håvard Emil Greger Danielsen (Oslo, NO); Tarjei Sveinsgjerd Hveem (Oslo, NO); Andreas Kleppe (Oslo, NO); Knut Liestøl (Oslo, NO)

Assignee: OSLO UNIVERSITETSSYKEHUS

G06N3/084A61B5/4848A61B34/10G06T7/0016G06V10/26G06V10/454G06V10/774G06V10/776G06V10/80G06V10/82G06V10/87G06V20/695G06V20/698G06V20/70G06V30/2504G16H50/20G06T2207/20021G06T2207/20081G06T2207/20084G06T2207/30024G06V2201/03

Abstract

Claims (65)

1. A computer implemented system for determining an overall-classifier of one or more source-histopathological-images, wherein each source-histopathological-image has been obtained from one or more histopathological samples obtained from one or more subjects, and wherein each subject has been diagnosed as having, is suspected of having, is being treated for, has previously been treated for, and/or has previously had, cancer, the system comprising:

a first tile generator configured to generate a plurality of first-tiles from the one or more source-histopathological-images, wherein each of the plurality of first-tiles comprises a plurality of pixels that represents a region of the one or more source-histopathological-images having a first-area and a first-resolution;

a second tile generator configured to generate a plurality of second-tiles from the one or more source-histopathological-images, wherein each of the plurality of second-tiles comprises a plurality of pixels that represents a region of the one or more source-histopathological-images having a second-area and a second-resolution, wherein:

the first-area of the first-tiles is larger than the second-area of the second-tiles; and

the second-resolution of the second-tiles is higher than the first-resolution of the first-tiles;

a first machine-learning network configured to process the plurality of first-tiles in order to determine a first-classifier for the one or more source-histopathological-images, wherein the first machine-learning network ( 311 ) comprises:

a first-neural-network configured to process the plurality of first-tiles in order to determine a tile-feature for each of the plurality of first-tiles;

a pooling-function configured to combine subsets of the tile-features to generate a bag-feature for each of the subsets; and

a second-neural-network configured to process the bag-features in order to determine a first-classifier for the one or more source-histopathological-images, wherein the second-neural-network is a classification network;

a second machine-learning network configured to process the plurality of second-tiles in order to determine a second-classifier for the one or more source-histopathological-images, wherein the second machine-learning network comprises:

a first neural-network configured to process the plurality of second-tiles in order to determine a tile-feature for each of the plurality of second-tiles;

a pooling-function configured to combine subsets of the tile-features to generate a bag-feature for each of the subsets; and

a second-neural-network configured to process the bag-features in order to determine a second-classifier for the one or more source-histopathological-images, wherein the second-neural-network is a classification network; and

a classifier combiner configured to combine the first-classifier and the second-classifier to determine the overall-classifier for the one or more source-histopathological-images.

2. The system of claim 1 , wherein the classifier combiner is configured to:

apply a thresholding function to the first-classifier in order to determine a thresholded-first-classifier;

apply a thresholding function to the second-classifier in order to determine a thresholded-second-classifier; and

combine the thresholded-first-classifier and the thresholded-second-classifier to determine the overall-classifier.

3. The system of claim 1 , wherein:

the first machine-learning network is configured to process the plurality of first-tiles in order to determine a plurality of first-classifiers for the one or more source-histopathological-images;

the second machine-learning network is configured to process the plurality of second-tiles in order to determine a plurality of second-classifiers for the one or more source-histopathological-images; and

the classifier combiner is configured to:

apply a statistical function to the plurality of first-classifiers in order to determine a combined-first-classifier;

apply a statistical function to the plurality of second-classifiers in order to determine a combined-second-classifier;

combine the combined-first-classifier and the combined-second-classifier to determine the overall-classifier.

4. The system of claim 1 , wherein the classifier combiner is configured to perform a logical combination of the first-classifier and the second-classifier to determine the overall-classifier for the one or more source-histopathological-images.

5. The system of claim 1 , wherein the first machine-learning network, the second machine-learning network, or both the first and second machine-learning networks-further comprise:

a loss-function configured to:

compare the classifier that is determined by the second-neural-network with a ground-truth that is represented by truth-data, and

set trainable parameters for the first-neural-network, the pooling-function, and the second-neural-network based on the result of the comparison.

6. The system of claim 1 , further comprising:

a segmentation block that is configured to apply an image segmentation method to a whole-slide-image-histopathological image in order to provide a source-histopathological-image.

7. The system according to claim 1 , wherein the first tile generator and the second tile generator are configured to generate their respective tiles independently of one another.

8. The system according to claim 1 , wherein:

at least one of the first machine-learning networks and second machine-learning networks have been trained using training-histopathological-images and associated ground-truths, and

the one or more training-histopathological-images has been obtained from one or more histopathological samples obtained from one or more subjects, and wherein each subject has, has been diagnosed as having, is suspected of having, is being treated for, has previously been treated for, and/or has previously had, cancer.

9. The system of claim 1 , wherein the cancer is selected from the group consisting of carcinoma, sarcoma, myeloma, leukemia, lymphoma and a mixed type of cancer.

10. The system of claim 9 , wherein the cancer is a colorectal cancer, or lung cancer.

11. A computer implemented method of determining an overall-classifier for one or more source-histopathological-images, the method comprising:

generating a plurality of first-tiles from the one or more source-histopathological-images, wherein each of the plurality of first-tiles comprises a plurality of pixels that represents a region of the one or more source-histopathological-images having a first-area and a first-resolution;

generating a plurality of second-tiles from the one or more source-histopathological-images, wherein each of the plurality of second-tiles comprises a plurality of pixels that represents a region of the one or more source-histopathological-images having a second-area and a second-resolution, wherein:

the first-area of the first-tiles is larger than the second-area of the second-tiles; and

the second-resolution of the second-tiles is higher than the first-resolution of the first-tiles;

applying a first machine-learning network to the plurality of first-tiles in order to determine a first-classifier for the one or more source-histopathological-images, wherein applying the first machine-learning network comprises:

applying a first-neural-network to the plurality of first-tiles in order to determine a tile-feature for each of the plurality of first-tiles;

combining subsets of the tile-features to generate a bag-feature for each of the subsets; and

applying a second-neural-network to the bag-features in order to determine a first-classifier for the one or more source-histopathological-images, wherein the second-neural-network is a classification network;

applying a second machine-learning network to the plurality of second-tiles in order to determine a second-classifier for the one or more source-histopathological-images, wherein applying the second machine-learning network comprises:

applying a first-neural-network to the plurality of second-tiles in order to determine a tile-feature for each of the plurality of second-tiles;

combining subsets of the tile-features to generate a bag-feature for each of the subsets; and

applying a second-neural-network to the bag-features in order to determine a second-classifier for the one or more source-histopathological-images, wherein the second-neural-network is a classification network; and

combining the first-classifier and the second-classifier to determine the overall-classifier for the one or more source-histopathological-images.

12. The computer implemented method of processing one claim 11 , and wherein the method further comprises:

attributing a diagnostic and/or prognostic evaluation to the classifier and/or overall-classifier.

13. The computer-implemented method of claim 12 , wherein the subject is a human.

14. The computer-implemented method of claim 12 , wherein the one or more histopathological samples obtained from the subject is, or are, obtained from the part of the subject's body that has, is suspected of having, is being treated for, has been treated for, and/or has previously had, cancer.

15. The computer-implemented method of claim 12 , wherein the method comprises assessing a plurality of source-histopathological-images obtained from a plurality of histopathological samples obtained from the subject in order to determine a plurality of classifiers and/or overall-classifiers, and: optionally attributing the diagnostic and/or prognostic evaluation to the plurality of classifiers and/or overall-classifiers.

16. The computer-implemented method of claim 12 , wherein the method comprises assessing one or more further diagnostic and/or prognostic markers for the cancer, and wherein the step of attributing a diagnostic and/or prognostic evaluation to the classifier and/or overall-classifier includes an assessment of the or each of the results of the assessment of the or each further diagnostic and/or prognostic markers.

17. The computer-implemented method of claim 12 , wherein the method further comprises making a treatment decision for the subject on the basis of the diagnostic and/or prognostic evaluation, optionally wherein the treatment decision is in respect of a diagnosed or prognosed cancer condition, for example a cancer selected from the group consisting of carcinoma, sarcoma, myeloma, leukemia, lymphoma and a mixed type of cancer, and optionally wherein the cancer is a colorectal cancer or lung cancer.

18. The computer-implemented method of claim 17 , wherein the diagnostic and/or prognostic evaluation of the subject includes the assessment of the effect on the subject of an earlier, or ongoing, treatment by surgery and/or non-surgical therapy, for example, in order to monitor the progress and/or effect of such treatment, and further optionally wherein the method includes the step of making a further treatment decision, such as the cessation, continuation, repetition or modification an earlier, or ongoing, treatment and/or the implementation of a different treatment modality, and optionally, implementing that further treatment decision in respect of the subject; wherein: the diagnostic and/or prognostic evaluation, the treatment and/or the treatment decision is in respect of a cancer condition, for example a cancer selected from the group consisting of carcinoma, sarcoma, myeloma, leukemia, lymphoma and a mixed type of cancer, and optionally wherein the cancer is a colorectal cancer.

19. The computer-implemented method of claim 12 , the method comprising treating the subject by a method of surgery and/or non-surgical therapy; wherein the treatment for the diagnosed or prognosed pathological condition is for the treatment of cancer, for example a cancer selected from the group consisting of carcinoma, sarcoma, myeloma, leukemia, lymphoma and a mixed type of cancer, and optionally wherein the cancer is a colorectal cancer or lung cancer.

20. The computer-implemented method of claim 19 , wherein the subject is a human.

21. The computer-implemented method of claim 19 , wherein the subject: (a) has, has been diagnosed as having, is suspected of having, is being treated for, has previously been treated for, and/or has previously had, cancer; and/or (b) wherein a diagnostic and/or prognostic evaluation of a cancer condition has been attributed to the subject by a method according to any of claims 10 to 17 .

22. The computer-implemented method of claim 21 , wherein the pathological condition is a cancer selected from the group consisting of carcinoma, sarcoma, myeloma, leukemia, lymphoma and a mixed type of cancer, and optionally wherein the cancer is a colorectal cancer or lung cancer.

23. The computer-implemented method of claim 19 , wherein the method comprises adapting one or more parameters of the surgery and/or non-surgical therapy in view of the diagnostic and/or prognostic evaluation that has been attributed to the subject by a method according to any of claims 10 to 17 , and optionally wherein the one or more parameters of the surgery and/or non-surgical therapy are selected from the group consisting of the nature of the surgery and/or non-surgical therapy, the timing of the surgery and/or non-surgical therapy, period of the surgery and/or non-surgical therapy, the dosage of the therapy, the route of administration of the non-surgical therapy, and the site(s) in the body that is targeted by the surgery and/or non-surgical therapy.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (38)