Method and apparatus for providing information associated with immune phenotypes for pathology slide image

1 . A computing device comprising:

a memory storing one or more instructions; and

a processor configured to execute the stored one or more instructions for:

acquiring a digital image corresponding to an H&E-stained pathology slide;

detecting, using a deep learning model executed by the processor, one or more target items including immune cells, a cancer area, and a cancer stroma area within a plurality of patches, wherein the plurality of patches are generated by dividing the digital image into equal-sized patches, and the deep learning model is trained based on a reference pathology slide image and label information for one or more reference target items;

determining, based on a density of immune cells in the cancer area and a density of immune cells in the cancer stroma area for each patch of the plurality of patches, immune phenotypes of first patches from among the plurality of patches that satisfy a predetermined condition associated with at least one of a number of the immune cells, an area of the cancer area, or an area of the cancer stroma area, and excluding one or more second patches from among the plurality of patches that do not satisfy the predetermined condition;

generating a prediction result indicating whether a patient associated with the digital image responds to an immune checkpoint inhibitor, based on immune phenotype scores calculated from information associated with the immune phenotypes of the first patches from among the plurality of patches;

overlaying a distinct visual representation for each of the first patches from among the plurality of patches on the digital image, the distinct visual representation being indicative of the immune phenotype of the corresponding patch and comprising at least one of color, brightness, saturation, or a mark corresponding to the immune phenotype; and

outputting, to a user, the digital image with the overlaid information and the prediction result, using a display device of the computing device.

2 . A computing device according to claim 1 , wherein the overlaying and outputting includes:

generating one or more visual representations corresponding to the immune phenotypes for the first patches; and

overlaying and outputting the generated one or more visual representations on the digital image,

wherein the immune phenotypes are determined based on at least one of a plurality of classes, each of which is indicative of an immune environment of a patch of the first patches, and

wherein the image including the one or more visual representations includes:

a first visual representation corresponding to a first class among the plurality of classes in a region corresponding to a patch of the first patches having an immune phenotype of the first class, and

a second visual representation corresponding to a second class among the plurality of classes in a region corresponding to a patch of the first patches having an immune phenotype of the second class.

3 . A computing device according to claim 1 , wherein one or more regions of interest (ROIs) are determined from among the plurality of patches obtained by dividing the digital image into equal sizes.

4 . A computing device according to claim 1 , wherein each of the immune phenotypes is indicative of an immune environment of the patch.

5 . A computing device according to claim 1 , wherein the determining of the immune phenotypes comprises:

determining an immune phenotype of a patch of the first patches as the immune inflamed, based on a first density of immune cells in the cancer area in the patch being greater than or equal to a first threshold density;

determining the immune phenotype of the patch as the immune excluded, based on the first density of immune cells in the cancer area in the patch being less than the first threshold density and a second density of immune cells in the cancer stroma area in the patch being greater than or equal to a second threshold density; and

determining the immune phenotype of the patch as the immune desert, based on the first density of immune cells in the cancer area in the patch being less than the first threshold density and the second density of immune cells in the cancer stroma area in the patch being less than the second threshold density.

6 . A method, performed by at least one computing device, for providing information associated with an immune phenotype for a digital image, comprising:

acquiring a digital image corresponding to an H&E-stained pathology slide;

detecting, using a deep learning model executed by the at least one computing device, one or more target items including immune cells, a cancer area, and a cancer stroma area within a plurality of patches, wherein the plurality of patches are generated by dividing the digital image into equal-sized patches, and the deep learning model trained, is trained based on a reference pathology slide image and label information for one or more reference target items, to detect one or more target items;

determining, based on a density of immune cells in the cancer area and a density of immune cells in the cancer stroma area for each patch of the plurality of patches, the immune phenotypes of first patches from among the plurality of patches that satisfy a predetermined condition associated with at least one of a number of the immune cells, an area of the cancer area, or an area of the cancer stroma area, and excluding one or more second patches from among the plurality of patches that do not satisfy the predetermined condition;

generating a prediction result indicating whether a patient associated with the digital image responds to an immune checkpoint inhibitor, based on immune phenotype scores calculated from information associated with the immune phenotypes of the first patches from among the plurality of patches;

overlaying a distinct visual representation for each of the first patches from among the plurality of patches on the digital image, the distinct visual representation being indicative of the immune phenotype of the corresponding patch and comprising at least one of color, brightness, saturation, or a mark corresponding to the immune phenotype; and

outputting, to a user, the digital image with the overlaid information and the prediction result.

7 . The method according to claim 6 , wherein one or more regions of interest (ROIs) are determined based on at least one of the detection result for the cancer area in the patch or the detection result for the cancer stroma area in the patch.

8 . The method according to claim 6 , wherein one or more regions of interest (ROIs) are regions that satisfy the predetermined condition associated with the immune cells, the cancer area and the cancer stroma area.

9 . The method according to claim 6 , wherein one or more regions of interest (ROIs) are determined from among the plurality of patches obtained by dividing the digital image into equal sizes.

10 . The method according to claim 6 , wherein each of the immune phenotypes is indicative of an immune environment of the patch.

11 . The method according to claim 6 , further comprising:

calculating an inflamed score based on information associated with the immune phenotype for each of the first patches; and

outputting the prediction result indicating whether the patient, from which the digital image has been generated, responds to the immune checkpoint inhibitor, based on the inflamed score.

12 . The method according to claim 6 , wherein the determining includes:

calculating a density of the immune cells, for each of the first patches; and

determining the immune phenotype of each of the first patches based on the calculated density of the immune cells.

13 . The method according to claim 12 , wherein the calculating includes calculating, for each of the first patches, a density of the immune cells in the cancer area in the patch and a density of the immune cells in the cancer stroma area in the patch.

14 . The method according to claim 6 , further comprising:

outputting at least one of an area of total tissue area in the digital image, an area of total cancer area in the digital image, an area of total regions of interest (ROIs), a graph showing proportions of immune phenotypes, an inflamed score calculated based on information associated with the immune phenotype for each of the first patches, or cutoff information used to predict whether the patient, from which the digital image has been generated, responds to the immune checkpoint inhibitor.

15 . The method according to claim 6 , wherein:

the detecting includes detecting tumor cells, the immune cells, the cancer area, and the cancer stroma area in the digital image,

the method further comprises overlaying and outputting a detection result on the digital image, wherein the detection result includes a detection result for at least one of the tumor cells, the immune cells, the cancer area or the cancer stroma area.

16 . The method according to claim 15 , wherein the tumor cells, the immune cells, the cancer area, and the cancer stroma area are visually different from each other on the digital image.

17 . A non-transitory computer-readable recording medium storing a computer program for executing, on a computer, the method for providing the information associated with the immune phenotype for the digital image according to claim 6 .

18 . The method according to claim 6 , wherein the determining of the immune phenotypes comprises:

determining an immune phenotype of a patch of the first patches as the immune inflamed, based on a first density of immune cells in the cancer area in the patch being greater than or equal to a first threshold density;

determining the immune phenotype of the patch as the immune excluded, based on the first density of immune cells in the cancer area in the patch being less than the first threshold density and a second density of immune cells in the cancer stroma area in the patch being greater than or equal to a second threshold density; and

determining the immune phenotype of the patch as the immune desert, based on the first density of immune cells in the cancer area in the patch being less than the first threshold density and the second density of immune cells in the cancer stroma area in the patch being less than the second threshold density.

19 . The method according to claim 18 , wherein the immune phenotype is determined based on at least one of a plurality of classes, each of which is indicative of an immune environment of the patch,

the image including the one or more visual representations includes:

a first visual representation corresponding to a first class among the plurality of classes in a region corresponding to a patch of the first patches having an immune phenotype of the first class, and

a second visual representation corresponding to a second class among the plurality of classes in a region corresponding to a patch of the first patches having an immune phenotype of the second class.

20 . The method according to claim 19 , wherein the first visual representation and the second visual representation are visually different from each other.