Method for classifying fundus image of subject and device using same
The present invention relates to a method for classifying a fundus image and a device using same. Specifically, according to the method of the present invention, a computing device acquires a fundus image of a subject, generates classification information of the fundus image, generates an interpretation text on the basis of the classification information, and provides the interpretation text to an external entity.
1. A method of classifying an image using a deep neural network, the method comprising:
acquiring, by a computing apparatus, the image;
generating, by a computing apparatus, classification information of the image based on the deep neural network; and
providing, by a computing apparatus, the classification information to an external entity,
wherein the classification information comprises a classification to which the image belongs and an activation map generated based on information of individual findings in the image related to the classification, and
wherein the activation map includes visualization information indicating at least one location related to the individual findings in the image.
2. The method of claim 1 , wherein the classification to which the image belongs is determined as at least one of a classification in which the image is non-interpretable, a classification in which the image is normal, and a classification in which the image is abnormal.
3. The method of claim 2 , wherein, when the image belongs to the classification in which the image is abnormal, the activation map is generated based on locations of the individual findings.
4. The method of claim 2 , wherein, when the image belongs to the classification in which the image is non-interpretable, an indication corresponding to “ungradable” is provided to the external entity.
5. The method of claim 2 , wherein the classification information further comprises confidential level information regarding the classification in which the image is normal or the classification in which the image is abnormal.
6. The method of claim 1 , further comprising:
providing, by the computing apparatus, an interface of modifying and editing the classification information.
7. The method of claim 1 , wherein the deep neural network is pretrained to decrease a value of a loss function calculated based on an input learning image, and
the loss function is calculated based on a combination of a classification loss function that is calculated based on classification results of the deep neural network and a guidance loss function that is calculated based on the activation map.
8. The method of claim 7 , wherein the guidance loss function is calculated based on an operation using a value of a first pixel comprised in a region mask provided as a label corresponding to the activation map and a value of a second pixel comprised in the activation map, and
a location of the first pixel is determined as a location on the region mask corresponding to a location of the second pixel in the activation map.
9. The method of claim 8 , wherein the guidance loss function is calculated based on a multiplication of a first function that is calculated based on the value of the first pixel and a second function that is calculated based on the value of the second pixel.
10. The method of claim 7 , wherein the guidance loss function is represented as
L
guide
(
A
,
M
)
=
1
k
W
F
H
F
∑
i
=
1
k
∑
l
=
1
W
F
H
F
(
1
-
m
l
i
)
f
(
a
l
i
+
ϵ
)
where the k denotes a number of pieces of mini-batch data that at least a portion of learning data is extracted as mini-batch learning data, the W F is a natural number and denotes a width of an activation map about target findings, the H F is a natural number and denotes a height of the activation map, the A={A 1 , . . . , A k } denotes the activation map about the target findings of the k mini-batch data where each of the A 1 , . . . , A k denotes a single element selected from a Descartes product of W F ×H F dimension of a set (0, 1) having real numbers between 0 and 1 as elements, the M={M 1 , . . . , M k } denotes a region mask about the target findings of the k mini-batch data where each of the M 1 , . . . , M k denotes a single element selected from a Descartes product of W F ×H F dimension of a set {0, 1} having 0 and 1 as elements, the m l i and the a l i , respectively, denote values of l th pixels of the M i and the A i for each natural number 1 greater than or equal to 1 and less than or equal to W F ×H F , the A i denotes a positive number introduced for a numerical stability, and the f denotes a function that is determined based on a log function.
11. The method of claim 10 , wherein the f(a l i ,∈) corresponds to one of log (a l i +∈) and log(max(a l i +∈)).
12. The method of claim 1 , wherein the deep neural network comprises a plurality of convolution layers, and comprises an output layer comprising a first output layer for outputting a guidance loss and a second output layer for outputting a classification loss as at least two output layers that follow a final 1×1 convolution layer.
13. The method of claim 12 , wherein the first output layer calculates the classification through application of a sigmoid function after global average pooling (GAP) of the final 1×1 convolution layer, and
the second output layer calculates the activation map through the sigmoid function application of the final 1×1 convolution layer.
14. The method of claim 1 , wherein, before the image is input to the deep neural network, the computing apparatus performs preprocessing of cropping and resizing the image such that a fundus portion is positioned at a center of the image and the image is resized to have a size of between 256×256 and 1024×1024 and normalizing a pixel value of the image such that the pixel value of the image belongs to the range of 0 or more and 1 or less.
15. The method of claim 14 , wherein, if the size of the image exceeds 512×512, a resolution of the image decreases for a size of 256×256 through a convolution of consecutive stride 2.
16. A non-transitory computer-readable record medium storing instructions that, when executed by a computing apparatus, cause the computing apparatus to perform the method of claim 1 .
17. A computing apparatus for classifying an image using a deep neural network, the computing apparatus comprising:
a communicator configured to acquire the image; and
a processor configured to perform a process of generating classification information of the image based on the deep neural network and a process of providing the classification information to an external entity,
wherein the classification information comprises a classification to which the image belongs and an activation map generated based on information of individual findings in the image related to the classification, and
wherein the activation map includes visualization information indicating at least one location related to the individual findings in the image.