Deep learning based image segmentation method including biodegradable stent in intravascular optical tomography image
Disclosed is an image segmentation device. The image segmentation device may include: a storage unit storing a segmentation model learned so as to segment at least one predetermined object; and at least one processor inputting input data into the segmentation model and segmenting at least one predetermined object in the input data, in which the segmentation model may include an encoder including at least one dimension reduction block reducing a dimension of the input data, a decoder including at least one dimension increase block increasing the dimension of output data output from the encoder by using data output from at least one dimension reduction block, and an auxiliary classification model receiving the output data and recognizing whether a specific object is included in the output data.
1 . A method of detecting at least one predetermined object in input data, the method comprising:
inputting the input data, using at least one processor, into a segmentation model of an image segmentation device that comprises the at least one processor and a non-transient, computer- readable storage unit storing the segmentation model trained using learning images so as to segment the at least one predetermined object, wherein the segmentation model includes
an encoder including at least one dimension reduction block reducing a dimension of the input data,
a decoder including at least one dimension increase block increasing the dimension of output data outputted from the encoder by using data outputted from at least one dimension reduction block, and
an auxiliary classification model receiving the output data outputted from the encoder and recognizing whether a specific object is included in the output data to produce a binary classification result,
wherein the at least one predetermined object includes at least one of a strut and tissue of a biodegradable stent, and
wherein the input data is an intravascular optical coherence tomography (OCT) image, and
wherein the at least one processor optimizes weight values included in the decoder of the segmentation model so as to express each of the at least one predetermined object included in the learning images in a final feature value map of the decoder and wherein the at least one processor optimizes weight values included in the encoder and the auxiliary classification model of the segmentation model so as to recognize whether the specific object is included in the learning images to produce the binary classification result through the auxiliary classification model so that a feature value for strut segmentation of the encoder and the decoder is further optimized when training the segmentation model, wherein further optimization of the encoder and the decoder comprises minimizing a loss function L that is calculated using a term for the binary classification result of the auxiliary classification model, and
wherein the learning images include
a plurality of first learning images including a first biodegradable stent configured by a strut having a first thickness, and
a plurality of second learning images including a second biodegradable stent configured by a strut having a second thickness different from the first thickness; and,
segmenting the at least one predetermined object in the input data, thereby detecting the at least one predetermined object in the input data.
2 . The method of claim 1 , wherein the learning images include at least one patch acquired by randomly cropping an image labeled with each of the at least one predetermined object.
3 . The method of claim 1 , wherein a final output end of the decoder further includes a boundary refine module adjusting a final feature map inputted into a final classification operation so that a boundary for area segmentation related to each of the at least one predetermined object is refined.
4 . The method of claim 3 , wherein the final classification operation includes an operation through at least one convolutional layer and a Softmax activation function.
5 . The method of claim 3 , wherein the boundary refine module has a residual block structure.
6 . The method of claim 1 , wherein the at least one dimension reduction block and the at least one dimension increase block include a multi-scale dilated residual block having a plurality of convolutional layers having different dilation rates arranged in parallel and including a residual connection.
7 . The method of claim 1 , wherein the at least one dimension reduction block includes a dropout layer for preventing overfitting.
8 . The method of claim 7 , wherein a final layer of the encoder includes the dropout layer.
9 . The method of claim 7 , wherein the dropout layer comprises a 2-dimension (D) dropout layer for preventing overfitting for 2D-structure input data.
10 . The method of claim 1 , wherein a bottleneck block is disposed on a final layer of the encoder.
11 . The method of claim 1 , wherein the at least one processor performs 3D rendering by using final data outputted by inputting the input data into the segmentation model, and the input data.
12 . The method of claim 1 , wherein the auxiliary classification model receives the output data output from a bottleneck block of the encoder.
13 . The method of claim 1 , wherein the at least one processor generates a graph so as to perform quantitative measurement and comparison for a degree of newborn membrane proliferation of implanted biodegradable stents.
14 . The method of claim 1 , wherein the specific object comprises a strut of a biodegradable stent.
15 . The method of claim 1 , wherein the specific object comprises a tissue.
16 . The method of claim 1 , wherein the learning images comprise at least one patch acquired by random crop of an image labeled with the at least one predetermined object.
17 . The method of claim 16 , wherein the auxiliary classification model is learned whether the specific object is included in a random cropped patch.
18 . The method of claim 1 , wherein the loss function L is calculated using a cross entropy term for the binary classification result of the auxiliary classification model.
19 . The method of claim 1 , wherein the loss function L is calculated using the following equation:
L
=
λ
1
L
logloss
+
λ
2
L
L
1
+
λ
3
L
aux
+
λ
4
W
F
2
where λ i is the weight, λ logloss is a weighted average cross entropy term, L L1 is an L1 distance term for considering sparsity in segmenting a strut area, L aux is a cross entropy term for the binary classification result of the auxiliary classification model, and
W
F
2
is a Frobenius norm of all weights of the segmentation model.
20 . The method of claim 1 , further comprising implanting the biodegradable stent into a body organ having a tubular structure of a patient prior to inputting the input data.