Breast ultrasound diagnosis method and system using weakly supervised deep-learning artificial intelligence
A breast ultrasound diagnosis method using weakly supervised deep-learning artificial intelligence comprises: an ultrasound image preprocessing step of generating input data including only an image region necessary for learning, by deleting personal information about a patient from a breast ultrasound image; a deep-learning step of receiving the input data, obtaining a feature map from the received input data by using a convolutional neural network (CNN) and global average pooling (GAP), and carrying out re-learning; and a differential diagnosis step of determining the input data as one of normal, benign, and malignant by using the GAP, and when the input data is determined to be malignant, calculating a probability of malignancy (POM) indicating accuracy of the determination.
1 . A method performed by a computing device for diagnosing ultrasound images, the method comprising:
acquiring ultrasound image data;
generating one or more feature maps based on the ultrasound image data by using a deep learning model;
computing a score for each class of the ultrasound image data based on the one or more feature maps by using the deep learning model;
generating classification information based on the score for each class;
computing a probability of malignancy based on the classification information;
generating a class activation map for each class based on the one or more feature maps;
normalizing the class activation map;
binarizing the normalized class activation map by using a preset threshold;
determining a contribution region based on the binarized class activation map; and
generating a diagnostic visualization information based on the probability of malignancy, the class activation map, the ultrasound image data and the contribution region,
wherein the deep learning model is a model trained based on classification information entirely of ultrasound image data for training without information related to segmentation of the ultrasound image data for training.
2 . The method of claim 1 , wherein the classification information includes at least one of normal, benign, or malignant.
3 . The method of claim 1 , wherein the diagnostic visualization information includes, for each pixel of the ultrasound image data, information in which the degree of each pixel contributing to generate the classification information is visualized.
4 . The method of claim 1 , wherein the deep learning model is a model trained with operations of:
preprocessing one or more ultrasound image data for training;
labeling the preprocessed ultrasound image data with classification information to form a training data set; and
training the deep learning model to output classification information for each of the ultrasound image data included in the training data set by using the training data set.
5 . The method of claim 4 , wherein the preprocessing of the one or more ultrasound image data for training includes:
removing personal information of a patient from the one or more ultrasound image data for training;
trimming edges of the ultrasound image data in which the personal information is removed; and
resizing the ultrasound image data in which the edges are trimmed.
6 . The method of claim 4 , further comprising:
performing data augmentation on the preprocessed ultrasound image data for training,
wherein the performing of the data augmentation on the preprocessed ultrasound image data for training includes:
performing at least one of a geometric transformation or an optical transformation on the preprocessed ultrasound image data for training to generate at least one new image data; and
including the at least one new image data in the training data set.
7 . The method of claim 6 , wherein the geometric transformation includes at least one of a horizontal flip, a vertical flip, a rotation, a scaling, or a translocation.
8 . The method of claim 6 , wherein the optical transformation includes at least one of deletion, speckle noise processing, sharpening, or Gaussian blurring.
9 . The method of claim 1 , wherein the generating of the one or more feature maps based on the ultrasound image data by using the deep learning model includes:
generating the one or more feature maps based on input of the ultrasound image data into a convolutional neural network including one or more convolutional layers.
10 . The method of claim 1 , wherein the computing of the score for each class of the ultrasound image data based on the one or more feature maps by using the deep learning model includes:
computing the score for each class of the ultrasound image data based on input of the one or more feature maps to a global average pooling layer.
11 . The method of claim 1 , wherein the generating the class activation map for each class based on the one or more feature maps includes:
generating the class activation map for each class based on the one or more feature maps and a weight of the deep learning model.
12 . The method of claim 11 , wherein the weight of the deep learning model includes:
a weight trained by the deep learning model in the process of generating the classification information.
13 . A computer program stored in a non-transitory computer-readable storage medium including operations for diagnosing an ultrasound image by a computing device, the operations comprising:
acquiring ultrasound image data;
generating one or more feature maps based on the ultrasound image data by using a deep learning model;
computing a score for each class of the ultrasound image data based on the one or more feature maps by using the deep learning model;
generating classification information based on the score for each class;
computing a probability of malignancy based on the classification information;
generating a class activation map for each class based on the one or more feature maps;
normalizing the class activation map;
binarizing the normalized class activation map by using a preset threshold;
determining a contribution region based on the binarized class activation map; and
generating a diagnostic visualization information based on the probability of malignancy, the class activation map, the ultrasound image data and the contribution region,
wherein the deep learning model is a model trained based on classification information of ultrasound image data for training, without information related to segmentation of the ultrasound image data for training.
14 . A computing device, comprising:
a processor including one or more cores; and
a memory,
wherein the processor:
acquires ultrasound image data,
generates one or more feature maps based on the ultrasound image data by using a deep learning model,
computes a score for each class of the ultrasound image data based on the one or more feature maps by using the deep learning model,
generates classification information based on the score for each class,
computes a probability of malignancy based on the classification information,
generates a class activation map for each class based on the one or more feature maps,
normalizes the class activation map,
binarizes the normalized class activation map by using a preset threshold,
determines a contribution region based on the binarized class activation map, and
generates a diagnostic visualization information based on the probability of malignancy, the class activation map, the ultrasound image data and the contribution region,
wherein the deep learning model is a model trained based on classification information of ultrasound image data for training, without information related to segmentation of the ultrasound image data for training.