Image processing system, training method for training device, and storage medium
An image processing system includes a processor configured to acquire, as a processing target image, an in-vivo image, operate in accordance with a trained model, and output a recognition result representing a result of recognition of a region of interest in the processing target image. The trained model is trained by having undergone pre-training using a first image group including images captured in a first observation method, and having undergone, after the pre-training, fine-tuning that uses a second image group including images captured in a second observation method, as well as that uses ground truth regarding the region of interest included in the second image group. The first observation method is an observation method using normal light as illumination light, and the second observation method is an observation method using special light as the illumination light or an observation method in which a pigment has been dispersed onto the subject.
1. An image processing system comprising:
a processor including hardware, the processor being configured to:
acquire, as a processing target image, an in-vivo image captured by an endoscope imaging device,
operate in accordance with a trained model, and
output a recognition result representing a result of recognition of a region of interest in the processing target image,
the trained model being trained by having undergone pre-training that uses a first image group including images captured in a first observation method, and then having undergone, after the pre-training, fine-tuning that uses a second image group including images captured in a second observation method, as well as that uses ground truth regarding the region of interest included in the second image group,
the first observation method being an observation method that uses normal light as illumination light,
the second observation method being an observation method that uses special light as the illumination light or an observation method in which a pigment has been sprayed onto a subject,
the second image group including at least one image captured with the region of interest,
the second image group being lesser in number of images than the first image group;
the ground truth being data including at least one of:
detection data regarding at least one of whether the region of interest is present in an image, and, if any, a position, a size, and a shape of the region of interest, and
local classification data regarding a degree of malignancy of the region of interest in the image, and
the recognition result being a result including at least one of:
a detection result regarding at least one of whether the region of interest is present in the image, and, if any, the position, the size, and the shape of the region of interest, and
a local classification result regarding the degree of malignancy of the region of interest in the image.
2. The image processing system as defined in claim 1 , wherein the region of interest is a polyp.
3. The image processing system as defined in claim 1 ,
wherein the processor obtains, based on the trained model, a detection score representing a probability of the detection result and a local classification score representing a probability of the local classification result, and
wherein the processor outputs, in a case where the detection score is greater than a given detection threshold, the detection score, and outputs, in a case where the local classification score is greater than a given classification threshold, the local classification result.
4. The image processing system as defined in claim 1 ,
wherein the processor determines, based on the detection result, whether or not a condition regarding at least one of the size of the region of interest, blur, or motion blur is satisfied,
wherein the processor outputs, in a case where the condition is satisfied, the detection result and the local classification result, and
wherein the processor outputs, in a case where the condition is not satisfied, the detection result without outputting the local classification result.
5. The image processing system as defined in claim 1 , wherein the processor performs an integration process of integrating a first recognition result obtained by operating in accordance with a first trained model and a second recognition result obtained by operating in accordance with a second trained model, and outputs a result of the integration process as the recognition result.
6. The image processing system as defined in claim 1 ,
wherein the processor operates in accordance with a first trained model to obtain, as the detection result, a first region representing a region corresponding to the region of interest, and operates in accordance with a second trained model to obtain, as the detection result, a second region representing a region corresponding to the region of interest, and
wherein the processor determines, based on a degree of overlap between the first region and the second region, whether or not the first region and the second region correspond to an identical region of interest.
7. The image processing system as defined in claim 6 ,
wherein each of the first region and the second region is a detection frame surrounding the region of interest in the image, and
wherein the processor calculates an Intersection over Union (IoU) representing the degree of overlap between the detection frame corresponding to the first region and the detection frame corresponding to the second region, and determines, in a case where the IoU is equal to or greater than a predetermined threshold, that the first region and the second region correspond to the identical region of interest.
8. The image processing system as defined in claim 6 , wherein the processor performs, when determining that the first region and the second region correspond to the identical region of interest in a case where the local classification result corresponding to the first region and the local classification result corresponding to the second region are different from each other, a process of selecting the local classification result having a higher score as a local classification score representing a probability of the local classification result.
9. The image processing system as defined in claim 1 , wherein the trained model comprises a convolutional neural network (CNN).
10. The image processing system as defined in claim 1 , wherein the trained model includes
a trained model for the second observation method that is trained by having undergone pre-training that uses the first image group, and then having undergone fine-tuning that uses the second image group, and
a trained model for the first observation method that is trained by having undergone pre-training that uses the first image group, and then having undergone fine-tuning that uses a third image group including images captured in the first observation method.
11. The image processing system as defined in claim 10 ,
wherein the first image group includes an image in which each image captured in the first observation method is provided with, as the ground truth, the detection data regarding at least one of whether the region of interest is present, and, if any, the position, the size, and the shape of the region of interest, and
wherein the third image group includes an image in which each image captured in the first observation method is provided with, as the ground truth, the detection data and local classification data representing the degree of malignancy of the region of interest.
12. The image processing system as defined in claim 1 ,
wherein the trained model includes a feature amount extraction layer that outputs a feature amount based on the processing target image, a detection layer that outputs information indicating the detection result based on the feature amount, and a local classification layer that outputs the local classification result based on the feature amount,
wherein the trained model is trained in a weight coefficient of the feature amount extraction layer and a weight coefficient of the detection layer by having undergone the pre-training that uses the first image group including images provided with the detection data as the ground truth, and
wherein the trained model is trained in the weight coefficient of the feature amount extraction layer, the weight coefficient of the detection layer, and a weight coefficient of the local classification layer by having undergone the fine-tuning that uses the second image group including images provided with the detection data and the local classification data as the ground truth.
13. A training device comprising:
a processor including hardware, the processor being configured to:
acquire, a first image group including images captured in a first observation method and a second image group including images captured in a second observation method,
generate a trained model that outputs, when a processing target image is input to the trained model, a recognition result representing a result of recognition of a region of interest in the processing target image by having undergone pre-training that uses the first image group, and then having undergone, after the pre-training, fine-tuning that uses the second image group, as well as that uses ground truth regarding the region of interest included in the second image group,
the first observation method being an observation method that uses normal light as illumination light,
the second observation method being an observation method that uses special light as the illumination light or an observation method in which a pigment has been sprayed onto a subject,
the second image group including at least one image captured with the region of interest,
the second image group being lesser in number of images than the first image group;
the ground truth being data including at least one of:
detection data regarding at least one of whether the region of interest is present in an image, and, if any, a position, a size, and a shape of the region of interest, and
local classification data regarding a degree of malignancy of the region of interest in the image, and
the recognition result being a result including at least one of:
a detection result regarding at least one of whether the region of interest is present in the image, and, if any, the position, the size, and the shape of the region of interest, and
a local classification result regarding the degree of malignancy of the region of interest in the image.
14. A training method comprising:
acquiring a first image group including images captured in a first observation method and a second image group including images captured in a second observation method;
generating a trained model that outputs, when a processing target image is input to the trained model, a recognition result representing a result of recognition of a region of interest in the processing target image by having undergone pre-training that uses the first image group, and then having undergone, after the pre-training, fine-tuning that uses the second image group, as well as that uses ground truth regarding the region of interest included in the second image group,
the first observation method being an observation method that uses normal light as illumination light,
the second observation method being an observation method that uses special light as the illumination light or an observation method in which a pigment has been sprayed onto a subject,
the second image group including at least one image captured with the region of interest,
the second image group being lesser in number of images than the first image group;
the ground truth being data including at least one of:
detection data regarding at least one of whether the region of interest is present in an image, and, if any, a position, a size, and a shape of the region of interest, and
local classification data regarding a degree of malignancy of the region of interest in the image, and
the recognition result being a result including at least one of:
a detection result regarding at least one of whether the region of interest is present in the image, and, if any, the position, the size, and the shape of the region of interest, and
a local classification result regarding the degree of malignancy of the region of interest in the image.
15. A computer readable non-transitory storage medium that stores a program that causes a computer to execute steps of:
acquiring, as a processing target image, an in-vivo image captured by an endoscope imaging device;
operating in accordance with a trained model; and
outputting a recognition result representing a result of recognition of a region of interest in the processing target image,
the trained model being trained by having undergone pre-training that uses a first image group including images captured in a first observation method, and then having undergone, after the pre-training, fine-tuning that uses a second image group including images captured in a second observation method, as well as that uses ground truth regarding the region of interest included in the second image group,
the first observation method being an observation method that uses normal light as illumination light,
the second observation method being an observation method that uses special light as the illumination light or an observation method in which a pigment has been sprayed onto a subject,
the second image group including at least one image captured with the region of interest,
the second image group being lesser in number of images than the first image group;
the ground truth being data including at least one of:
detection data regarding at least one of whether the region of interest is present in an image, and, if any, a position, a size, and a shape of the region of interest, and
local classification data regarding a degree of malignancy of the region of interest in the image, and
the recognition result being a result including at least one of:
a detection result regarding at least one of whether the region of interest is present in the image, and, if any, the position, the size, and the shape of the region of interest, and
a local classification result regarding the degree of malignancy of the region of interest in the image.