Image processing device, image processing method, and storage medium
The image processing device 1 X includes a classification means 31 X, an image selection means 33 X, and a region extraction means 34 X. The classification means 31 X classifies each captured image acquired in time series by photographing an inspection target by a photographing unit provided in an endoscope, according to whether or not the each captured image includes an attention part to be paid attention to. The image selection means 33 X selects a target image to be subjected to extraction of a region of the attention part from the each captured image, based on a result of the classification. The region extraction means 34 X extracts the area of the attention part from the target image. The image processing device can support decision-making based on based on images captured by the endoscope.
1 . An image processing device comprising:
at least one memory configured to store instructions; and
at least one processor configured to execute the instructions to:
classify each captured image acquired in time series by photographing an inspection target by a photographing unit provided in an endoscope, according to whether or not the each captured image includes an attention part to be paid attention to based on a classification model, wherein the classification model is trained by machine learning to output information regarding presence or absence of the attention part in the image in case where an image is inputted to the classification model, and wherein the classification model is configured to output a confidence score regarding the presence or absence of the attention part;
in a case where a condition regarding the confidence score is satisfied consecutively for a predetermined number of captured images in the time series, select a target image to be subjected to extraction of a region of the attention part from the predetermined number of the captured images, based on the confidence score; and
extract the region of the attention part from the target image.
2 . The image processing device according to claim 1 ,
wherein the at least one processor is configured to execute the instructions, in a case where a condition regarding the result of the classification is satisfied consecutively for a predetermined number of captured images in the time series, to select the target image from the predetermined number of the captured images.
3 . The image processing device according to claim 2 ,
wherein the at least one processor is configured to execute the instructions to change the predetermined number after selecting the target image, during a time segment in which consecutive captured images classified as presence of the attention part are acquired.
4 . The image processing device according to claim 1 ,
wherein the at least one processor is configured to execute the instructions to perform feature extraction of the each captured image, and
wherein the at least one processor is configured to execute the instructions to select the target image based on feature vectors obtained by the feature extraction.
5 . The image processing device according to claim 4 ,
wherein the at least one processor is configured to execute the instructions, in a case where a condition regarding a similarity of a predetermined number of consecutive captured images in the time series is satisfied, to select the target image from the predetermined number of the captured images, the similarity being calculated by the feature vectors.
6 . The image processing device according to claim 1 ,
wherein the at least one processor is configured to execute the instructions to extract the region of the attention part from the target image, based on a segmentation model that is a model configured to output, when an image is inputted to the model, information regarding the region of the attention part in the inputted image.
7 . The image processing device according to claim 1 ,
wherein the at least one processor is configured to further execute the instructions to display information regarding the region of the attention part on a display device.
8 . The image processing device according to claim 7 ,
wherein, in a case where plural target images are selected during a time segment in which consecutive captured images classified as including the attention part are acquired, the at least one processor is configured to execute the instructions to determine, based on area of the region of the attention part, the target image to be used for display the information regarding the region of the attention part.
9 . The image processing device according to claim 7 ,
wherein the at least one processor is configured to execute the instructions to stop updating the display of the information regarding the region of the attention part, based on an external input.
10 . An image processing method executed by a computer, the image processing method comprising:
classifying each captured image acquired in time series by photographing an inspection target by a photographing unit provided in an endoscope, according to whether or not the each captured image includes an attention part to be paid attention to based on a classification model, wherein the classification model is trained by machine learning to output information regarding presence or absence of the attention part in the image in case where an image is inputted to the classification model, and wherein the classification model is configured to output a confidence score regarding the presence or absence of the attention part;
in a case where a condition regarding the confidence score is satisfied consecutively for a predetermined number of captured images in the time series, selecting a target image to be subjected to extraction of a region of the attention part from the predetermined number of the captured images, based on the confidence score; and
extracting the region of the attention part from the target image.
11 . The image processing method according to claim 10 , the image processing method comprising:
in a case where a condition regarding the result of the classification is satisfied consecutively for a predetermined number of captured images in the time series, selecting the target image from the predetermined number of the captured images.
12 . The image processing method according to claim 11 , the image processing method comprising:
changing the predetermined number after selecting the target image, during a time segment in which consecutive captured images classified as presence of the attention part are acquired.
13 . The image processing method according to claim 10 , the image processing method comprising:
performing feature extraction of the each captured image, and
selecting the target image based on feature vectors obtained by the feature extraction.
14 . The image processing method according to claim 13 , the image processing method comprising:
in a case where a condition regarding a similarity of a predetermined number of consecutive captured images in the time series is satisfied, selecting the target image from the predetermined number of the captured images, the similarity being calculated by the feature vectors.
15 . The image processing method according to claim 10 , the image processing method comprising:
extracting the region of the attention part from the target image, based on a segmentation model that is a model configured to output, when an image is inputted to the model, information regarding the region of the attention part in the inputted image.
16 . The image processing method according to claim 10 , the image processing method comprising:
displaying information regarding the region of the attention part on a display device.
17 . The image processing method according to claim 16 , the image processing method comprising:
in a case where plural target images are selected during a time segment in which consecutive captured images classified as including the attention part are acquired, determining, based on area of the region of the attention part, the target image to be used for display the information regarding the region of the attention part.
18 . The image processing method according to claim 16 , the image processing method comprising:
stopping updating the display of the information regarding the region of the attention part, based on an external input.
19 . A non-transitory computer readable storage medium storing a program executed by a computer, the program causing the computer to:
classify each captured image acquired in time series by photographing an inspection target by a photographing unit provided in an endoscope, according to whether or not the each captured image includes an attention part to be paid attention to based on a classification model, wherein the classification model is trained by machine learning to output information regarding presence or absence of the attention part in the image in case where an image is inputted to the classification model, and wherein the classification model is configured to output a confidence score regarding the presence or absence of the attention part;
in a case where a condition regarding the confidence score is satisfied consecutively for a predetermined number of captured images in the time series, select a target image to be subjected to extraction of a region of the attention part from the predetermined number of the captured images, based on the confidence score; and
extract the region of the attention part from the target image.