Endoscopic image learning device, endoscopic image learning method, endoscopic image learning program, and endoscopic image recognition device
An object is to provide an endoscopic image learning device, an endoscopic image learning method, an endoscopic image learning program, and an endoscopic image recognition device that appropriately learn a learning model for image recognition for recognizing an endoscopic image in which a treatment tool for an endoscope appears. The object is achieved by an endoscopic image learning device including an image generation unit and a machine learning unit. The image generation unit generates a superimposed image where a foreground image in which a treatment tool for an endoscope is extracted is superimposed on a background-endoscopic image serving as a background of the foreground image, and the machine learning unit performs the learning of a learning model for image recognition using the superimposed image.
1. An endoscope system comprising:
an endoscope including a distal end part having an image pickup element and a forceps outlet from which a treatment tool is protruded; and
one or more processors configured to:
acquire an endoscopic image through the image pickup element; and
perform a recognition process of the endoscopic image using a learning model that is learned by a machine learning process using a plurality of superimposed images in which a foreground image including an image of the treatment tool is superimposed on a background image that is an image of a living body,
wherein at least one of followings: presentation of the treatment tool in the endoscopic image, a type of the treatment tool in the endoscopic image, and an area of the treatment tool in the endoscopic image is recognized in the recognition process.
2. The endoscope system according to claim 1 , further comprising a display,
wherein the one or more processors are further configured to display a result of the recognition on the display.
3. The endoscope system according to claim 2 , wherein the one or more processors are further configured to display the endoscopic image.
4. The endoscope system according to claim 1 , wherein:
the endoscope includes an insertion part having a bendable part at a distal end side of the insertion part; and
the distal end part is positioned at a distal end side of the bendable part.
5. The endoscope system according to claim 1 , wherein the foreground image is processed by a specific processing step.
6. The endoscope system according to claim 5 , wherein the specific processing step is at least one of an affine transformation processing step, a color conversion processing step, or a noise application processing step.
7. The endoscope system according to claim 1 , wherein the foreground image is generated by cutting out the image of the treatment tool from a foreground-material image including the image of the treatment tool.
8. The endoscope system according to claim 7 , wherein the foreground-material image is an endoscopic image that is picked up in a case where the treatment tool is utilized in the endoscope.
9. The endoscope system according to claim 7 , wherein the foreground-material image is an image other than an endoscopic image.
10. The endoscope system according to claim 1 , wherein the learning model includes a convolution neural network.
11. The endoscope system according to claim 1 , wherein the one or more processors are further configured to:
generate the superimposed images; and
perform the learning process of the learning model through utilizing the superimposed images.
12. The endoscope system according to claim 11 , wherein the specific processing step is at least one of an affine transformation processing step, a color conversion processing step, or s noise application processing step.
13. The endoscope system according to claim 11 , wherein the one or more processors are configured to perform the learning process through utilizing a convolution neural network.
14. The endoscope system according to claim 1 , wherein the learning model is learned using a label assigned to the treatment tool in the superimposed images.
15. The endoscope system according to claim 1 , wherein the learning model performs segmentation to distinguish between the area of the treatment tool and areas other than the area of the treatment tool.
16. The endoscope system according to claim 1 , wherein the superimposed images include a plurality of images with different positions of the treatment tool in the image.
17. The endoscope system according to claim 1 , wherein the superimposed images include an image obtained by superimposing different types of treatment tools.
18. The endoscope system according to claim 17 , wherein the one or more processors are configured to recognize multiple types of treatment tools using the learning model.
19. The endoscope system according to claim 18 , wherein:
the endoscope includes an insertion part having a bendable part at a distal end side of the insertion part; and
the distal end part is positioned at a distal end side of the bendable part.