Learning method, learning system, learned model, program, and super resolution image generating device
Provided are a learning method and a learning system of a generative model, a program, a learned model, and a super resolution image generating device that can handle input data of any size and can suppress the amount of calculation at the time of image generation. A learning method according to an embodiment of the present disclosure is a learning method for performing machine learning of a generative model that estimates, from a first image, a second image including higher resolution image information than the first image, the method comprising using a generative adversarial network including a generator which is the generative model and a discriminator which is an identification model that identifies whether provided data is data of a correct image for learning or data derived from an output from the generator and implementing a self-attention mechanism only in a network of the discriminator among the generator and the discriminator.
1. A learning method for performing machine learning of a generative model that estimates, from a first image, a second image including higher resolution image information than the first image, the method comprising:
using a generative adversarial network including a generator which is the generative model and a discriminator which is an identification model that identifies whether provided data is data of a correct image for learning or data derived from an output from the generator;
using, as learning data, a first learning image including first resolution information having a lower resolution than the second image and a second learning image including second resolution information having a higher resolution than the first learning image, the second learning image being the correct image corresponding to the first learning image;
providing only the first learning image, among the first learning image and the second learning image, to be input to the generator; and
implementing a self-attention mechanism only in a network of the discriminator among the generator and the discriminator, wherein the self-attention mechanism being applied in the network of the discriminator is a part or all of a plurality of convolutional layers of the discriminator.
2. The learning method according to claim 1 ,
wherein each network of the generator and the discriminator is a convolutional neural network.
3. The learning method according to claim 1 ,
wherein the first image is a three-dimensional tomographic image, and
the second image has at least a higher resolution than the first image in a slice thickness direction of the three-dimensional tomographic image.
4. The learning method according to claim 1 ,
wherein the second learning image is an image acquired by using a computed tomography device, and
the first learning image is an image generated by image processing based on the second learning image.
5. The learning method according to claim 4 ,
wherein the image processing of generating the first learning image from the second learning image includes down-sampling processing of the second learning image.
6. The learning method according to claim 5 ,
wherein the image processing of generating the first learning image from the second learning image includes up-sampling processing of an image obtained by the down-sampling processing by performing interpolation processing.
7. The learning method according to claim 4 ,
wherein the image processing of generating the first learning image from the second learning image includes smoothing processing using a Gaussian filter.
8. The learning method according to claim 1 ,
wherein the first learning images and the second learning images in a plurality of types of the learning data used for the machine learning have the same size, respectively.
9. The learning method according to claim 1 ,
wherein the second image is a high-frequency component image indicating information of a high-frequency component, and
the generator estimates a high-frequency component required to increase a resolution of an input image and outputs the high-frequency component image indicating the information of the high-frequency component.
10. The learning method according to claim 9 , further comprising adding the high-frequency component image output from the generator and the image input to the generator,
wherein a virtual second image obtained by the adding is provided to be input to the discriminator.
11. A non-transitory computer-readable recording medium that causes a computer to execute the learning method according to claim 1 in a case in which a command stored in the recording medium is read by the computer.
12. A learned model that is learned by performing the learning method according to claim 1 , the learned model being the generative model that estimates, from the first image, the second image including higher resolution image information than the first image.
13. A super resolution image generating device comprising the generative model which is a learned model that is learned by performing the learning method according to claim 1 ,
wherein the super resolution image generating device generates, from a third image which is input, a fourth image including higher resolution image information than the third image.
14. The super resolution image generating device according to claim 13 ,
wherein an image size of the third image is different from an image size of the first learning image.
15. The super resolution image generating device according to claim 13 , further comprising:
a first interpolation processing unit that performs interpolation processing on the third image to generate an interpolation image; and
a first addition unit that adds the interpolation image and a high-frequency component generated by the generative model,
wherein the interpolation image is input to the generative model, and
the generative model generates the high-frequency component required to increase a resolution of the interpolation image.
16. A learning system for performing machine learning of a generative model that estimates, from a first image, a second image including higher resolution image information than the first image, the system comprising a generative adversarial network including a generator which is the generative model and a discriminator which is an identification model that identifies whether provided data is data of a correct image for learning or data derived from an output from the generator,
wherein a self-attention mechanism is implemented only in a network of the discriminator among the generator and the discriminator, wherein the self-attention mechanism being applied in the network of the discriminator is a part or all of a plurality of convolutional layers of the discriminator,
a first learning image including first resolution information having a lower resolution than the second image and a second learning image including second resolution information having a higher resolution than the first learning image, the second learning image being the correct image corresponding to the first learning image are used as learning data, and
only the first learning image among the first learning image and the second learning image is provided to be input to the generator and learning of the generative adversarial network is performed.
17. The learning system according to claim 16 , further comprising a learning data generating unit that generates the learning data,
wherein the learning data generating unit includes
a fixed-size region cutout unit that cuts out a fixed-size region from an original image including the second resolution information, and
a down-sampling processing unit that performs down-sampling of an image of the fixed-size region cut out by the fixed-size region cutout unit,
the image of the fixed-size region cut out by the fixed-size region cutout unit is used as the second learning image, and
the first learning image is generated by performing down-sampling processing on the second learning image.
18. The learning system according to claim 17 ,
wherein the learning data generating unit further includes
a second interpolation processing unit that performs interpolation processing on an image obtained by the down-sampling processing, and
a smoothing processing unit that performs smoothing by using a Gaussian filter.
19. The learning system according to claim 16 ,
wherein the generator is configured to estimate a high-frequency component required to increase a resolution of an input image and output the high-frequency component image indicating the information of the high-frequency component, and
the learning system further comprises a second addition unit that adds the high-frequency component image output from the generator and the image input to the generator.