Image processing method, image processing apparatus, learning method, learning apparatus, and storage medium

1. A method for processing an image, the method comprising:

converting an input low resolution image into a first feature map and generating a first intermediate image and a second intermediate image from the first feature map, the first intermediate image and the second intermediate image each having resolution higher than resolution of the low resolution image, by using a generator which is a machine learning model; and

generating an estimated image having higher resolution than the resolution of the low resolution image based on the first intermediate image and the second intermediate image,

wherein the generator includes an upsampling layer and one or more linear sum layers,

wherein the upsampling layer upsamples an input to the upsampling layer,

wherein each of the linear sum layers outputs a linear sum of an input to the linear sum layer and weights of the linear sum layer, and

wherein a number of the linear sum layers is less on an output side of a first upsampling layer which is closest to the output side of the generator among upsampling layers included in the generator than on an input side of the first upsampling layer.

2. The method according to claim 1 , wherein the first intermediate image includes fewer high frequency components than the second intermediate image.

3. The method according to claim 1 , wherein the first intermediate image and the second intermediate image each include more pixels than the low resolution image.

4. The method according to claim 1 ,

wherein the generator generates the first intermediate image by summing a first residual component generated from the first feature map and the low resolution image, and

wherein the generator generates the second intermediate image by summing a second residual component generated from the first feature map and the low resolution image.

5. The method according to claim 4 , wherein the low resolution image is upsampled before being summed with the first residual component and the second residual component so that a number of pixels of the low resolution image matches a number of pixels of each of the first residual component and the second residual component.

6. The method according to claim 1 ,

wherein the generator includes a plurality of linear sum layers,

wherein each of the linear sum layers outputs a linear sum of an input to the linear sum layer and weights of the linear sum layer, and

wherein an output of at least one of the plurality of linear sum layers is concatenated to an input of the linear sum layer in a channel direction.

7. The method according to claim 1 ,

wherein the generator includes a plurality of linear sum layers,

wherein each of the linear sum layers outputs a linear sum of an input to the linear sum layer and weights of the linear sum layer, and

wherein outputs of at least half of the plurality of linear sum layers are concatenated to inputs of the respective linear sum layers in a channel direction.

8. The method according to claim 1 ,

wherein the generator includes a plurality of residual blocks each including a plurality of linear sum layers,

wherein each of the linear sum layers outputs a linear sum of an input to the linear sum layer and weights of the linear sum layer, and

wherein each of the residual blocks outputs a sum of an input to the residual block and a result of calculation on the input by the residual block.

9. The method according to claim 8 , wherein the residual block includes two or more activation functions which perform non-linear transformation.

10. The method according to claim 1 ,

wherein the generator includes a plurality of residual blocks each including a plurality of linear sum layers,

wherein each of the linear sum layers outputs a linear sum of an input to the linear sum layer and weights of the linear sum layer,

wherein each of the residual blocks outputs a sum of an input to the residual block and a result of calculation on the input by the residual block, and

wherein half or less of the plurality of residual blocks each includes a batch regularization layer which performs batch regularization.

11. The method according to claim 1 , wherein the estimated image is generated by a weighted mean of the first intermediate image and the second intermediate image.

12. The method according to claim 1 ,

wherein the generator is a machine learning model which is trained using a loss including a first loss and a second loss,

wherein the first loss is defined based on a difference between the first intermediate image and a ground truth image and a difference between the second intermediate image and the ground truth image, and

wherein the second loss is defined based on a discrimination result of whether the second intermediate image is an image generated by the generator.

13. The method according to claim 12 ,

wherein the loss further includes a third loss which is defined based on a discrimination result of whether the first intermediate image is an image generated by the generator, and

wherein a ratio of the third loss to the loss is smaller than a ratio of the second loss to the loss.

14. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the method according to claim 1 .

15. A method for learning, the method comprising:

acquiring a low resolution image and a high resolution image corresponding to the low resolution image;

converting the low resolution image into a first feature map and generating a first intermediate image and a second intermediate image from the first feature map, the first intermediate image and the second intermediate image each having resolution higher than resolution of the low resolution image, by using a generator;

discriminating, by using a discriminator, whether an image input to the discriminator is an image generated by the generator; and

training the generator based on a loss including a first loss and a second loss,

wherein the first loss is defined based on a difference between the high resolution image and the first intermediate image and a difference between the high resolution image and the second intermediate image, and

wherein the second loss is defined based on a discrimination output of the discriminator in a case where the second intermediate image is input to the discriminator.

16. The method according to claim 15 ,

wherein the loss further includes a third loss which is defined based on a discrimination output of the discriminator in a case where the first intermediate image is input to the discriminator, and

wherein a ratio of the third loss to the loss is smaller than a ratio of the second loss to the loss.

17. The method according to claim 15 , further comprising converting an image into a feature map using a feature extractor,

wherein, in the training the generator, the generator is trained based on a fourth loss based on a difference between a second feature map converted from the high resolution image by the feature extractor and a third feature map converted from the second intermediate image.

18. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the method according to claim 15 .

19. An image processing apparatus comprising a processing unit configured to generate an estimated image obtained by performing resolution enhancement on an input low resolution image,

wherein the processing unit executes:

converting the input low resolution image into a first feature map and generating a first intermediate image and a second intermediate image from the first feature map, the first intermediate image and the second intermediate image each having resolution higher than resolution of the low resolution image, by using a generator which is a machine learning model; and

generating an estimated image having higher resolution than the resolution of the low resolution image based on the first intermediate image and the second intermediate image,

wherein the generator includes an upsampling layer and one or more linear sum layers,

wherein the upsampling layer upsamples an input to the upsampling layer,

wherein each of the linear sum layers outputs a linear sum of an input to the linear sum layer and weights of the linear sum layer, and

wherein a number of the linear sum layers is less on an output side of a first upsampling layer which is closest to the output side of the generator among upsampling layers included in the generator than on an input side of the first upsampling layer.

20. A learning apparatus comprising a processing unit configured to train a machine learning model,

wherein the processing unit executes:

acquiring a low resolution image and a high resolution image corresponding to the low resolution image;

converting the low resolution image into a first feature map and generating a first intermediate image and a second intermediate image from the first feature map, the first intermediate image and the second intermediate image each having resolution higher than resolution of the low resolution image, by using a generator;

discriminating, by using a discriminator, whether an image input to the discriminator is an image generated by the generator; and

training the generator based on a loss including a first loss and a second loss,

wherein the first loss is defined based on a difference between the high resolution image and the first intermediate image and a difference between the high resolution image and the second intermediate image, and

wherein the second loss is defined based on a discrimination output of the discriminator in a case where the second intermediate image is input to the discriminator.