Image processing method, image processing apparatus, learning apparatus, manufacturing method of learned model, and storage medium

1 . An image processing method comprising:

generating a first grayscale image based on a first color image, the first color image being acquired by image capturing that uses an image sensor;

generating, by dividing the first grayscale image, a plurality of second grayscale images where each has less number of pixels than that of the first grayscale image;

generating a plurality of third grayscale images upscaled by inputting the plurality of second grayscale images and an ISO sensitivity of the image sensor in the image capturing to a machine learning model, wherein in the generating the plurality of third grayscale images, the plurality of second grayscale images and the ISO sensitivity are concatenated in a channel direction and inputted to the machine learning model; and

generating a second color image based on the plurality of third grayscale images, wherein the second color image has more pixels than the first color image.

2 . The image processing method according to claim 1 , wherein a number of pixels of each of the plurality of second grayscale images are the same as each other.

3 . The image processing method according to claim 1 , further comprising generating a fourth grayscale image by combining the plurality of third grayscale images.

4 . The image processing method according to claim 3 , wherein a number of pixels of the fourth grayscale image and a sum of numbers of pixels of the plurality of third grayscale images are equal.

5 . The image processing method according to claim 3 , further comprising:

generating the first grayscale image and a plurality of first chrominance images from the first color image; and

generating the second color image based on the fourth grayscale image and the plurality of first chrominance images.

6 . The image processing method according to claim 5 , further comprising generating a plurality of second chrominance images by upscaling the plurality of first chrominance images,

wherein the second color image is generated based on the fourth grayscale image and the plurality of second chrominance images.

7 . The image processing method according to claim 6 , wherein a number of pixels of each of the plurality of second chrominance images is the same as the number of pixels of the fourth grayscale image.

8 . The image processing method according to claim 5 ,

wherein the first color image is acquired by image capturing using an optical system and the image sensor, and

wherein the generating the plurality of third grayscale images generates the plurality of third grayscale images upscaled by inputting the plurality of second grayscale images and an image capturing condition in the image capturing to the machine learning model.

9 . The image processing method according to claim 1 , further comprising inputting an image capturing condition including at least one of a pixel pitch of the image sensor or a type of an optical low-pass filter of the optical system to the machine learning model.

10 . The image processing method according to claim 8 , wherein the image capturing condition includes at least one of noise removal strength, sharpness strength, or an image compression rate.

11 . A non-transitory computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 1 .

12 . An image processing apparatus comprising:

at least one memory storing instructions; and

at least one processor that executes the instructions to:

generate a first grayscale image based on a first color image, the first color image being acquired by image capturing that uses an image sensor;

generate, by dividing the first grayscale image, a plurality of second grayscale images each including a number of pixels smaller than a number of pixels of the first grayscale image;

generate a plurality of third grayscale images upscaled by inputting the plurality of second grayscale images and an ISO sensitivity of the image sensor in the image capturing to a machine learning model, wherein in the generating the plurality of third grayscale images, the plurality of second grayscale images and the ISO sensitivity are concatenated in a channel direction and inputted to the machine learning model; and

generate a second color image based on the plurality of third grayscale images, wherein the second color image has more pixels than the first color image.

13 . A learning apparatus comprising:

at least one memory storing instructions; and

at least one processor that executes the instructions to:

acquire a first training image and a first ground truth image;

generate, by dividing the first training image and the first ground truth image, a plurality of second training images each including a number of pixels smaller than a number of pixels of the first training image, and a plurality of second ground truth images each including a number of pixels smaller than a number of pixels of the first ground truth image;

generate a plurality of estimated images upscaled by inputting the plurality of second training images to a machine learning model; and

update a weight of a neural network based on the plurality of estimated images and the plurality of second ground truth images;

generate a first grayscale image based on a first color image, the first color image being acquired by image capturing that uses an image sensor;

generate, by dividing the first grayscale image, a plurality of second grayscale images each including a number of pixels smaller than a number of pixels of the first grayscale image;

generate a plurality of third grayscale images upscaled by inputting the plurality of second grayscale images and an ISO sensitivity of the image sensor in the image capturing to the machine learning model, wherein in the generating the plurality of third grayscale images, the plurality of second grayscale images and the ISO sensitivity are concatenated in a channel direction and inputted to the machine learning model; and

generate a second color image based on the plurality of third grayscale images, wherein the second color image has more pixels than the first color image.

14 . A manufacturing method of a learned model, the manufacturing method comprising:

acquiring a first training image and a first ground truth image;

generating, by dividing the first training image and the first ground truth image, a plurality of second training images where each has less number of pixels than that of the first training image, and a plurality of second ground truth images where each has less number of pixels than that of the first ground truth image;

generating a plurality of estimated images upscaled by inputting the plurality of second training images to a machine learning model;

updating a weight of a neural network based on the plurality of estimated images and the plurality of second ground truth images;

generating a first grayscale image based on a first color image, the first color image being acquired by image capturing that uses an image sensor;

generating, by dividing the first grayscale image, a plurality of second grayscale images each including a number of pixels smaller than a number of pixels of the first grayscale image;

generating a plurality of third grayscale images upscaled by inputting the plurality of second grayscale images and an ISO sensitivity of the image sensor in the image capturing to the machine learning model, wherein in the generating the plurality of third grayscale images, the plurality of second grayscale images and the ISO sensitivity are concatenated in a channel direction and inputted to the machine learning model; and

generating a second color image based on the plurality of third grayscale images, wherein the second color image has more pixels than the first color image.

15 . A non-transitory computer-readable storage medium storing a program for causing a computer to execute the manufacturing method of a learned model according to claim 14 .

16 . An image processing system including an imaging apparatus and a learning apparatus configured to communicate with the imaging apparatus,

wherein the learning apparatus includes

at least one memory storing instructions; and

at least one processor that executes the instructions to:

acquire a first training image and a first ground truth image,

generate, by dividing the first training image and the first ground truth image, a plurality of second training images where each has less number of pixels than that of the first training image, and a plurality of second ground truth images where each has less number of pixels than that of the first ground truth image;

generate a plurality of estimated images upscaled by inputting the plurality of second training images to a machine learning model, and

update a weight of a neural network based on the plurality of estimated images and the plurality of second ground truth images,

wherein the imaging apparatus includes an optical system, an image sensor, and an image estimation unit, and

wherein the image estimation unit includes

at least one memory storing instructions; and

at least one processor that executes the instructions to:

acquire a first grayscale image based on a first color image, the first color image being acquired by image capturing that uses an image sensor,

generate, by dividing the first grayscale image, a plurality of second grayscale images each including a number of pixels smaller than a number of pixels of the first grayscale image,

generate a plurality of third grayscale images upscaled by inputting the plurality of second grayscale images and an ISO sensitivity of the image sensor in the image capturing to the machine learning model, wherein in the generating the plurality of third grayscale images, the plurality of second grayscale images and the ISO sensitivity are concatenated in a channel direction and inputted to the machine learning model; and

generate a second color image based on the plurality of third grayscale images, wherein the second color image has more pixels than the first color image.

17 . An image processing system including a control apparatus and an image processing apparatus configured to communicate with the control apparatus,

wherein the control apparatus includes a unit configured to transmit a request for causing the image processing apparatus to execute processing on a captured image, and

wherein the image processing apparatus includes

at least one memory storing instructions; and

at least one processor that executes the instructions to:

receive the request,

acquire the captured image, wherein the captured image is a first color image acquired by image capturing using an image sensor,

generate a first grayscale image based on the first color image,

generate, by dividing the first grayscale image, a plurality of second grayscale images each including a number of pixels smaller than a number of pixels of the first grayscale image,

generate a plurality of third grayscale images upscaled by inputting the plurality of second grayscale images and an ISO sensitivity of the image sensor in the image capturing to a machine learning model, wherein in the generating the plurality of third grayscale images, the plurality of second grayscale images and the ISO sensitivity are concatenated in a channel direction and inputted to the machine learning model; and

generating a second color image based on the plurality of third grayscale images, wherein the second color image has more pixels than the first color image.