Image processing apparatus and method, and storage medium for generating high-definition images
There is provided an image processing apparatus that renders images of a second image group high-definition using a first image group, the second image group including less high frequency components than the first image group does. The image processing apparatus selects, from among supervisory data pieces that each include an image included in the first image group as one of an image pair, a supervisory data piece to be used in learning based on a high definition target image selected from the second image group, learns a learning model using the selected supervisory data piece, infers high frequency components of the high definition target image using the learning model, and generates a high-definition image based on the high definition target image and the inferred high frequency components.
1 . An image processing apparatus that renders images of a second image group high-definition using a first image group, the second image group including less high frequency components than the first image group does, the image processing apparatus comprising:
at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to:
select, from among supervisory data pieces that each include (i) an image included in the first image group as one of an image pair and (ii) a corresponding paired image that includes less high frequency components than the image included in the first image group, a supervisory data piece to be used in training based on a high definition target image selected from the second image group, wherein selecting the supervisory data piece comprises selecting, from among the supervisory data pieces, the supervisory data piece satisfying at least one of: (a) an imaging time difference between the high definition target image and an image included in the supervisory data piece being smaller than a predetermined threshold value, or (b) a similarity between the high definition target image and the image included in the supervisory data piece being higher than a predetermined threshold value;
train a learning model using the selected supervisory data piece;
infer high frequency components of the high definition target image using the learning model; and
generate a high-definition image based on the high definition target image and the inferred high frequency components.
2 . The image processing apparatus according to claim 1 , wherein the instructions further cause the at least one processor to:
obtain, as supervisory data candidates, pairs of an image selected from the first image group and an image which includes less high frequency components than the selected image does and which is related to the selected image,
wherein selecting the supervisory data piece to be used in training comprising selecting from among the obtained supervisory data candidates.
3 . The image processing apparatus according to claim 2 , wherein
obtaining the supervisory data candidates comprises obtaining, from the second image group, images related to images selected from the first image group.
4 . The image processing apparatus according to claim 3 , wherein
obtaining the supervisory data candidates comprises obtaining, from the second image group, images that have the same imaging times as the selected images.
5 . The image processing apparatus according to claim 3 , wherein
obtaining the supervisory data candidates comprises obtaining, from the second image group, images with imaging times that are different from imaging times of the selected images by an amount smaller than a predetermined threshold.
6 . The image processing apparatus according to claim 3 , wherein
obtaining the supervisory data candidates comprises obtaining, from the second image group, images that exhibit the highest similarity to the selected images from the second image group.
7 . The image processing apparatus according to claim 6 , wherein
the first image group has a first resolution, and the second image group has a second resolution lower than the first resolution, and
obtaining the supervisory data candidates comprises obtaining similarities between images obtained by reducing the selected images to the second resolution and images of the second image group.
8 . The image processing apparatus according to claim 2 , wherein
obtaining the supervisory data candidates comprises obtaining, images obtained by lowering a resolution of the selected images through reduction of the selected images.
9 . The image processing apparatus according to claim 8 , wherein
the first image group has a first resolution, and the second image group has a second resolution lower than the first resolution, and
the obtained images related to the selected images are images obtained by reducing the selected images to the second resolution.
10 . The image processing apparatus according to claim 2 , wherein
selecting unit selects, as the supervisory data piece to be used in training comprises selecting from among the supervisory data candidates a supervisory data candidate that includes an image with an imaging time that is different from an imaging time of the high definition target image by an amount smaller than a predetermined threshold value.
11 . The image processing apparatus according to claim 2 , wherein
selecting unit selects, as the supervisory data piece to be used in training comprises selecting from among the supervisory data candidates, a supervisory data candidate that includes an image whose similarity to the high definition target image is higher than a predetermined threshold value.
12 . The image processing apparatus according to claim 1 , wherein
training the learning model comprises controlling an update of a parameter through back propagation in the training based on the supervisory data piece to be used in training and on the high definition target image.
13 . The image processing apparatus according to claim 12 , wherein
training the learning model comprises determining a coefficient based on the supervisory data piece to be used in learning and on the high definition target image, and controlling an amount of the update of the parameter through the back propagation based on the coefficient.
14 . The image processing apparatus according to claim 12 , wherein
training the learning model comprises determining a coefficient based on the supervisory data pieces to be used in training and on the high definition target image, and controlling the number of times the update of the parameter through the back propagation is repeated based on the coefficient.
15 . The image processing apparatus according to claim 13 , wherein
training the learning model comprises determining the coefficient based on a difference between an imaging time of an image of the supervisory data piece to be used in training and the imaging time of the high definition target image.
16 . The image processing apparatus according to claim 13 , wherein
training the learning model comprises determining the coefficient based on a similarity between an image of the supervisory data piece to be used in training and the high definition target image.
17 . The image processing apparatus according to claim 1 , wherein
the first image group and the second image group are two image groups obtained by executing different types of image processing with respect to one set of images shot by one image sensor included in one image capturing apparatus.
18 . The image processing apparatus according to claim 1 , wherein
the first image group and the second image group are image groups that have been respectively shot by two image sensors.
19 . The image processing apparatus according to claim 1 , wherein
a frame rate of the first image group is lower than a frame rate of the second image group.
20 . An image processing method that renders images of a second image group high-definition using a first image group, the second image group including less high frequency components than the first image group does, the image processing method comprising:
selecting, from among supervisory data pieces that each include (i) an image included in the first image group as one of an image pair (ii) a corresponding paired image that includes less high frequency components than the image included in the first image group, a supervisory data piece to be used in training based on a high definition target image selected from the second image group, wherein selecting the supervisory data piece comprises selecting, from among the supervisory data pieces, the supervisory data piece satisfying at least one of: (a) an imaging time difference between the high definition target image and an image included in the supervisory data piece being smaller than a predetermined threshold value, or (b) a similarity between the high definition target image and the image included in the supervisory data piece being higher than a predetermined threshold value;
training a learning model using the selected supervisory data piece;
inferring high frequency components of the high definition target image using the learning model; and
generating a high-definition image based on the high definition target image and the inferred high frequency components.
21 . A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method that renders images of a second image group high-definition using a first image group, the second image group including less high frequency components than the first image group does, the image processing method comprising:
selecting, from among supervisory data pieces that each include (i) an image included in the first image group as one of an image pair (ii) a corresponding paired image that includes less high frequency components than the image included in the first image group, a supervisory data piece to be used in training based on a high definition target image selected from the second image group, wherein selecting the supervisory data piece comprises selecting, from among the supervisory data pieces, the supervisory data piece satisfying at least one of: (a) an imaging time difference between the high definition target image and an image included in the supervisory data piece being smaller than a predetermined threshold value, or (b) a similarity between the high definition target image and the image included in the supervisory data piece being higher than a predetermined threshold value;
training a learning model using the selected supervisory data piece;
inferring high frequency components of the high definition target image using the learning model; and
generating a high-definition image based on the high definition target image and the inferred high frequency components.