Image processing apparatus, image processing method and storage medium
An image processing apparatus generates a learning model for making an image high definition using teacher data selected from among a plurality of teacher data which use an image included in the first image group as one of a pair of images. The image processing apparatus calculates a degree of similarity between a current image selected as a high definition target from a second image group and a previous image which is a high definition target previous to the current image, infers high frequency components of the current image using the generated learning model when the degree of similarity is equal to or less than a threshold, and infers high frequency components of a current image using a learning model used to make the previous image high definition when the degree of similarity is greater than the threshold.
1 . An image processing apparatus that uses a first image group to make an image of a second image group with fewer high frequency components than an image of the first image group high definition, comprising:
at least one processor; and
at least one memory having instructions stored thereon which, when executed by the at least one processor, cause the image processing apparatus at least to:
calculate a degree of similarity between a current image selected as a high definition target from the second image group and a previous image selected from the second image group which is a high definition target previous to the current image;
a select, on a basis of the current image, teacher data to be used in learning from among a plurality of teacher data which use an image included in the first image group as one of a pair of images;
generate a learning model for making the current image high definition using the teacher data;
a infer high frequency components of the current image using the learning model when the degree of similarity is equal to or less than a threshold and infer high frequency components of the current image using a learning model used to make the previous image high definition when the degree of similarity is greater than the threshold; and
generate a high definition image on a basis of the current image and the inferred high frequency components.
2 . The image processing apparatus according to claim 1 , wherein the instructions, when executed by the at least one processor, further cause the image processing apparatus at least to:
obtain, as a candidate for the teacher data, a pair including a first image selected from the first image group and a third image related to the first image with fewer high frequency components than the first image,
wherein the teacher data to be used in the learning is selected from the candidate for the teacher data.
3 . The image processing apparatus according to claim 2 , wherein the candidate for the teacher data is obtained by obtaining the third image from the second image group.
4 . The image processing apparatus according to claim 3 , wherein an image with imaging time identical to imaging time of the first image is obtained from the second image group, as the third image.
5 . The image processing apparatus according to claim 3 , wherein an image with a difference in imaging time to the first image that is less than a predetermined threshold is obtained from the second image group, as the third image.
6 . The image processing apparatus according to claim 3 , wherein an image with the highest similarity to the first image is obtained from the second image group, as the third image.
7 . The image processing apparatus according to claim 6 , wherein a similarity between an image of the first image reduced to a resolution of the second image group and an image of the second image group is determined.
8 . The image processing apparatus according to claim 2 , wherein an image of the first image reduced in size and with lower resolution is obtained, as the third image.
9 . The image processing apparatus according to claim 8 , wherein the third image is an image of the first image reduced to a resolution of the second image group.
10 . The image processing apparatus according to claim 2 , wherein a candidate for teacher data including an image with a difference in imaging time to the current image that is less than a predetermined threshold is selected, as the teacher data to be used in the learning.
11 . The image processing apparatus according to claim 2 , wherein teacher data including an image with a similarity to the current image that is greater than a predetermined threshold is selected, from among the candidate for the teacher data, as the teacher data to be used in the learning.
12 . The image processing apparatus according to claim 1 , wherein updating a parameter via backpropagation in the learning is controlled, on a basis of the teacher data to be used in the learning and the current image.
13 . The image processing apparatus according to claim 12 , wherein a coefficient on a basis of the teacher data to be used in the learning and the current image is determined and an update amount for the parameter via the backpropagation is controlled on a basis of the coefficient.
14 . The image processing apparatus according to claim 13 , wherein the coefficient is determined on a basis of a difference between imaging time of an image of teacher data to be used in the learning and imaging time of the current image.
15 . The image processing apparatus according to claim 13 , wherein the coefficient is determined on a basis of a similarity between an image of teacher data to be used in the learning and the current image.
16 . The image processing apparatus according to claim 12 , wherein a coefficient is determined on a basis of the teacher data to be used in the learning and the current image, and a number of repetitions of updating the parameter via the backpropagation is controlled on a basis of the coefficient.
17 . The image processing apparatus according to claim 1 , wherein the instructions, when executed by the at least one processor, further cause the image processing apparatus at least to:
obtain, from the teacher data, a pair of images corresponding to a local region extracted from the current image,
wherein a learning model is generated of the local region using the pair of images,
wherein high frequency components of the local region of the current image are inferred using a learning model generated when the degree of similarity is equal to or less than the threshold and using a learning model used to make a region corresponding to the local region of the previous image high definition when the degree of similarity is greater than the threshold, and
wherein a high definition image of the local region is generated using the high frequency components of the local region and an image of the local region of the current image, and high definition images generated for each local region are combined.
18 . The image processing apparatus according to claim 17 , wherein a degree of similarity between the current image and the previous image is calculated for each local region, and
wherein a learning model is used for a local region where the degree of similarity is equal to or less than the threshold, a learning model is used to make the region of the previous image high definition for a local region where the degree of similarity is greater than the threshold, and high frequency components of the local region of the current image are inferred.
19 . The image processing apparatus according to claim 17 , wherein a pair of images of a region corresponding to an identical coordinate position as the local region is obtained from teacher data.
20 . The image processing apparatus according to claim 19 , wherein a high definition image is generated by combining high definition images of each local region on a basis of a coordinate position of a local region.
21 . The image processing apparatus according to claim 17 , wherein a pair of images with the highest degree of similarity to an image of the local region is obtained from among a plurality of pairs of images extracted from the teacher data.
22 . The image processing apparatus according to claim 17 , wherein a pair of images corresponding to the local region is obtained from the teacher data on a basis of a motion vector set for a block included in the local region or on a basis of a motion vector referencing the block included in the local region.
23 . The image processing apparatus according to claim 17 , wherein a plurality of pairs of images corresponding to a plurality of regions is obtained from the teacher data on a basis of a position of the local region, and
wherein a number of time for learning to be performed is determined using each one of the plurality of pairs of images in generating the learning model on a basis of a degree of similarity between an image of the local region of the current image and each one of the plurality of pairs of images.
24 . The image processing apparatus according to claim 23 , wherein the plurality of regions includes a first region corresponding to a position of the local region and a second region adjacent to the first region.
25 . The image processing apparatus according to claim 23 , wherein the model generation unit does not perform learning using a pair of images with a degree of similarity is not performed with an image of the local region that is equal to or less than a threshold.
26 . The image processing apparatus according to claim 1 , wherein the first image group and the second image group are two image groups obtained by different image processing being executed on one image captured by one image sensor included in one image capture apparatus.
27 . The image processing apparatus according to claim 1 , wherein the first image group and the second image group are image groups captured by two different image sensors.
28 . The image processing apparatus according to claim 1 , wherein the first image group has a lower frame rate than the second image group.
29 . An image processing method that uses a first image group to make an image of a second image group with fewer high frequency components than an image of the first image group high definition, comprising:
calculating a degree of similarity between a current image selected as a high definition target from the second image group and a previous image selected from the second image group which is a high definition target previous to the current image;
selecting, on a basis of the current image, teacher data to be used in learning from among a plurality of teacher data which use an image included in the first image group as one of a pair of images;
generating a learning model for making the current image high definition using the selected teacher data;
inferring high frequency components of the current image using the learning model generated in the generating when the degree of similarity is equal to or less than a threshold, and inferring high frequency components of the current image using a learning model used to make the previous image high definition when the degree of similarity is greater than the threshold; and
generating a high definition image on a basis of the current image and the high frequency components inferred in the inferring.
30 . A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method that uses a first image group to make an image of a second image group with fewer high frequency components than an image of the first image group high definition, the method comprising:
calculating a degree of similarity between a current image selected as a high definition target from the second image group and a previous image selected from the second image group which is a high definition target previous to the current image;
selecting, on a basis of the current image, teacher data to be used in learning from among a plurality of teacher data which use an image included in the first image group as one of a pair of images;
generating a learning model for making the current image high definition using the selected teacher data;
inferring high frequency components of the current image using the learning model generated in the generating when the degree of similarity is equal to or less than a threshold, and inferring high frequency components of the current image using a learning model used to make the previous image high definition when the degree of similarity is greater than the threshold; and
generating a high definition image on a basis of the current image and the high frequency components inferred in the inferring.