Image processing apparatus, image processing method and storage medium for enhancing resolution of image
The image processing apparatus has: an image acquisition unit configured to acquire a captured image of an image capturing area in which an object is located; a determination unit configured to determine parameters to be used for image processing to improve resolution of an image of the object, by learning using a dataset of images of the object, wherein the dataset is generated based on object information indicating a degree of importance of the object; and a processing unit configured to perform the image processing to improve the resolution of an image of the object included in the acquired captured image, using the parameters determined by the determination unit.
1. An image processing apparatus comprising:
one or more memories storing instructions; and
one or more processors executing the instructions to:
acquire a captured image of an image capturing area in which an object is located;
determine parameters to be used for image processing to improve resolution of an image of the object, by learning using a dataset of images of the object, wherein the dataset is generated based on object information indicating a degree of importance of the object and a number of pairs of images included in the dataset used for learning depends on the degree of importance of the object, each of the pairs of images including two images with different resolutions; and
perform the image processing to improve the resolution of the image of the object included in the acquired captured image, using the determined parameters,
wherein, in a case where an image of a first object and an image of a second object are included in the acquired captured image and a degree of importance of the first object is higher than the degree of importance of the second object, a number of pairs of images included in the dataset used for learning for the first object is larger than a number of pairs of images included in the dataset used for learning for the second object.
2. The image processing apparatus according to claim 1 , wherein
the dataset for learning is a set of pairs of a low-resolution image of the object and a high-resolution image of the object.
3. The image processing apparatus according to claim 2 , wherein
the higher a degree of importance of the object indicated by the object information is, the higher the resolution of the high-resolution image included in the dataset of images of the object.
4. The image processing apparatus according to claim 2 , wherein
the one or more processors further execute the instructions to generate the dataset for learning by performing conversion processing in accordance with the degree of importance of the object indicated by the object information for a training image, which is the image of the object.
5. The image processing apparatus according to claim 4 , wherein
the conversion processing includes at least one of processing to change the resolution of the training image and processing to divide the training image.
6. The image processing apparatus according to claim 4 , wherein
a degree of specialization of the dataset for learning for a specific object or a specific environment is controlled in accordance with the degree of importance of the object.
7. The image processing apparatus according to claim 1 , wherein
processing to improve resolution of the image of the first object is performed based on parameters determined by learning using the dataset in accordance with the degree of importance of the first object and processing to improve resolution of the image of the second object is performed based on parameters determined by learning using the dataset in accordance with the degree of importance of the second object, and
the degree of importance of the first object and the degree of importance of the second object are different.
8. The image processing apparatus according to claim 7 , wherein
the resolution of the image of the first object and the resolution of the image of the second object, which are in the captured image after processing, are different.
9. The image processing apparatus according to claim 1 , wherein
the object information includes information indicating at least one of an attribute of the object, a behavior of the object, and a behavior of another object for the object.
10. The image processing apparatus according to claim 1 , wherein
the one or more processors further execute the instructions to set a resolution level relating to a change in the resolution of the image of the object based on a user operation, and
the object information indicates the resolution level as the degree of importance of the object.
11. The image processing apparatus according to claim 10 , wherein
the one or more processors further execute the instructions to display the resolution level and an image of the resolution in accordance with the resolution level on a display screen for the user operation.
12. The image processing apparatus according to claim 1 , wherein
the one or more processors further execute the instructions to generate a virtual viewpoint image in accordance with a position of a specified virtual viewpoint and a view direction from the specified virtual viewpoint based on the captured image after processing.
13. The image processing apparatus according to claim 12 , wherein
the object information includes information indicating at least one of a distance between the virtual viewpoint and the object, a position relationship between the virtual viewpoint and the object, a frequency with which the object is observed from the virtual viewpoint, and a frequency with which the front side of the object is observed from the virtual viewpoint.
14. The image processing apparatus according to claim 1 , wherein
the object is a person or a region of a person.
15. The image processing apparatus according to claim 1 , wherein
the parameter is determined by performing learning by a neural network by using the dataset.
16. The image processing apparatus according to claim 1 , wherein the resolution of the image of the object included in the captured image is improved by performing estimation processing by a neural network by using the parameter.
17. An image processing method comprising:
acquiring a captured image of an image capturing area in which an object is located by an image acquisition unit;
determining parameters to be used for image processing to improve resolution of an image of the object, by learning using a dataset of images of the object, wherein the dataset is generated based on object information indicating a degree of importance of the object and a number of pairs of images included in the dataset used for learning depends on the degree of importance of the object, each of the pairs of images including two images with different resolutions; and
performing the image processing to improve the resolution of the image of the object included in the acquired captured image, using the determined parameters,
wherein, in a case where an image of a first object and an image of a second object are included in the acquired captured image and a degree of importance of the first object is higher than the degree of importance of the second object, a number of pairs of images included in the dataset used for learning for the first object is larger than a number of pairs of images included in the dataset used for learning for the second object.
18. The image processing method according to claim 17 , wherein
processing to improve resolution of the image of the first object is performed based on parameters determined by learning using the dataset in accordance with the degree of importance of the first object and processing to improve resolution of the image of the second object is performed based on parameters determined by learning using the dataset in accordance with the degree of importance of the second object and
the degree of importance of the first object and the degree of importance of the second object are different.
19. The image processing method according to claim 17 , further comprising:
generating a virtual viewpoint image in accordance with a position of a specified virtual viewpoint and a view direction from the specified virtual viewpoint based on the captured image after processing by the image processing.
20. A non-transitory computer readable storage medium storing a program for causing a computer to perform an image processing method, the image processing method comprising:
acquiring a captured image of an image capturing area in which an object is located by an image acquisition unit;
determining parameters to be used for image processing to improve resolution of an image of the object, by learning using a dataset of images of the object, wherein the dataset is generated based on object information indicating a degree of importance of the object and a number of pairs of images included in the dataset used for learning depends on the degree of importance of the object, each of the pairs of images including two images with different resolutions; and
performing the image processing to improve the resolution of the image of the object included in the acquired captured image, using the determined parameters
wherein, in a case where an image of a first object and an image of a second object are included in the acquired captured image and a degree of importance of the first object is higher than the degree of importance of the second object, a number of pairs of images included in the dataset used for learning for the first object is larger than a number of pairs of images included in the dataset used for learning for the second object.