Image processing device, image processing method, and storage medium
The provided is an image processing device, an image processing method, and a program that can solve the above described problems and can normalize the distortion in the a fisheye image without being influenced by receiving the influence of the a fisheye optical system even when the positional relationship between the an optical axis of the a fisheye camera and the a plane to which the an object is grounded does not satisfy a certain conditions. An image processing device 10 includes: a fisheye image acquisition unit 11 that acquires a fisheye image; a viewpoint compensation vector acquisition unit 12 that acquires, based on a roll angle around an optical axis and a pitch angle of the optical axis of an imaging apparatus, a viewpoint compensation vector for converting the fisheye image into an image that is obtainable by taking an image of the object from a direction parallel to a ground plane of the object; and an image generation unit 13 that sets a plurality of viewpoints for a converted fisheye image that is the fisheye image converted using the viewpoint compensation vector, the plurality of viewpoints being parallel to the ground plane, performs a distortion correction by perspective projection approximation based on a coordinate system axes of which include a line of sight from a viewpoint for each of the plurality of viewpoints, and generates a single normalized image using image elements of vertical direction, the image elements each extracted from corrected fisheye images.
1. An image processing device comprising:
a memory that stores a set of instructions; and
at least one processor configured to execute the set of instructions to:
acquire a fisheye image in which an object is;
acquire, based on a roll angle around an optical axis and a pitch angle of the optical axis of an imaging apparatus, a viewpoint compensation vector for converting the fisheye image into an image that is obtainable by taking an image of the object from a direction parallel to a ground plane of the object, the roll angle and the pitch angle being based on a plane parallel to the ground plane of the object; and
set a plurality of viewpoints for a converted fisheye image along a horizontal direction of the converted fisheye image, the plurality of viewpoints being parallel to the ground plane, the converted fisheye image being the fisheye image converted using the viewpoint compensation vector,
perform a distortion correction by perspective projection approximation on the converted fisheye image based on a coordinate system axes of which include a line of sight from a viewpoint for each of the plurality of viewpoints that are set, and
generate a single normalized image using image elements of vertical direction, the image elements each extracted from corrected fisheye images.
2. The image processing device according to claim 1 , wherein
the at least one processor is further configured to
set the viewpoint based on an all pixel number in a horizontal direction of the normalized image to be generated,
perform, for each of the plurality of viewpoints, cutting out from the corrected fisheye image in a vertical direction after performing the distortion correction, extract slice images to each of which a line of sight from a viewpoint enters,
arrange the extracted slice images in predetermined order in a horizontal direction, and generate the single normalized image.
3. The image processing device according to claim 1 , wherein
the at least one processor is further configured to
determine, in the distortion correction by perspective projection approximation for each of the plurality of viewpoints, a distance of a projection plane to an origin in the coordinate system based on a size of the normalized image to be generated, a range of a viewing angle in a horizontal direction of the normalized image, and an aspect ratio of the normalized image.
4. The image processing device according to claim 1 , wherein
the at least one processor is further configured to
acquire, as the viewpoint compensation vector, a table describing an association between coordinates on an image which is obtainable by taking an image of the object from a direction parallel to the ground plane and coordinates on the fisheye image.
5. The image processing device according to claim 1 , wherein
the at least one processor is further configured to
acquire the roll angle around the optical axis and the pitch angle of the optical axis of the imaging apparatus from an angle sensor attached to the imaging apparatus, and generate the viewpoint compensation vector based on the roll angle and the pitch angle which are acquired.
6. An image processing method comprising:
acquiring a fisheye image in which an object is;
acquiring, based on a roll angle around an optical axis and a pitch angle of the optical axis of an imaging apparatus, a viewpoint compensation vector for converting the fisheye image into an image that is obtainable by taking an image of the object from a direction parallel to a ground plane of the object, the roll angle and the pitch angle being based on a plane parallel to the ground plane of the object; and
setting a plurality of viewpoints for a converted fisheye image along a horizontal direction of the converted fisheye image, the plurality of viewpoints being parallel to the ground plane, the converted fisheye image being the fisheye image converted using the viewpoint compensation vector,
performing a distortion correction by perspective projection approximation on the converted fisheye image based on a coordinate system axes of which include a line of sight from a viewpoint for each of the plurality of viewpoints that are set, and
performing image generation of generating a single normalized image using image elements of a vertical direction, the image elements each extracted from corrected fisheye images.
7. The image processing method according to claim 6 , wherein
the image generation includes
setting the viewpoint based on an all pixel number in a horizontal direction of the normalized image to be generated,
performing, for each of the plurality of viewpoints, cutting out from the corrected fisheye image in a vertical direction after performing the distortion correction, extracting slice images to each of which a line of sight from a viewpoint enters,
arranging the extracted slice images in predetermined order in a horizontal direction, and generating a single normalized image.
8. The image processing method according to claim 6 , wherein
the image generation includes
determining, in the distortion correction by perspective projection approximation for each of the plurality of viewpoints, a distance of a projection plane to an origin in the coordinate system is determined based on a size of the normalized image to be generated, a range of a viewing angle in a horizontal direction of the normalized image, and an aspect ratio of the normalized image.
9. The image processing method according to claim 6 , wherein
the acquiring the viewpoint compensation vector includes acquiring, as the viewpoint compensation vector, a table describing an association between coordinates on an image that is obtainable by taking an image of the object from a direction parallel to the ground plane and coordinates on the fisheye image is acquired.
10. The image processing method according to claim 6 , further comprising
acquiring the roll angle around the optical axis and the pitch angle of the optical axis of the imaging apparatus from an angle sensor attached to the imaging apparatus, and calculating the viewpoint compensation vector based on the roll angle and the pitch angle which are acquired.
11. A non-transitory computer readable storage medium storing a program that causes a computer to execute:
fisheye image acquisition processing of acquiring a fisheye image in which an object is;
viewpoint compensation vector acquisition processing of acquiring, based on a roll angle around an optical axis and a pitch angle of the optical axis of an imaging apparatus, a viewpoint compensation vector for converting the fisheye image into an image that is obtainable by taking an image of the object from a direction parallel to a ground plane of the object, the roll angle and the pitch angle being based on a plane parallel to the ground plane of the object; and
image generation processing of setting a plurality of viewpoints for a converted fisheye image along a horizontal direction of the converted fisheye, the plurality of viewpoints being parallel to the ground plane, the converted fisheye image being the fisheye image converted image using the viewpoint compensation vector,
performing a distortion correction by perspective projection approximation to the converted fisheye image based on a coordinate system axes of which include a line of sight from a viewpoint for each of the plurality of viewpoints that are set, and
generating a single normalized image using image elements of a vertical direction, the image elements each extracted from corrected fisheye images.
12. The storage medium according to claim 11 , wherein
the image generation processing
sets the viewpoint based on an all pixel number in a horizontal direction of the normalized image to be generated,
performs, for each of the plurality of viewpoints, cutting out from the corrected fisheye image in a vertical direction after performing the distortion correction, extracts slice images to each of which a line of sight from the viewpoint enters,
arranges the extracted slice images in predetermined order in a horizontal direction, and generates the single normalized image.
13. The storage medium according to claim 11 , wherein
the image generation processing
determines, in the distortion correction by perspective projection approximation for each of the plurality of viewpoints, a distance of a projection plane to an origin in the coordinate system based on a size of the normalized image to be generated, a range of a viewing angle in a horizontal direction of the normalized image, and an aspect ratio of the normalized image.
14. The storage medium according to claim 11 , wherein
the viewpoint compensation vector acquisition processing acquires, as the viewpoint compensation vector, a table describing an association between coordinates on an image that is obtainable by taking an image of the object from a direction parallel to the ground plane and coordinates on the fisheye image is acquired.
15. The storage medium according to claim 11 , the program further causing a computer to execute:
viewpoint compensation vector generation processing of acquiring the roll angle around the optical axis and the pitch angle of the optical axis of the imaging apparatus from an angle sensor attached to the imaging apparatus, and calculating the viewpoint compensation vector based on the roll angle and the pitch angle which are acquired.