Image processing apparatus that forms distance image and control method thereof, as well as storage medium
An image processing apparatus capable of correcting an error during computation of a correlation degree without storing data for correcting distance information in advance in a recording medium, to highly accurately form a distance image. The image processing apparatus has a distance estimation unit that estimates a subject distance in each area, from a correlation degree of a plurality of images, a distance reliability calculation unit that calculates the reliability of the subject distance estimated by the distance estimation unit, and a distance image forming unit that corrects the subject distance estimated by the distance estimation unit based on the reliability calculated by the distance reliability calculation unit and to form a distance image based on the corrected subject distance.
1. An image processing apparatus comprising:
a processor processing input data of a plurality of images, the processor performing the functions of:
a distance estimation unit configured to estimate distance information corresponding to a subject distance in each area of the plurality of images, from a correlation degree of the plurality of images;
a distance reliability calculation unit configured to calculate a reliability of the distance information estimated by the distance estimation unit; and
a distance image forming unit configured to correct the distance information estimated by the distance estimation unit based on the reliability calculated by the distance reliability calculation unit and to form a distance image based on the corrected distance information,
wherein the distance image includes a monochrome image which changes in gray scale according to the estimated distance information or a color image which changes in tone according to the estimated distance information.
2. The image processing apparatus as claimed in claim 1 , wherein the plurality of images are a pair of images with parallax or a pair of images with different focusing positions.
3. The image processing apparatus as claimed in claim 1 , wherein the distance estimation unit determines whether to replace the calculated distance information with distance information of infinity based on a first threshold that is set in advance.
4. The image processing apparatus as claimed in claim 3 , wherein the distance reliability calculation unit determines whether the distance information estimated by the distance estimation unit is distance information exceeding infinity based on a second threshold that is set in advance and allocates a low reliability if the distance reliability calculation unit determines that the distance information exceeds infinity.
5. The image processing apparatus as claimed in claim 1 , wherein the distance image forming unit is configured to use the reliability calculated by the distance reliability calculation unit and to perform a correction of replacing the distance information of an area with low reliability with the estimated distance information of an area with a high reliability among areas around the area with low reliability.
6. The image processing apparatus as claimed in claim 1 , wherein the distance image forming unit uses the reliability calculated by the distance reliability calculation unit and performs a correction of replacing a value of the distance information of an area with low reliability with an average value of the estimated distance information of a plurality of areas with high reliabilities around the area with low reliability.
7. The image processing apparatus as claimed in claim 4 , wherein the processor further functions as a subject recognition unit configured to set the first threshold and the second threshold to the distance information of infinity in an area including a subject with a specific luminance recognized by the subject recognition unit.
8. The image processing apparatus as claimed in claim 7 , wherein the distance reliability calculation unit is configured to allocate a low reliability to the area including the subject with the specific luminance recognized by the subject recognition unit.
9. The image processing apparatus as claimed in claim 1 , further comprising an input unit configured to input the plurality of images to the distance estimation unit.
10. The image processing apparatus as claimed in claim 9 , wherein the input unit is an image pickup unit configured to pick up an image of a subject.
11. A control method of an image processing apparatus comprising a processor processing input data of a plurality of images, the control method being performed by the processor and comprising:
a distance estimation step of estimating distance information corresponding to a subject distance in each area of a plurality of images from a correlation degree of the plurality of images;
a distance reliability calculation step of calculating a reliability of the distance information estimated at the distance estimation step; and
a distance image forming step of correcting the distance information estimated at the distance estimation step based on the reliability calculated by the distance reliability calculation step and forming a distance image based on the corrected distance information,
wherein the distance image includes a monochrome image which changes in gray scale according to the estimated distance information or a color image which changes in tone according to the estimated distance information.
12. A non-transitory computer-readable storage medium storing a program causing a computer to implement a control method of an image processing apparatus, comprising a processor processing input data of a plurality of images, the control method being performed by the processor and, the control method comprising:
a distance estimation step of estimating distance information corresponding to a subject distance in each area of a plurality of images, from a correlation degree of the plurality of images;
a distance reliability calculation step of calculating a reliability of the distance information estimated at the distance estimation step; and
a distance image forming step of correcting the distance information estimated at the distance estimation step based on the reliability calculated by the distance reliability calculation step and forming a distance image based on the corrected distance information,
wherein the distance image includes a monochrome image which changes in gray scale according to the estimated distance information or a color image which changes in tone according to the estimated distance information.
13. An image processing apparatus comprising:
a processor processing input data of a plurality of images, performing the functions of:
a distance estimation unit configured to estimate distance information corresponding to a subject distance in each image area of the plurality of images from a correlation degree of the plurality of images;
a distance reliability calculation unit configured to calculate a reliability of the distance information estimated by the distance estimation unit, by comparing the estimated distance information and distance information corresponding to a specified subject distance; and
a distance image forming unit configured to correct the distance information estimated by the distance estimation unit based on the reliability calculated by the distance reliability calculation unit and to form a distance image based on the estimated distance information and the corrected distance information.
14. The image processing apparatus as claimed in claim 13 , wherein the plurality of images are a pair of images with parallax or a pair of images with different focusing positions.
15. The image processing apparatus as claimed in claim 13 , wherein the distance estimation unit determines whether to replace the calculated distance information with distance information of infinity based on a first threshold that is set in advance.
16. The image processing apparatus as claimed in claim 15 , wherein the distance reliability calculation unit determines whether the distance information calculated by the distance estimation unit is distance information exceeding infinity based on a second threshold that is set in advance and to allocate a low reliability if the distance reliability calculation unit determines that the distance information exceeds infinity.
17. The image processing apparatus as claimed in claim 13 , wherein the distance image forming unit is configured to use the reliability calculated by the distance reliability calculation unit and to perform a correction of replacing the distance information of the image area with low reliability with the estimated distance information of the image area with a high reliability among areas around the area with low reliability.
18. The image processing apparatus as claimed in claim 13 , wherein the distance image forming unit uses the reliability calculated by the distance reliability calculation unit and performs a correction of replacing a value of the subject distance of the image area with low reliability with an average value of the estimated subject distances of a plurality of the image areas with high reliabilities around the area with low reliability.
19. The image processing apparatus as claimed in claim 1 , wherein the distance information estimated by the distance estimation unit indicates a defocus amount.
20. The image processing apparatus as claimed in claim 13 , wherein the distance information estimated by the distance estimation unit indicates a defocus amount.
21. The image processing apparatus as claimed in claim 13 , wherein the specified subject distance indicates a subject distance of infinity.
22. An image processing apparatus comprising:
a processor processing input data of a plurality of images, the processor performing the functions of:
a distance estimation unit configured to estimate distance information corresponding to a subject distance in each image area of the plurality of images from a correlation degree of the plurality of images;
a correcting unit configured to correct distance information corresponding to a subject distance which exceeds a specified subject distance to correspond to the specified subject distance; and
a distance data array forming unit configured to form a distance data array based on the estimated distance information and the corrected distance information.
23. The image processing apparatus as claimed in claim 22 , wherein the plurality of images are a pair of images with parallax or a pair of images with different focusing positions.
24. The image processing apparatus as claimed in claim 22 , wherein the distance information estimated by the distance estimation unit indicates a defocus amount.
25. A control method of an image processing apparatus comprising a processor processing input data of a plurality of images, the control method being performed by the processor and comprising:
a distance estimation step of estimating distance information corresponding to a subject distance in each image area of the plurality of images from a correlation degree of the plurality of images;
a distance reliability calculation step of calculating a reliability of the distance information estimated by the distance estimation step by comparing the estimated distance information and distance information corresponding to a specified subject distance; and
a distance image forming step of correcting the distance information estimated by the distance estimation step based on the calculated reliability calculated by the distance reliability calculation step and forming a distance image based on the estimated distance information and the corrected distance information.
26. A control method of an image processing apparatus comprising a processor processing input data of a plurality of images, the control method being performed by the processor and comprising:
a distance estimation step of estimating distance information corresponding to a subject distance in each image area of the plurality of images from a correlation degree of the plurality of images; and
a distance image forming step of correcting distance information corresponding to a subject distance which exceeds a specified subject distance to correspond to the specified subject distance; and
a distance array forming step of forming a distance data array based on the estimated distance information and the corrected distance information.