Image processing device, image capturing device, control method of image processing device, control method of image capturing device, and storage medium

1. An image processing device that corrects an image captured by an imaging sensor capable of setting an exposure condition for each of a plurality of regions, the image processing device comprising:

at least one memory coupled to at least one processor that is configured to operate as:

(a) an acquisition unit configured to acquire the image by setting the exposure condition for each region in the imaging sensor;

(b) a derivation unit configured, regarding a boundary between a region of interest and at least one adjacent region adjacent to the region of interest in the image, to derive a correction amount in the region of interest based on pixel values of a boundary pixel group in contact with the boundary included in the region of interest and pixel values of a boundary pixel group in contact with the boundary included in the adjacent region; and

(c) a correction unit configured to correct pixel values of pixels in the region of interest based on the derived correction amount,

wherein the derivation unit derives the correction amount based on an average value of the pixel values of the boundary pixel group in contact with the boundary included in the region of interest and an average value of the pixel values of the boundary pixel group in contact with the boundary included in the adjacent region.

2. The image processing device according to claim 1 , that is further configured to operate as a boundary information acquisition unit configured to acquire boundary information indicating a boundary between regions for which the imaging sensor can set exposure,

wherein the derivation unit derives the correction amount based on the boundary information.

3. The image processing device according to claim 1 , wherein the regions have a shape in which a plurality of pixels are disposed in a first direction and a second direction different from the first direction.

4. The image processing device according to claim 1 , wherein an image in which pixels corresponding to a plurality of colors are disposed is captured by the imaging sensor, and

wherein processing by the derivation unit and the correction unit is performed for each of the plurality of colors.

5. An image processing device that corrects an image captured by an imaging sensor capable of setting an exposure condition for each of a plurality of regions, the image processing device comprising:

at least one memory coupled to at least one processor that is configured to operate as:

(a) an acquisition unit configured to acquire the image by setting the exposure condition for each region in the imaging sensor;

(b) a derivation unit configured, regarding a boundary between a region of interest and at least one adjacent region adjacent to the region of interest in the image, to derive a correction amount in the region of interest based on pixel values of a boundary pixel group in contact with the boundary included in the region of interest and pixel values of a boundary pixel group in contact with the boundary included in the adjacent region; and

(c) a correction unit configured to correct pixel values of pixels in the region of interest based on the derived correction amount,

wherein the derivation unit derives an evaluation value that indicates evaluation of a step of gradation occurring on the boundary of each region based on an average value of pixel values of a boundary pixel group in contact with the boundary included in the region of interest and an average value of pixel values of a boundary pixel group in contact with the boundary included in the adjacent region, estimates an exposure setting error in each of the regions based on the evaluation value, and derives a correction value for correcting the estimated exposure setting error.

6. The image processing device according to claim 5 , wherein the derivation unit estimates an exposure setting error in each of the regions based on an overall evaluation value obtained by adding the evaluation value corresponding to each of the regions.

7. The image processing device according to claim 6 , wherein the derivation unit estimates an exposure setting error in each of the regions that minimizes the overall evaluation value.

8. The image processing device according to claim 5 , wherein the derivation unit derives the evaluation value corresponding to the region of interest using an average value and a derivative value of pixel values of a boundary pixel group on the region of interest side and an average value and a derivative value of pixel values of a boundary pixel group on the adjacent region side on the boundary between the region of interest and the adjacent region.

9. The image processing device according to claim 8 , wherein the derivation unit derives a derivative value of peripheral pixels on the region of interest side and a derivative value of peripheral pixels on the adjacent region side using peripheral pixels of at least two-pixel width adjacent to the boundary.

10. The image processing device according to claim 5 , wherein the derivation unit derives the correction amount after changing the estimated value of the exposure setting error to a value within a predetermined range when the estimated value of the exposure setting error exceeds the predetermined range in which the imaging sensor can take.

11. The image processing device according to claim 10 , wherein the derivation unit determines whether the estimated value of the exposure setting error exceeds the predetermined range based on error characteristic information indicating the predetermined range obtained by measuring the imaging sensor in advance.

12. The image processing device according to claim 5 , wherein the derivation unit:

derives a regularization term from a distribution characteristic of the exposure setting error in each of the regions obtained by measuring the imaging sensor in advance; and

estimates the exposure setting error in each of the regions based on the evaluation value to which the derived regularization term is added.

13. The image processing device according to claim 5 , wherein the derivation unit excludes pixel values corresponding to a saturated pixel and a defective pixel from object pixel values for deriving the evaluation value when deriving the evaluation value.

14. An image processing device that corrects an image captured by an imaging sensor capable of setting an exposure condition for each of a plurality of regions, the image processing device comprising:

at least one memory coupled to at least one processor that is configured to operate as:

(a) an acquisition unit configured to acquire the image by setting the exposure condition for each region in the imaging sensor;

(b) a derivation unit configured, regarding a boundary between a region of interest and at least one adjacent region adjacent to the region of interest in the image, to derive a correction amount in the region of interest based on pixel values of a boundary pixel group in contact with the boundary included in the region of interest and pixel values of a boundary pixel group in contact with the boundary included in the adjacent region; and

(c) a correction unit configured to correct pixel values of pixels in the region of interest based on the derived correction amount,

wherein the derivation unit derives the correction amount after performing a correction process for cancelling out a difference in the exposure condition set for each of the regions on the image acquired by the acquisition unit.

15. The image processing device according to claim 14 , wherein the derivation unit performs the correction process on the image acquired by the acquisition unit based on the exposure condition set for each of the regions so that a light amount received by the imaging sensor has a linear relationship with a pixel value of the image acquired by the acquisition unit.

16. A control method of controlling an image processing device that corrects an image captured by an imaging sensor capable of setting an exposure condition for each of a plurality of regions, the control method comprising:

acquiring the image by setting the exposure condition for each region in the imaging sensor;

regarding a boundary between a region of interest and at least one adjacent region adjacent to the region of interest in the image, deriving a correction amount in the region of interest based on pixel values of a boundary pixel group in contact with the boundary included in the region of interest and pixel values of a boundary pixel group in contact with the boundary included in the adjacent region; and

correcting pixel values of pixels in the region of interest based on the derived correction amount,

wherein the derivation derives the correction amount based on an average value of the pixel values of the boundary pixel group in contact with the boundary included in the region of interest and an average value of the pixel values of the boundary pixel group in contact with the boundary included in the adjacent region.

17. A control method of controlling an image capturing device including an imaging sensor capable of setting an exposure condition for each of a plurality of regions, the control method comprising:

acquiring an image by setting the exposure condition for each region by the imaging sensor;

regarding a boundary between a region of interest and at least one adjacent region adjacent to the region of interest in the image, deriving a correction amount in the region of interest based on pixel values of a boundary pixel group in contact with the boundary included in the region of interest and pixel values of a boundary pixel group in contact with the boundary included in the adjacent region; and

correcting pixel values of pixels in the region of interest based on the derived correction amount,

wherein the derivation derives an evaluation value that indicates evaluation of a step of gradation occurring on the boundary of each region based on an average value of pixel values of a boundary pixel group in contact with the boundary included in the region of interest and an average value of pixel values of a boundary pixel group in contact with the boundary included in the adjacent region, estimates an exposure setting error in each of the regions based on the evaluation value, and derives a correction value for correcting the estimated exposure setting error.

18. A non-transitory computer readable storage medium storing a program for causing a computer to perform each step in a control method of controlling an image processing device that corrects an image captured by an imaging sensor capable of setting an exposure condition for each of a plurality of regions, the control method comprising:

acquiring the image by setting the exposure condition for each region in the imaging sensor;

regarding a boundary between a region of interest and at least one adjacent region adjacent to the region of interest in the image, deriving a correction amount in the region of interest based on pixel values of a boundary pixel group in contact with the boundary included in the region of interest and pixel values of a boundary pixel group in contact with the boundary included in the adjacent region; and

correcting pixel values of pixels in the region of interest based on the derived correction amount,

wherein the derivation derives the correction amount based on an average value of the pixel values of the boundary pixel group in contact with the boundary included in the region of interest and an average value of the pixel values of the boundary pixel group in contact with the boundary included in the adjacent region.