Image processing device, imaging device, image processing method, and program having different exposure start times
View Patent ↗A device and method are provided to generate an image with a wide dynamic range, based on one photographed image. An exposure time control is executed in which different exposure times are set in units of rows of a pixel portion, in units of pixel regions, or similar, and different, multiple pixel information which are the pixel values of the pixels set with different exposures are obtained. For example, high sensitivity pixel information is obtained from long exposure pixels, low sensitivity information is obtained from short exposure pixels, and a pixel value for the output image is calculated based on the pixel information with these different sensitivities. For example, for high luminosity regions, the sensitivity pixel information may have a saturated value, and so a large weight is set to the low sensitivity pixel information, also for low luminosity regions, the SN ratio of low sensitivity pixel information is estimated to be poor, and so a large weight is set to the high sensitivity pixel information to determine the output pixel value.
1. An image processing device, comprising:
a control unit configured to control an exposure time of pixel regions by setting exposure start times for the pixel regions in accordance with different exposure times of the pixel regions;
a pixel portion configured to output pixel information including high sensitivity pixel information from pixel regions of long exposure times and low sensitivity pixel information from pixel regions of short exposure times; and
a pixel information combining unit configured to set weighting coefficients for the high sensitivity pixel information from the pixel regions of long exposure times and for the low sensitivity pixel information from the pixel regions of short exposure times based on a brightness of a photographic subject, and to calculate a pixel value for one or more pixels of an output image based on the weighting coefficients, the high sensitivity pixel information from the pixel regions of long exposure times, and the low sensitivity pixel information from the pixel regions of short exposure times.
2. The image processing device according to claim 1 ,
wherein the exposure time of pixel regions is controlled in units of rows of the pixel portion, and
wherein the pixel portion outputs pixel information with different exposure times from a plurality of different rows.
3. The image processing device according to claim 1 , wherein a sum of the weighting coefficients for the high sensitivity pixel information from the pixel regions of long exposure times and the low sensitivity pixel information from the pixel regions of short exposure times equals a predefined value.
4. The image processing device according to claim 3 , wherein when the high sensitivity pixel information from the pixel regions of long exposure times is at or above a predetermined threshold, the pixel information combining unit sets the weighting coefficients of the high sensitivity pixel information from the pixel regions of long exposure times to zero or a small number.
5. The image processing device according to claim 3 , wherein when the low sensitivity pixel information from the pixel regions of short exposure times is below a predetermined threshold, the pixel information combining unit sets the weighting coefficients of the low sensitivity pixel information from the pixel regions of short exposure times to zero or a small number.
6. The image processing device according to claim 3 , wherein when the low sensitivity pixel information from the pixel regions of short exposure times is at or above a predetermined threshold, the pixel information combining unit sets the weighting coefficients of the high sensitivity pixel information from the pixel regions of long exposure times to zero or a small number.
7. The image processing device according to claim 3 ,
wherein the control unit is configured to set the exposure time of pixel regions of long exposure times and the exposure time of pixel regions of short exposure times in units of two rows of the pixel portion, and
wherein the pixel information combining unit calculates the pixel value for the one or more pixels of the output image based on a plurality of pixel information from both the high sensitivity pixel information from the pixel regions of long exposure times and the low sensitivity pixel information from the pixel regions of short exposure times.
8. The image processing device according to claim 1 , wherein the control unit is configured to control the exposure time in units of rows by controlling a shutter in units of rows of the pixel portion.
9. The image processing device according to claim 1 , wherein the control unit is configured to control the exposure time in units of color by controlling a shutter in units of color of the pixel portion.
10. The image processing device according to claim 1 , wherein the pixel portion is configured to output pixel information including medium sensitivity pixel information from pixel regions of medium exposure times having exposure times between the long exposure times and the short exposure times, and the pixel information combining unit is configured to set weighting coefficients for the medium sensitivity pixel information from the pixel regions of medium exposure times, and to calculate the pixel value for the one or more pixels of the output image based on the weighting coefficients, the high sensitivity pixel information from the pixel regions of long exposure times, the medium sensitivity pixel information from the pixel regions of medium exposure times, and the low sensitivity pixel information from the pixel regions of short exposure times.
11. The image processing device according to claim 1 , further comprising a gradation conversion unit configured to reduce a number of bits of the pixel values for one or more output images.
12. The image processing device according to claim 1 , further comprising a signal processing unit configured to process one or more output images generated from the pixel information combining unit.
13. The image processing device according to claim 1 , further comprising a codec configured to encode output images generated by the pixel information combining unit.
14. The image processing device according to claim 1 , wherein the pixel portion has a configuration in which pixel information with different exposure times of the same color is output from pixel regions that are at least 3×3 pixels.
15. The image processing device according to claim 1 , wherein the pixel portion includes a Bayer array or an RGBW array.
16. An imaging device, comprising:
an imaging unit; and
the image processing device according to any one of claims 1 - 9 and 10 - 15 .
17. An image processing method executed by an image processing device, the method comprising:
controlling an exposure time of pixel regions by setting exposure start times for the pixel regions in accordance with different exposure times of the pixel regions;
outputting pixel information including high sensitivity pixel information from pixel regions of long exposure times and low sensitivity pixel information from pixel regions of short exposure times;
setting weighting coefficients for the high sensitivity pixel information from the pixel regions of long exposure times and for the low sensitivity pixel information from the pixel regions of short exposure times based on a brightness of a photographic subject; and
calculating a pixel value for one or more pixels of an output image based on the weighting coefficients, the high sensitivity pixel information from the pixel regions of long exposure times, and the low sensitivity pixel information from the pixel regions of short exposure times.
18. A non-transitory computer-readable storage medium storing a program for executing the image processing method of claim 17 .
19. An image processing device, comprising:
a control unit that controls an exposure time of pixel regions by setting exposure start times for the pixel regions in accordance with different exposure times of the pixel regions;
a pixel portion that outputs pixel information including high sensitivity pixel information from pixel regions of long exposure times and low sensitivity pixel information from pixel regions of short exposure times; and
a pixel information combining unit that sets weighting coefficients for the high sensitivity pixel information from the pixel regions of long exposure times and for the low sensitivity pixel information from the pixel regions of short exposure times based on a brightness of a photographic subject, and calculates a pixel value for one or more pixels of an output image based on the weighting coefficients, the high sensitivity pixel information from the pixel regions of long exposure times, and the low sensitivity pixel information from the pixel regions of short exposure times.
20. The image processing device according to claim 19 , wherein a sum of the weighting coefficients for the high sensitivity pixel information from the pixel regions of long exposure times and the low sensitivity pixel information from the pixel regions of short exposure times equals a predefined value.
21. The image processing device according to claim 20 , wherein,
when the high sensitivity pixel information from the pixel regions of long exposure times is at or above a predetermined threshold, the pixel information combining unit sets the weighting coefficients of the high sensitivity pixel information from the pixel regions of long exposure times to zero or a small number, and
when the low sensitivity pixel information from the pixel regions of short exposure times is below a predetermined threshold, the pixel information combining unit sets the weighting coefficients of the low sensitivity pixel information from the pixel regions of short exposure times to zero or a small number.