Methods and apparatus for using a controllable physical light filter as part of an image capture system and for processing captured images
Methods and apparatus for using a controllable filter, e.g., an liquid crystal panel, in front of a camera are described. The filter is controlled based on the luminosity of object in a scene being captured by the camera to reduce or eliminate luminosity related image defects such as flaring, blooming or ghosting. Multiple cameras and filters can be used to capture multiple images as part of a depth determination processes where pixel values captured by cameras at different locations are matched to determine the depth, e.g., distance from the camera or camera system to object in the environment. Pixel values are normalized in some embodiments based on the amount of filtering applied to a sensor region and sensor exposure time. The filtering allows for regional sensor exposure control at an individual camera even though the overall exposure time of the pixel sensors may be and often will be the same.
1. A method involving image capture, the method comprising:
controlling, during a first time period, a filter positioned in front of a first image sensor of a first camera to allow maximum light transmission through the filter to the first image sensor;
processing one or more images captured during the first time period by the first image sensor to detect one or more image regions with defects due to excessive light intensity, said detected one or more image regions including at least a first image region;
controlling the filter to reduce light transmission by a first amount through a first portion of the filter corresponding to a portion of the first image sensor used to capture a portion of an environment corresponding to the first image region; and
capturing at least a first additional image at a first time while the light transmission through the first portion of the filter is reduced by the first amount;
generating a first depth map from at least the first additional image; and
wherein said step of generating a first depth map from at least the first additional image includes processing the first additional image to compensate for a decrease in light intensity caused by the first portion of the filter.
2. The method of claim 1 , wherein said filter is a liquid crystal display (LCD) filter.
3. The method of claim 1 , wherein processing one or more images captured during the first time period by the first image sensor to detect one or more image regions with defects due to excessive light intensity includes:
identifying one, more than one, or all of: i) image defects due to flaring, ii) image defects due to blooming; and iii) image defects due to ghosting.
4. The method of claim 1 , wherein generating a first depth map from at least the first additional image includes:
generating said first depth map from at least the first additional image and at least a second additional image captured by a second image sensor of a second camera.
5. The method of claim 4 , wherein processing the first additional image to compensate for the decrease in light intensity caused by the first portion of the filter includes scaling luminance values in the first additional image corresponding to the first portion of the filter by an amount which is based on the reduction of light transmission through the first portion of the filter.
6. The method of claim 5 , wherein processing one or more images captured during the first time period by the first image sensor to detect one or more image regions with defects due to excessive light intensity includes:
identifying one, more or all of: i) image defects due to flaring, ii) image defects due to blooming; and iii) image defects due to ghosting.
7. The method of claim 6 , further comprising:
detecting motion of an object corresponding to a detected image defect in a sequence of images captured by the first image sensor; and
changing which portion of the filter is subject to a light reduction operation corresponding to the object based on the expected motion of the object.
8. The method of claim 6 , further comprising:
controlling the filter positioned in front of the first image sensor of the first camera during a second time period to allow maximum light transmission through the filter to the first image sensor;
processing one or more images captured by the first image sensor during the second time period to detect one or more image areas with defects due to excessive light intensity, said one or more image areas including at least a third image area;
controlling the filter to reduce light transmission by a second amount through a third portion of the filter corresponding to a portion of the first image sensor used to capture a portion of an environment corresponding to the third image area based on an image defect detected in an image captured during said second time period; and
capturing another image following said second time period while the light transmission through the third portion of the filter is reduced by the second amount.
9. The method of claim 8 , further comprising:
generating a second depth map from said another image following said second time period while the light transmission through the third portion of the filter is reduced by the second amount.
10. The method of claim 1 ,
wherein said filter is a liquid crystal display (LCD) filter positioned at least one centimeter in front of a first lens of the first camera;
wherein each pixel of the LCD filter is individually controlled; and
wherein said first additional image is captured by the first image sensor of the first camera.
11. A system, comprising:
a first camera;
a controllable light filter positioned in front of a first image sensor of the first camera; and
a processor configured to:
control, during a first time period, a filter positioned in front of the first camera to allow maximum light transmission through the filter to a first image sensor included in the first camera;
process one or more images captured during the first time period by the first image sensor to detect one or more image regions with defects due to excessive light intensity, said detected one or more image regions including at least a first image region;
control the controllable light filter to reduce light transmission by a first amount through a first portion of the filter corresponding to a portion of the first image sensor used to capture a portion of an environment corresponding to the first image region;
control capture of at least a first additional image at a first time while the light transmission through the first portion of the filter is reduced by the first amount; and
generate a first depth map from at least the first additional image, said generating a first depth map including processing the first additional image to compensate for a decrease in light intensity caused by the first portion of the filter.
12. The system of claim 11 , wherein said filter is a liquid crystal display (LCD) filter.
13. The system of claim 11 , wherein the processor is configured to:
identify one, more than one, or all of: i) image defects due to flaring, ii) image defects due to blooming; and iii) image defects due to ghosting as part of being configured to process one or more images captured during the first time period by the image sensor to detect one or more image regions with defects due to excessive light intensity.
14. The system of claim 11 , further comprising:
a second camera including a second image sensor.
15. The system of claim 14 , wherein the processor is configured, as part of processing the first additional image to compensate for the decrease in light intensity caused by the first portion of the filter to:
scale luminance values in the first additional image corresponding to the first portion of the filter by an amount which is a based on the reduction of light transmission through the first portion of the filter by an amount which is inversely proportional to the amount of light transmission reduction by the first portion of the filter.
16. The system of claim 15 , wherein the processor is configured as part of processing one or more images captured during the first time period by the first image sensor to detect one or more image regions with defects due to excessive light intensity, to:
identify one, more or all of: i) image defects due to flaring, ii) image defects due to blooming; and iii) image defects due to ghosting.
17. The system of claim 16 , wherein the processor is further configured to:
detect motion of an object corresponding to a detected image defect in a sequence of images captured by the first image sensor; and
change the location of the portion of the filter subject to a light reduction operation corresponding to the object based on the expected motion of the object.
18. The system of claim 16 , wherein the processor is further configured to:
control the controllable light filter positioned in front of the first image sensor of the first camera during a second time period to allow maximum light transmission through the filter to the first image sensor;
process one or more images captured by the first image sensor during the second time period to detect one or more image areas with defects due to excessive light intensity, said one or more image areas including at least a third image area;
control the controllable light filter to reduce light transmission by a second amount through a third portion of the controllable light filter corresponding to a portion of the first image sensor used to capture a portion of an environment corresponding to the third image area based on an image defect detected in an image captured during said second time period said first, second and third portions of the controllable light filter being different portions; and
control the first image sensor to capture another image following said second time period while the light transmission through the third portion of the filter is reduced by the second amount.
19. The system of claim 18 , wherein the processor is further configured to:
generate a second depth map from said another image following said second time period while the light transmission through the third portion of the filter is reduced by the second amount.
20. The system of claim 11 ,
wherein said filter is a liquid crystal display (LCD) filter positioned at least one centimeter in front of a first lens of the first camera;
wherein each pixel of the LCD filter is individually controlled by the processor; and
wherein said first additional image is captured by the first image sensor of the first camera.