Capturing and processing of images including occlusions focused on an image sensor by a lens stack array
Systems and methods for implementing array cameras configured to perform super-resolution processing to generate higher resolution super-resolved images using a plurality of captured images and lens stack arrays that can be utilized in array cameras are disclosed. An imaging device in accordance with one embodiment of the invention includes at least one imager array, and each imager in the array comprises a plurality of light sensing elements and a lens stack including at least one lens surface, where the lens stack is configured to form an image on the light sensing elements, control circuitry configured to capture images formed on the light sensing elements of each of the imagers, and a super-resolution processing module configured to generate at least one higher resolution super-resolved image using a plurality of the captured images.
1. A camera array, comprising:
a plurality of cameras configured to capture images of a scene, where each camera comprises:
optics comprising at least one lens element and at least one aperture; and
a sensor comprising a two-dimensional array of pixels and control circuitry for controlling imaging parameters;
a processor configured by software to:
capture a plurality of images from different viewpoints using the plurality of cameras, where each image captured by the plurality of cameras includes pixels that are occluded in at least one other image captured by the plurality of cameras; and
normalize the plurality of images based upon calibration data to enable scan-line based parallax searches;
measure parallax between the normalized images by adaptively comparing the similarity of neighborhoods of pixels for different parallax-induced shifts along scan-lines;
identify occluded pixels based upon the measured parallax information;
generate a depth map using the measured parallax information;
select at least one distance as an “in best focus” distance; and
blur an image produced by the camera array based upon the “in best focus” distance and distance information from the depth map.
2. The camera array of claim 1 , wherein the plurality of cameras comprises an array of camera arrays.
3. The camera array of claim 1 , wherein the plurality of cameras are arranged as a first set of cameras forming a first camera array, and a second set of cameras forming a second camera array.
4. The camera array of claim 1 , wherein at least two of the plurality of cameras are separated by a distance approximating the separation of human eyes.
5. The camera array of claim 1 , wherein each camera includes a spectral filter configured to pass a specific spectral band of light selected from the group consisting of a Bayer filter, one or more Blue filters, one or more Green filters, one or more Red filters, one or more shifted spectral filters, one or more near-IR filters, and one or more hyper-spectral filters.
6. The camera array of claim 1 , wherein the control circuitry in each of plurality of cameras is capable of configuring the plurality of cameras to operate with at least one difference in operating parameters.
7. The camera array of claim 6 , wherein the at least one difference in operating parameters includes at least one imaging parameter selected from the group consisting of exposure time, gain, and black level offset.
8. The camera array of claim 6 , wherein the camera array is a monolithic camera array assembly comprising: a lens element array forming the optics of each camera; and a single semiconductor substrate on which all of the pixels and control circuitry for each camera are formed.
9. The camera array of claim 1 , wherein the plurality of cameras are formed on separate semiconductor substrates.
10. The camera array of claim 9 , wherein the plurality of cameras are mounted to a printed circuit board.
11. The camera array of claim 1 , wherein at least two cameras have different focal lengths.
12. The camera array of claim 1 , wherein the sensor in each of at least two cameras have different resolutions.
13. The camera array of claim 1 , wherein the sensor in each of at least two cameras have different sizes.
14. The camera array of claim 1 , wherein the processor is further configured by software to measure parallax by determining the parallax that yields the highest correlation between pixels from images captured by the plurality of cameras accounting for the positions of the cameras that captured the images.
15. The camera array of claim 1 , wherein the processor is further configured by software to perform pair-wise measurements of neighborhoods of pixels to determine pixel similarity for different parallax-induced shifts.
16. The camera array of claim 15 , wherein the processor is further configured by software to determine a parallax that yields a highest similarity between pixels from images captured by keeping track of various pair-wise measurements of neighborhoods of pixels and calculating a parallax that yields the highest similarity as the best least squares fit of the pair-wise measurements of neighborhoods of pixels.
17. The camera array of claim 1 , wherein the processor is further configured by software to generate at least one image by fusing aligned portions of the plurality of captured images using the depth map.
18. The camera array of claim 17 , wherein the processor is further configured by software to perform super-resolution processing on the fused image portions to synthesize a super-resolution image.
19. The camera array of claim 1 , wherein the processor is further configured by software to generate a higher resolution super-resolved image synthesized using the plurality of images and the parallax measurements to compensate for parallax in the plurality of images.
20. A camera array, comprising:
a plurality of cameras configured to capture images of a scene, where each camera comprises:
optics comprising at least one lens element and at least one aperture; and
a sensor comprising a two-dimensional array of pixels and control circuitry for controlling imaging parameters;
wherein at least one of the plurality of cameras comprises a Bayer filter and the control circuitry in each of plurality of cameras is capable of configuring the plurality of cameras to operate with at least one difference in operating parameters;
a processor configured by software to:
capture a plurality of images from different viewpoints using the plurality of cameras, where each image captured by the plurality of cameras includes pixels that are occluded in at least one other image captured by the plurality of cameras; and
normalize the plurality of images based upon calibration data to enable scan-line based parallax searches;
measure parallax between the normalized images by adaptively comparing the similarity of neighborhoods of pixels for different parallax-induced shifts along scan-lines by performing pair-wise measurements of neighborhoods of pixels to determine pixel similarity for different parallax-induced shifts;
identify occluded pixels based upon the measured parallax information;
generate a depth map using the measured parallax information by discarding measured parallax information determined using occluded pixels;
select at least one distance as an “in best focus” distance; and
blur an image produced by the camera array based upon the “in best focus” distance and distance information from the depth map.