Capturing and processing of images including occlusions focused on an image sensor by a lens stack array
Systems and methods for implementing array cameras configured to perform super-resolution processing to generate higher resolution super-resolved images using a plurality of captured images and lens stack arrays that can be utilized in array cameras are disclosed. An imaging device in accordance with one embodiment of the invention includes at least one imager array, and each imager in the array comprises a plurality of light sensing elements and a lens stack including at least one lens surface, where the lens stack is configured to form an image on the light sensing elements, control circuitry configured to capture images formed on the light sensing elements of each of the imagers, and a super-resolution processing module configured to generate at least one higher resolution super-resolved image using a plurality of the captured images.
1. An image enhancement method, comprising:
normalizing a set of images using calibration data stored in a storage device using an address conversion module, where:
the set of images comprises a plurality of images captured from different viewpoints; and
the plurality of images include different occlusions sets, where the occlusion set of a first image is the portion of a scene visible in a second image that is occluded in the first image;
detecting and measuring parallax using a parallax confirmation and measurement module, where detecting and measuring parallax comprises determining the parallax that yields a best match between pixels from images in the set of normalized images by:
performing redundant pair-wise measurements between pixels from different images in the set of images to determine pixel similarity at different parallax shifts; and
keeping track of various pair-wise measurements and calculating a parallax difference that yields the best match between pixels from images in the set of normalized images;
generating a depth map using a parallax compensation module based upon the measured parallax, where the parallax compensation module identifies pixels in the set of normalized images having calculated parallax differences that should not be considered in the generation of the depth map;
processing at least one image from the set of images based upon the depth map using an image processing pipeline module; and
displaying the at least one processed image on a display device.
2. The method of claim 1 , further comprising capturing the set of images using an imaging device comprising a camera array.
3. The method of claim 2 , wherein:
the calibration data indicates mappings between addresses of physical pixels in imagers and logical addresses within an image; and
the address conversion module resolves offsets between the set of images by converting addresses of physical pixels in imagers to logical addresses.
4. The method of claim 1 , wherein calculating the parallax difference that yields the best match between pixels from images in the set of normalized images comprises determining the parallax difference that provides the best least squares fit.
5. The method of claim 1 , wherein processing at least one image from the set of images based upon the depth map using an image processing pipeline module comprises:
aligning portions of images in the set of images to compensate for parallax using the parallax compensation module based upon the measured parallax and the stored calibration data; and
obtaining a higher resolution image having a resolution that is higher than the resolutions of the images in the set of images using a super-resolution module.
6. The method of claim 5 , wherein obtaining the higher resolution image comprises:
fusing the aligned image portions using the super-resolution module; and
performing super-resolution processing on the fused image portions to synthesize a super-resolution image using the super-resolution module.
7. The method of claim 1 , wherein processing at least one image from the set of images based upon the depth map using an image processing pipeline module comprises selecting at least one distance as a focal plane and applying blurring to pixels in the at least one image that the depth map indicates are not proximate the focal plane.
8. The method of claim 1 , further comprising estimating distance to an object visible in the set of images using the parallax confirmation and measurement module.
9. The method of claim 1 , wherein the different parallax shifts considered when performing the redundant pair-wise measurements between pixels from different images in the set of images include parallax shifts having sub-pixel offsets.
10. The method of claim 1 , wherein:
the set of images includes:
images captured using the same imaging characteristics; and
images captured using different imaging characteristics;
the redundant pair-wise measurements between pixels from different images in the set of images are pair-wise measurements between pixels from images in the set of images captured using the same imaging characteristics.
11. A method for synthesizing an image from a set of images captured from different viewpoints, comprising:
normalizing, using at least one processor, a set of images using calibration data stored in a storage device, where:
the set of images comprises a plurality of images that are captured from different viewpoints; and
the plurality of images include different occlusions sets, where the occlusion set of a first image is the portion of a scene visible in a second image that is occluded in the first image;
detecting and measuring parallax using the at least one processor, where detecting and measuring parallax comprises determining the parallax that yields the best match between pixels from images in the set of normalized images by:
performing redundant pair-wise measurements between pixels from different images in the set of images to determine similarity of neighborhoods of pixels at different parallax shifts; and
keeping track of various pair-wise measurements and calculating a parallax difference that yields the best match between pixels from images in the set of normalized images;
generating, using the at least one processor, a depth map based upon the measured parallax, where generating the depth map comprises identifying pixels in the set of normalized images having calculated parallax differences that should not be considered;
synthesizing an image, using the at least one processor, based upon at least one image from the set of images and the depth map; and
displaying the synthesized image on a display device.
12. The method of claim 11 , further comprising capturing the set of images using an imaging device comprising a camera array.
13. The method of claim 12 , wherein:
the calibration data indicates mappings between addresses of physical pixels in imagers and logical addresses within an image; and
the at least one processor resolves offsets between the set of images by converting addresses of physical pixels in imagers to logical addresses.
14. The method of claim 11 , wherein calculating the parallax difference that yields the best match between pixels from images in the set of normalized images comprises determining the parallax difference that provides the best least squares fit.
15. The method of claim 11 , wherein synthesizing the synthesized image comprises:
aligning portions of different images in the set of images to compensate for parallax based upon the measured parallax and the stored calibration data; and
obtaining a higher resolution image having a resolution that is higher than the resolutions of the images in the set of images.
16. The method of claim 15 , wherein obtaining the higher resolution image comprises fusing the aligned image portions.
17. The method of claim 16 , wherein obtaining the higher resolution image comprises performing super-resolution processing on the fused image portions to synthesize a super-resolution image.
18. The method of claim 11 , wherein synthesizing the synthesized image comprises selecting at least one distance as a focal plane and applying blurring to pixels in the at least one image that the depth map indicates are not proximate the focal plane.
19. The method of claim 11 , further comprising estimating distance to an object visible in the set of images.
20. The method of claim 11 , wherein the different parallax shifts considered when performing the redundant pair-wise measurements between pixels from different images in the set of images include parallax shifts having sub-pixel offsets.
21. The method of claim 11 , wherein:
the set of images includes:
images captured using the same imaging characteristics; and
images captured using different imaging characteristics;
the redundant pair-wise measurements between pixels from different images in the set of images are pair-wise measurements between pixels from images in the set of images captured using the same imaging characteristics.
22. A method of synthesizing an image using a set of images captured using a camera array, comprising:
capturing a set of images using an imaging device comprising a camera array, where:
the set of images comprises a plurality of images that are captured from different viewpoints; and
the plurality of images include different occlusions sets, where the occlusion set of a first image is the portion of a scene visible in a second image that is occluded in the first image;
normalizing, using at least one processor, the set of images using calibration data stored in a storage device;
detecting and measuring parallax using the at least one processor, where detecting and measuring parallax comprises determining the parallax that yields the best match between pixels from images in the set of normalized images by:
performing redundant pair-wise measurements between pixels from different pairs of images in the set of images to determine similarity of neighborhoods of pixels at different parallax shifts, wherein the different parallax shifts considered when performing the redundant pair-wise measurements include parallax shifts having sub-pixel offsets; and
keeping track of various pair-wise measurements and calculating a parallax difference that yields the best match between pixels from images in the set of normalized images;
generating, using the at least one processor, a depth map based upon the measured parallax, where generating the depth map comprises identifying pixels in the set of normalized images having calculated parallax differences that should not be considered;
synthesizing an image, using the at least one processor, based upon at least one image from the set of images and the depth map; and
displaying the synthesized image on a display device.
23. The method of claim 22 , wherein:
the calibration data indicates mappings between addresses of physical pixels in imagers and logical addresses within an image; and
the at least one processor resolves offsets between the set of images by converting addresses of physical pixels in imagers to logical addresses.
24. The method of claim 22 , wherein calculating the parallax difference that yields the best match between pixels from images in the set of normalized images comprises determining the parallax difference that provides the best least squares fit.
25. The method of claim 22 , wherein synthesizing the synthesized image comprises:
aligning portions of different images in the set of images to compensate for parallax based upon the measured parallax and the stored calibration data; and
obtaining a higher resolution image having a resolution that is higher than the resolutions of the images in the set of images.
26. The method of claim 25 , wherein obtaining the higher resolution image comprises fusing the aligned image portions.
27. The method of claim 26 , wherein obtaining the higher resolution image comprises performing super-resolution processing on the fused image portions to synthesize a super-resolution image.
28. The method of claim 22 , wherein synthesizing the synthesized image comprises selecting at least one distance as a focal plane and applying blurring to pixels in the at least one image that the depth map indicates are not proximate the focal plane.
29. The method of claim 22 , further comprising estimating distance to an object visible in the set of images.
30. The method of claim 23 , wherein:
the set of images includes:
images captured using the same imaging characteristics; and
images captured using different imaging characteristics;
the redundant pair-wise measurements between pixels from different images in the set of images are pair-wise measurements between pixels from images in the set of images captured using the same imaging characteristics.