Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers
Systems and methods in accordance with embodiments of the invention are disclosed that use super-resolution (SR) processes to use information from a plurality of low resolution (LR) images captured by an array camera to produce a synthesized higher resolution image. One embodiment includes obtaining input images using the plurality of imagers, using a microprocessor to determine an initial estimate of at least a portion of a high resolution image using a plurality of pixels from the input images, and using a microprocessor to determine a high resolution image that when mapped through the forward imaging transformation matches the input images to within at least one predetermined criterion using the initial estimate of at least a portion of the high resolution image. In addition, each forward imaging transformation corresponds to the manner in which each imager in the imaging array generate the input images, and the high resolution image synthesized by the microprocessor has a resolution that is greater than any of the input images.
1. An array camera, comprising:
a plurality of cameras with different fields of view and control circuitry for independently triggering each camera, where:
a first camera in the plurality of cameras captures low resolution (LR) images having a first field of view with a first magnification and a first resolution;
a second camera in the plurality of cameras captures higher resolution (HR) images having a second field of view with a second magnification and a second resolution, wherein the first field of view is different from the second field of view and the second resolution is different than the first resolution;
each of the cameras receives light through a separate lens system and includes a sensor;
the control circuitry controls the imaging parameters of the sensor of each camera;
at least one processor; and
memory containing an image processing pipeline software application;
wherein the image processing pipeline software directs the at least one processor to:
trigger the first camera to capture an LR input image and store the LR input image in memory;
trigger the second camera to capture an HR input image and store the HR input image in memory;
shifting pixels from the LR input image relative to a reference view;
fusing pixels from the LR input image and the HR input image;
synthesize an image using the LR input image and the HR input image, wherein the synthesized image has a higher resolution than the LR input image.
2. The array camera of claim 1 , where the LR input image and the HR input image capture a scene in which depths of points in the scene vary and the LR input image differs from the HR input image due to scene dependent geometric displacements due to parallax,
normalizing the LR input image and the HR input image to increase the similarity of corresponding pixels within the images.
3. The array camera of claim 2 , wherein the image processing pipeline software directs the at least one processor to fuse at least a portion of a plurality of pixels from the LR input image and the HR input image by populating a high resolution grid corresponding to pixel locations of the LR input mage and the HR input image using geometric correction information.
4. The array camera of claim 3 , wherein fusing at least a portion of a plurality of pixels from the LR input image and the HR input image further comprises interpolating the grid to obtain filtered pixel values for each pixel of the image.
5. The array camera of 2 , wherein the image processing pipeline software directs the at least one processor to photometrically normalize each of the LR input image and the HR input image to obtain a plurality of normalized images and storing the plurality of normalized images in memory using the at least one processor configured by image processing pipeline software.
6. The array camera of claim 1 , wherein the image processing pipeline software directs the at least one processor to generate a depth map for the synthesized image.
7. The array camera of claim 6 , wherein generating the depth map further comprises:
determining depth information for pixels in the synthesized image based upon the LR input image and the HR input image; and
interpolating the depth information to obtain depth information for every pixel in the synthesized image.
8. The array camera of claim 1 , wherein the different fields of view provide for different magnifications of a scene being captured.
9. The array camera of claim 1 , wherein the first camera has a field of view that is a maximum field of view and the second camera has a field of view that is ½ the maximum field of view.
10. The array camera of claim 1 , wherein the first camera has a field of view that is ½ a maximum field of view and the second camera has a field of view that is the maximum field of view.
11. A method of synthesizing an image using an array camera, the method comprising:
capturing a low resolution (LR) image using a first camera in a plurality of cameras with different fields of view of the array camera, where the LR image has a first field of view with a first magnification and a first resolution;
capture a higher resolution (HR) image using a second camera in the plurality of cameras, where the HR image has a second field of view with a second magnification and a second resolution, wherein the first field of view is different from the second field of view and the second resolution is different than the first resolution;
wherein each of the cameras receives light through a separate lens system and includes a sensor;
store the LR input image in memory;
store the HR input image in memory;
shift pixels from the LR input image relative to a reference view;
fuse pixels from the LR input image and the HR input image
synthesize an image using the LR input image and the HR input image, wherein the synthesized image has a higher resolution than each of the LR input image.
12. The method of claim 11 , wherein the LR input image and the HR input image capture a scene in which depths of points in the scene vary and the LR input image differs from the HR input image due to scene dependent geometric displacements due to parallax;
wherein the method further comprises:
normalizing the LR input image and the HR input image to increase the similarity of corresponding pixels within the images.
13. The method of claim 12 , further comprising photometrically normalizing each of the LR input image and the HR input image to obtain a plurality of normalized images and storing the plurality of normalized images in memory.
14. The method of claim 11 , further comprising fusing at least a portion of a plurality of pixels from the LR input image and the HR input image by populating a high resolution grid corresponding to pixel locations of the LR input mage and the HR input image using geometric correction information.
15. The method of claim 14 , wherein fusing at least a portion of a plurality of pixels from the LR input image and the HR input image further comprises interpolating the grid to obtain filtered pixel values for each pixel of the image.
16. The method of claim 11 , further comprising generating a depth map for the synthesized image.
17. The method of claim 16 , wherein generating the depth map further comprises:
determining depth information for pixels in the synthesized image based upon the LR input image and the HR input image; and
interpolating the depth information to obtain depth information for every pixel in the synthesized image.
18. The method of claim 11 , wherein the different fields of view provide for different magnifications of a scene being captured.
19. The method of claim 11 , wherein the first camera has a field of view that is a maximum field of view and the second camera has a field of view that is ½ the maximum field of view.
20. The method of claim 11 , wherein the first camera has a field of view that is ½ a maximum field of view and the second camera has a field of view that is the maximum field of view.