Array camera imaging system and method
View Patent ↗Aspects of the disclosed technology relate to an imaging system and method in which an array camera is employed along with an image processor to make use of parallax convergence differences in color and luminance between different cameras or groups of cameras within an array camera to provide improved super resolution performance.
1. An imaging device comprising:
an array camera having at least M×N cameras, where each of M and N is at least two, the array camera producing M×N images, wherein the array camera includes N groups of M cameras; and
an image processor operatively coupled to the array camera, the image processor configured to:
for each group of M cameras, parallax converge the images from the group of M cameras based on a different predefined offset point within a predefined sampling pattern, producing N group outputs;
parallax converge the N group outputs based on the different predefined offset points within the predefined sampling pattern to create a color sub-pixel sampling pattern;
parallax converge luminance output of all cameras within the array camera to the predefined sampling pattern to create a luminance sub-pixel sampling pattern;
create a super resolution image by:
sampling the color sub-pixel sampling pattern at a first increased resolution;
sampling the luminance sub-pixel sampling pattern at a second increased resolution greater than the first increased resolution; and
combining the output from the sampled color sub-pixel sampling pattern and the luminance sub-pixel sampling pattern.
2. The imaging device of claim 1 , wherein M=N=4 and the cameras within the array camera are monochromatic cameras.
3. The imaging device of claim 2 , wherein the first increased resolution is at least 3×, and the second increased resolution is at least 4×.
4. The imaging device of claim 1 , wherein the image processor is configured to sample the color sub-pixel sampling pattern using a base sample contribution falloff.
5. The imaging device of claim 1 , wherein the image processor is configured to sample the luminance sub-pixel sampling pattern using a base sample contribution falloff.
6. The imaging device of claim 1 , wherein the image processor is configured to sample the color sub-pixel sampling pattern using a spherical sample contribution falloff sampling to surrounding samples.
7. The imaging device of claim 1 , wherein the image processor is configured to sample the luminance sub-pixel sampling pattern using a spherical sample contribution falloff sampling to surrounding samples.
8. The imaging device of claim 1 , wherein the predefined sampling pattern is one of a Hilbert sampling pattern or a recursive sampling pattern.
9. The imaging device of claim 1 , wherein the different predefined offset points create vertical, horizontal and diagonal differences between the 4 group outputs.
10. The imaging device of claim 1 , wherein the predefined offset points include at least four points per pixel that are relatively offset vertically, horizontally and diagonally.
11. A method of generating an image, the method comprising:
capturing image data using an array camera having at least M×N cameras, where each of M and N is at least two, the array camera producing M×N images, wherein the array camera includes N groups of M cameras;
for each group of M cameras, parallax converging the images from the group of M cameras based on a different predefined offset point within a predefined sampling pattern, producing N group outputs;
parallax converging the N group outputs based on the different predefined offset points within the predefined sampling pattern to create a color sub-pixel sampling pattern;
parallax converge luminance output of all cameras within the array camera to the predefined sampling pattern to create a luminance sub-pixel sampling pattern;
creating a super resolution image by:
sampling the color sub-pixel sampling pattern at a first increased resolution;
sampling the luminance sub-pixel sampling pattern at a second increased resolution greater than the first increased resolution; and
combining the output from the sampled color sub-pixel sampling pattern and the luminance sub-pixel sampling pattern.
12. The method of claim 11 , wherein M=N=4 and the cameras within the array camera are monochromatic cameras.
13. The method of claim 11 , wherein the first increased resolution is at least 3×, and the second increased resolution is at least 4×.
14. The method of claim 11 , wherein sampling the color sub-pixel sampling pattern includes using one of a base sample contribution falloff or a spherical sample contribution falloff to surrounding samples.
15. The method of claim 11 , wherein sampling the luminance sub-pixel sampling pattern includes using one of a base sample contribution falloff or a spherical sample contribution falloff to surrounding samples.
16. The method of claim 11 , wherein the predefined sampling pattern is one of a Hilbert sampling pattern or a recursive sampling pattern.
17. The method of claim 11 , wherein the different predefined offset points create vertical, horizontal and diagonal differences between the 4 group outputs.
18. The method of claim 11 , wherein the predefined offset points include at least four points per pixel that are relatively offset vertically, horizontally and diagonally.
19. The method of claim 11 , wherein M=N=3.