Flexible multi-camera focal plane: a light-field dynamic homography approach
View Patent ↗A method is provided for using Light-Field Dynamic Homography (DH) to generate a large virtual focal plane from a non-rigid camera array with narrow overlaps between their fields of view (FOV). The method incorporates the 3D geometry of the cameras and employs non-linear least square optimization to dynamically estimate the inter-view homography transformations. Remarkably, only two feature correspondences are required between adjacent views to stitch the images and generate a wide virtual focal plane array, eliminating the need for significant FOV overlaps between the multiple cameras.
1 . A system for image stitching to generate a single virtual focal plane image mosaic, the system comprising:
a flexible multi-camera array;
at least one processor; and
one or more computer storage media storing computer executable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving, from the flexible multi-camera array, a first camera image from a first camera of the flexible multi-camera array and a second camera image from a second camera of the flexible multi-camera array, the first camera image and the second camera image having a first narrow overlap portion;
detecting two point correspondences between the first camera image and the second camera image within the first narrow overlap portion;
generating an infinite homography transformation matrix for the first camera image and the second camera image using dynamic homography, wherein the infinite homography transformation matrix comprises 3-degrees of freedom to align the first camera image and the second camera image; and
based on the infinite homography transformation matrix, generating the single virtual focal plane image mosaic, wherein the single virtual focal plane image mosaic has a field of view that is larger than a first field of view of the first camera, or a second field of view of the second camera.
2 . The system of claim 1 , wherein the first camera image and the second camera image are captured synchronously, and wherein the infinite homography transformation matrix is refined through optimization of a relative rotation of each camera in the flexible multi-camera array to improve accuracy.
3 . The system of claim 1 , wherein the infinite homography transformation matrix is applied to transform the second camera image into a coordinate system of the first camera image through perspective warping.
4 . The system of claim 1 , wherein transformation of the second camera image includes applying interpolation techniques to estimate pixel values at non-grid positions.
5 . The system of claim 1 , the operations further comprising performing image blending techniques to merge the first camera image and second camera image in the single virtual focal plane image mosaic, wherein the first narrow overlap portion is minimal such that a size of the single virtual focal plane image mosaic is nearly equal to a sum of a first size of the first camera image and a second size of the second camera image.
6 . The system of claim 1 , wherein the flexible multi-camera array comprises a plurality of cameras with rigid and/or non-rigid positioning and orientation relative to one another.
7 . The system of claim 6 , wherein the single virtual focal plane image mosaic is generated by aligning and blending multiple images captured by the plurality of cameras in the flexible multi-camera array based on their corresponding infinite homography matrices between pairs of cameras having a narrow overlap field of view.
8 . The system of claim 1 , further comprising receiving a third image and a fourth image from the flexible multi-camera array.
9 . The system of claim 8 , further comprising detecting two or more point correspondences between any pair of narrowly overlapping camera images, wherein the flexible multi-camera array comprises more than two cameras.
10 . The system of claim 8 , further comprising generating additional infinite homography matrices for each narrowly overlapping pair of camera images received from the flexible multi-camera array.
11 . One or more computer storage media storing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for generating a single virtual focal plane image mosaic, the method comprising:
receiving from a flexible multi-camera array, a first camera image from a first camera of the flexible multi-camera array and a second camera image from a second camera of the flexible multi-camera array, the first camera image and the second camera image having a first narrow overlap portion;
detecting two point correspondences between the first camera image and the second camera image within the first narrow overlap portion;
generating an infinite homography transformation matrix for the first camera image and the second camera image using dynamic homography, wherein the infinite homography transformation matrix comprises 3-degrees of freedom to align the first camera image and the second camera image; and
based on the infinite homography transformation matrix, generating the single virtual focal plane image mosaic, wherein the single virtual focal plane image mosaic has a field of view that is larger than a first field of view of the first camera, or a second field of view of the second camera.
12 . The one or more computer storage media of claim 11 , wherein the flexible multi-camera array comprises a plurality of cameras with rigid and/or non-rigid positioning and orientation relative to one another.
13 . The one or more computer storage media of claim 11 , wherein the single virtual focal plane image mosaic is generated by aligning and blending each narrow overlap pair of camera images received from the flexible multi-camera array by generating additional infinite homography matrices.
14 . The one or more computer storage media of claim 11 , further comprising receiving a third camera image from a third camera of the flexible multi-camera array and a fourth image from a fourth camera of the flexible multi-camera array.
15 . The one or more computer storage media of claim 14 , further comprising detecting two point correspondences between any pair of narrowly overlapping camera images received from the flexible multi-camera array.
16 . The one or more computer storage media of claim 15 , further comprising generating additional infinite homography matrices for each narrowly overlapping pair of camera images received from the flexible multi-camera array.
17 . A method for generating a single virtual focal plane image mosaic, the method comprising:
receiving from a flexible multi-camera array, a set of camera images comprising a first camera image from a first camera of the flexible multi-camera array, a second camera image from a second camera of the flexible multi-camera array, a third camera image from a third camera of the flexible multi-camera array, and a fourth camera image from a fourth camera of the flexible multi-camera array;
detecting two point correspondences within each narrowly overlapping portion between a first pair of the first camera image and the second camera image, a second pair of the second camera image and the third camera image, a third pair of the third camera image and the fourth camera image, and a fourth pair of the fourth camera image and the first camera image;
generating an infinite homography transformation matrix for each of the first pair, the second pair, the third pair, and the fourth pair of the set of camera images using dynamic homography, wherein the infinite homography transformation matrix comprises 3-degrees of freedom to align each of the first pair, the second pair, the third pair, and the fourth pair of the set of camera images; and
based on the infinite homography transformation matrix, generating the single virtual focal plane image mosaic, wherein the single virtual focal plane image mosaic has a field of view that is larger than a first field of view of the first camera, or a second field of view of the second camera, or a third field of view of the third camera, or a fourth field of view of the fourth camera.
18 . The method of claim 17 , wherein the flexible multi-camera array comprises a 2 by 2 configuration, and wherein the infinite homography transformation matrix is refined through optimization of a relative rotation of each of the first camera, the second camera, the third camera, and the fourth camera in the flexible multi-camera array.
19 . The method of claim 17 , wherein the infinite homography transformation matrix is applied to transform the second camera image into a coordinate system of the first camera image through perspective warping.
20 . The method of claim 19 , wherein transformation of the second camera image includes applying interpolation techniques to estimate pixel values at non-grid positions.