Panoramic video from unstructured camera arrays with globally consistent parallax removal

1. A computer-implemented method for constructing a panoramic video, comprising:

receiving a plurality of input image-frames, wherein the plurality of input image-frames are temporally corresponding frames of respective input video streams;

aligning the plurality of input image-frames to create a plurality of pairs of aligned image-frames and a reference panorama frame;

warping the plurality of pairs of aligned image-frames according to a warping-order to create an intermediate panorama frame including a plurality of warped image-frames, wherein the warping-order is determined based on an overlap-region size corresponding to the plurality of pairs of aligned image-frames;

relaxing the plurality of warped image-frames to create a relaxed panorama frame that is temporally coherent and including a plurality of relaxed image-frames; and

blending the plurality of relaxed image-frames, to create an output panorama frame.

2. The method of claim 1 , wherein the input video streams are captured by cameras positioned in an unstructured configuration.

3. The method of claim 1 , wherein the input video streams are captured by dynamic-cameras, wherein the dynamic-cameras move jointly and relative to each other.

4. The method of claim 1 ,

wherein the warping includes warping a first aligned image-frame of one of the plurality of pairs of aligned image-frames onto a second aligned image-frame of the one of the plurality of pairs of aligned image-frames using an optical-flow to create a first warped image-frame; and

wherein the warping is optimized based on a parallax-warping-error, wherein the parallax-warping-error quantifies differences between the first aligned image-frame and the second aligned image-frame contributed by a plurality of parallax artifacts.

5. The method of claim 4 , wherein the warping further includes comparing a warped patch from the first warped image-frame with a first corresponding patch from the first aligned image-frame, wherein the comparing comprises mapping the warped patch onto the first corresponding patch using a homograpy, wherein the homography is fitted to the optical flow defined within the warped patch.

6. The method of claim 1 , wherein the warping-order is determined based on a graph including nodes, edges, and weights, wherein:

each node is associated with one of the plurality of aligned image-frames;

each edge connects a pair of nodes, wherein the pair of nodes corresponds to the plurality of pairs of aligned image-frames; and

each weight is associated with each edge and is proportional to a warping-quality score associated with the edge-connected pair of nodes.

7. The method of claim 6 , wherein the warping-quality score is a function of an overlap-region between the plurality of aligned image-frames associated with the edge-connected pair of nodes and a parallax-warping-error that quantifies differences within the overlap-region contributed by a plurality of parallax artifacts.

8. The method of claim 1 , wherein the relaxing the plurality of warped image-frames includes

calculating a backward-map that maps an output-pixel from a first warped image frame in the intermediate panorama to a corresponding input-pixel in a first aligned image-frame in the reference panorama and wherein if more than one corresponding input-pixel from more than one aligned image-frames contribute to the output-pixel, the backward-map maps the output-pixel to an average position of the more than one corresponding input-pixels; and

computing a relaxed-map that minimizes an energy function including a distance-measure between the relaxed-map and the backward-map and a smoothness constraint.

9. The method of claim 1 , further comprising cutting at least one program, wherein the at least one program includes a region-of-interest extracted from the output panorama frame.

10. A system for constructing a panoramic video, comprising:

a camera rig, configured to hold a plurality of cameras in an unstructured configuration, wherein the cameras operate to capture input video streams; and

a processor for constructing the panoramic video, wherein the processor receives the input video streams and is configured to:

buffer the input video streams to produce a plurality of input image-frames, wherein the input image-frames are temporally corresponding frames of respective input video streams;

align the plurality of input image-frames to create a plurality of pairs of aligned image-frames and a reference panorama frame;

warp the plurality of pairs of aligned image-frames according to a warping-order to create an intermediate panorama frame including a plurality of warped image-frames, wherein the warping-order is determined based on an overlap-region size corresponding to the plurality of pairs of aligned image-frames;

relax the plurality of warped image-frames to create a relaxed panorama frame that is temporally coherent and including a plurality of relaxed image-frames; and

blend the plurality of relaxed image-frames to create an output panorama frames.

11. The system of claim 10 ,

wherein the warping includes warping a first aligned image-frame of one of the plurality of pairs of aligned image-frames onto a second aligned image-frame of the one of the plurality of pairs of aligned image-frames using an optical-flow to create a first warped image-frame; and

wherein the warping is optimized based on a parallax-warping-error, wherein the parallax-warping-error quantifies differences between the first aligned image-frame and the second aligned image-frame contributed by a plurality of parallax artifacts.

12. The system of claim 11 , wherein the warping further includes comparing a warped patch with a first corresponding patch comprises mapping the warped patch onto the first corresponding patch using a homograpy, wherein the homography is fitted to the optical flow defined within the warped patch.

13. The system of claim 10 , wherein the warping-order is determined based on a graph including nodes, edges, and weights, wherein

each node is associated with one of the plurality of aligned image-frames;

each edge connects a pair of nodes, wherein the pair of nodes corresponds to the plurality of pairs of aligned image-frames; and

each weight is associated with each edge and is proportional to a warping-quality score associated with the edge-connected pair of nodes.

14. The system of claim 13 , wherein the warping-quality score is a function of an overlap-region between the aligned image-frames associated with the edge-connected pair of nodes and a parallax-warping-error that quantifies differences within the overlap-region that are contributed by a plurality of parallax artifacts.

15. The system of claim 10 , wherein the relaxing the warped image-frames includes

calculating a backward-map that maps an output-pixel from a first warped image frame in the intermediate panorama to a corresponding input-pixel in a first aligned image-frame in the reference panorama and wherein if more than one corresponding input-pixel from more than one aligned image-frames contribute to the output-pixel, the backward-map maps the output-pixel to an average position of the more than one corresponding input-pixels; and

computing a relaxed-map that minimizes an energy function including a distance-measure between the relaxed-map and the backward-map and a smoothness constraint.

16. The system of claim 10 , wherein the plurality of cameras are dynamic-cameras, moving jointly and relative to each other.

17. The system of claim 10 , wherein the processor is further configured to preprocess the plurality of input image-frames by compensating for lens distortions and radiometric differences.

18. The system of claim 10 , wherein the processor is further configured to cut at least one program, wherein the at least one program includes region-of-interest extracted from the output panorama frame.

19. A non-transitory computer-readable storage medium storing a set of instructions that is executable by a processor, the set of instructions, when executed by the processor, causing the processor to perform operations comprising:

receiving a plurality of input image-frames, wherein the plurality of input image-frames are temporally corresponding frames of respective input video streams;

aligning the plurality of input image-frames to create a plurality of pairs of aligned image-frames and a reference panorama;

warping the plurality of pairs of aligned image-frames according to a warping-order to create an intermediate panorama frame including a plurality of warped image-frames, wherein the warping-order is determined based on an overlap-region size corresponding to the plurality of pairs of aligned image-frames;

relaxing the plurality of warped image-frames, to create a relaxed panorama frame that is temporally coherent and including a plurality of relaxed image-frames; and

blending the plurality of relaxed image-frames, to create an output panorama frame.