Methods and apparatus for subspace video stabilization
Methods, apparatus, and computer-readable storage media for subspace video stabilization. A subspace video stabilization technique may provide a robust and efficient approach to video stabilization that achieves high-quality camera motion for a wide range of videos. The technique may transform a set of input two-dimensional (2D) motion trajectories so that they are both smooth and resemble visually plausible views of the imaged scene; this may be achieved by enforcing subspace constraints on feature trajectories while smoothing them. The technique may assemble tracked features in the video into a trajectory matrix, factor the trajectory matrix into two low-rank matrices, and perform filtering or curve fitting in a low-dimensional linear space. The technique may employ a moving factorization technique that is both efficient and streamable.
1. A non-transitory computer-readable storage medium comprising program instructions that, responsive to execution by one or more processors, implement:
factoring two-dimensional (2D) feature trajectories from an input video sequence into a coefficient matrix representing features in the input video sequence and eigen-trajectories representing camera motion over time in the input video sequence;
applying an interactive spline fitting technique to the eigen-trajectories effective to generate smoothed eigen-trajectories that simulate smooth camera motion;
multiplying the smoothed eigen-trajectories that simulate smooth camera motion with the coefficient matrix representing features in the input video sequence effective to generate smoothed 2D feature trajectories; and
warping the input video sequence according to the smoothed 2D feature trajectories to generate a stabilized output video sequence.
2. The non-transitory computer-readable storage medium as recited in claim 1 , wherein the coefficient matrix describes each feature as a linear combination of the eigen-trajectories.
3. The non-transitory computer-readable storage medium as recited in claim 1 , wherein, in said factoring 2D feature trajectories from an input video sequence, the program instructions are further executable by the one or more processors to implement applying a moving factorization technique that repeats performing factorization in a window of k frames of the input video sequence, moving the window forward b frames, and performing factorization in the moved window, where k and b are positive integers, and where k is greater than b so that the factored windows overlap.
4. The non-transitory computer-readable storage medium as recited in claim 3 , wherein at least one eigen-trajectory is kept fixed when said factorization is performed in the moved window.
5. The non-transitory computer-readable storage medium as recited in claim 1 , wherein said warping is performed in accordance with a content-preserving warp technique.
6. A non-transitory computer-readable storage medium comprising program instructions that, responsive to execution by one or more processors, implement a subspace video stabilization module configured to:
apply a two-dimensional (2D) point tracking technique to an input video sequence of a scene to determine 2D trajectories of multiple scene points;
assemble the 2D trajectories into an incomplete trajectory matrix;
perform moving factorization to determine a time-varying subspace approximation of input motion that locally represents the incomplete trajectory matrix as eigen-trajectories representing camera motion over time and a coefficient matrix that describes each feature as a linear combination of the eigen-trajectories;
apply an interactive spline fitting technique to the eigen-trajectories to smooth the input motion while respecting the low rank relationship of the motion of points in the scene, the interactive spline fitting enabling a user to select frames to which a quadratic or cubic spline is applied to smooth the input motion;
multiply the smoothed eigen-trajectories that represent the smooth input motion with the coefficient matrix that describes each feature to generate a set of smoothed 2D trajectories; and
apply a rendering technique to the input video sequence to generate a stabilized output video sequence according to the set of smoothed 2D trajectories.
7. The non-transitory computer-readable storage medium as recited in claim 6 , wherein, in said applying a rendering technique to the input video sequence, the subspace video stabilization module is configured to apply a content-preserving warp technique to the input video sequence.
8. A method, comprising:
factoring two-dimensional (2D) feature trajectories from an input video sequence into a coefficient matrix representing features in the input video sequence and eigen-trajectories representing camera motion over time in the input video sequence;
applying an interactive spline fitting technique to the eigen-trajectories effective to generate smoothed eigen-trajectories that simulate smooth camera motion;
multiplying the smoothed eigen-trajectories that simulate smooth camera motion with the coefficient matrix representing features in the input video sequence effective to generate smoothed 2D feature trajectories; and
warping the input video sequence according to the smoothed 2D feature trajectories to generate a stabilized output video sequence.
9. The method as recited in claim 8 , wherein the coefficient matrix describes each feature as a linear combination of the eigen-trajectories.
10. The method as recited in claim 8 , wherein said factoring 2D feature trajectories from an input video sequence is performed according to a moving factorization technique that repeatedly performs the factorization in a window of k frames of the input video sequence, moving the window forward b frames, and performing factorization in the moved window, where k and b are positive integers, and where k is greater than b so that the factored windows overlap.
11. The method as recited in claim 10 , wherein said moving factorization is performed according to a linear approximation technique.
12. The method as recited in claim 10 , wherein at least one eigen-trajectory is kept fixed when said factorization is performed in the moved window.
13. The method as recited in claim 10 , wherein said warping is performed according to a content-preserving warp technique.
14. The method as recited in claim 8 , further comprising receiving selection of a quadratic spline or a cubic spline, and wherein the interactive spline fitting technique is applied to the eigen-trajectories based on the selection of the quadratic spline or the cubic spline.
15. A system comprising:
at least one processor; and
a memory comprising program instructions that, responsive to execution by the at least one processor, implement a video stabilization module;
factor two-dimensional (2D) feature trajectories from an input video sequence into a coefficient matrix representing features in the input video sequence and eigen-trajectories representing camera motion over time in the input video sequence;
apply an interactive spline fitting technique to the eigen-trajectories effective to generate smoothed eigen-trajectories that simulate smooth camera motion;
multiply the smoothed eigen-trajectories that simulate smooth camera motion with the coefficient matrix representing features in the input video sequence effective to generate smoothed 2D feature trajectories; and
warp the input video sequence according to the smoothed 2D feature trajectories to generate a stabilized output video sequence.
16. The system as recited in claim 15 , wherein the coefficient matrix describes each feature as a linear combination of the eigen-trajectories.
17. The system as recited in claim 15 , wherein, to factor the 2D feature trajectories, the video stabilization module is further implemented to apply a moving factorization technique that repeatedly performs the factorization in a window of k frames of the input video sequence, moving the window forward b frames, and performing factorization in the moved window, where k and b are positive integers, and where k is greater than b so that the factored windows overlap.
18. The system as recited in claim 17 , wherein said factorization is performed according to a linear approximation technique.
19. The system of claim 15 , wherein to apply the interactive spline fitting technique, the video stabilization module is further implemented to:
receive selection of a quadratic spline or cubic spline; and
apply the interactive spline fitting technique based on the selection of the quadratic spline or the cubic spline.