Granted Patent B2

US 8,384,787 · App. 13/034,010 · Granted Feb 26, 2013

Abstract

Claims (49)

1. A method for providing a stabilized digital video sequence, comprising:

receiving a digital video sequence of a scene captured using a digital video camera, the digital video sequence having a time sequence of image frames, each image frame having a two-dimensional array of image pixels in response to activation of a user interface control;

analyzing at least a portion of a plurality of the image frames captured at different times to determine a corresponding one-dimensional image frame representation having an associated spatial dimension;

combining the one-dimensional frame representations for the plurality of image frames to form a two-dimensional spatiotemporal representation of the video sequence, one of the dimensions being a time dimension and the other dimension being the spatial dimension associated with the one-dimensional image frame representations;

identifying a set of trajectories in the two-dimensional spatiotemporal representation of the video sequence, each trajectory corresponding to a structure in the two-dimensional spatiotemporal representation of the video sequence and representing the spatial position of a feature in the two-dimensional spatiotemporal representation of the video sequence as a function of time;

analyzing the set of trajectories to identify a set of foreground trajectory segments representing foreground motion information and a set of background trajectory segments representing background motion information;

analyzing the background trajectory segments to estimate a motion pattern for the digital video camera;

analyzing the motion pattern for the digital video camera to determine an undesired motion portion;

applying spatial shifts to at least some of the image frames of the digital video sequence to provide a stabilized digital video sequence, wherein the spatial shifts are determined in response to the determined undesired motion portion to substantially counteract the unintended camera shaking motion; and

storing the stabilized digital video sequence in a processor-accessible memory.

2. The method of claim 1 wherein the one-dimensional image frame representation is determined by applying an integral transform to at least a portion of the corresponding image frame.

3. The method of claim 2 wherein the integral transform includes summing the image pixels along a specified set of integration paths through the image frame.

4. The method of claim 2 wherein the integral transform includes summing the image pixels along rows or columns of the image frame.

5. The method of claim 1 wherein the one-dimensional image frame representation is determined by extracting a trace through the corresponding image frame.

6. The method of claim 1 wherein the process of identifying the set of trajectories includes:

determining a two-dimensional edge map identifying a set of edge pixels by applying an edge detection operation to the two-dimensional spatiotemporal representation of the video sequence; and

analyzing the two-dimensional edge map to identify a plurality of trajectories, each trajectory corresponding to a set of related edge pixels.

7. The method of claim 6 wherein the set of related edge pixels are connected edge pixels.

8. The method of claim 1 wherein the set of background trajectory segments and the set of foreground trajectory segments are identified by:

forming a parametric model relating the spatial positions of the trajectories at two different times;

designating portions of the trajectories where a difference between the actual spatial positions of the trajectories and the spatial positions predicted by the parametric model is less than a specified background threshold to be background trajectory segments; and

designating portions of the trajectories where the difference between the actual spatial positions of the trajectories and the spatial positions predicted by the parametric model is greater than a specified foreground threshold to be foreground trajectory segments.

9. The method of claim 8 wherein the parametric model is analyzed to estimate the motion pattern for the digital video camera.

10. The method of claim 9 wherein the parametric model is adjusted to removed the effect of the foreground trajectory segments before analyzing the parametric model to estimate the motion pattern for the digital video camera.

11. The method of claim 9 wherein the motion pattern includes a panning motion, a zooming motion, or a shaking motion, or a combination thereof.

12. The method of claim 1 further including analyzing the background trajectory segments to estimate a motion pattern for the digital video camera.

13. The method of claim 12 wherein the motion pattern includes a panning motion, a zooming motion, or a shaking motion, or a combination thereof.

14. The method of claim 1 wherein the undesired motion portion is determined by applying a low-pass filter to the motion pattern and computing a difference between the motion pattern and the low-pass filtered motion pattern.

15. The digital video camera of claim 1 wherein the undesired motion portion corresponds to an unintended camera shaking motion.

16. The method of claim 1 wherein the spatial shifts include horizontal spatial shifts determined using one-dimensional frame representations for a horizontal spatial direction and vertical spatial shifts determined using one-dimensional frame representations for a vertical spatial direction.

17. The method of claim 1 wherein the image pixels have a plurality of color values corresponding to a plurality of color channels.

18. The method of claim 17 wherein the one-dimensional frame representations are determined using color values from a single color channel.

19. The method of claim 17 wherein the one-dimensional frame representations are determined using a combined color values determined by combining the color values for a plurality of the color channels.

20. The method of claim 19 wherein the combined color channel is a luminance color channel.

21. A digital video camera providing a stabilized digital video sequence, comprising:

an image sensor for capturing a digital image;

an optical system for forming an image of a scene onto the image sensor;

a data processing system;

a storage memory for storing captured digital video sequences; and

a program memory communicatively connected to the data processing system and storing instructions configured to cause the data processing system to implement a method for providing a stabilized digital video sequence, wherein the instructions include:

capturing a digital video sequence of a scene, the digital video sequence having a time sequence of image frames, each image frame having a two-dimensional array of image pixels in response to activation of a user interface control;

analyzing at least a portion of a plurality of the image frames captured at different times to determine a corresponding one-dimensional image frame representation having an associated spatial dimension;

combining the one-dimensional frame representations for the plurality of image frames to form a two-dimensional spatiotemporal representation of the video sequence, one of the dimensions being a time dimension and the other dimension being the spatial dimension associated with the one-dimensional image frame representations;

identifying a set of trajectories in the two-dimensional spatiotemporal representation of the video sequence, each trajectory corresponding to a structure in the two-dimensional spatiotemporal representation of the video sequence and representing the spatial position of a feature in the two-dimensional spatiotemporal representation of the video sequence as a function of time;

analyzing the set of trajectories to identify a set of foreground trajectory segments representing foreground motion information and a set of background trajectory segments representing background motion information;

analyzing the background trajectory segments to estimate a motion pattern for the digital video camera;

analyzing the motion pattern for the digital video camera to determine an undesired motion portion;

applying spatial shifts to at least some of the image frames of the digital video sequence to provide a stabilized digital video sequence, wherein the spatial shifts are determined in response to the determined undesired motion portion to substantially counteract the unintended camera shaking motion; and

storing the stabilized digital video sequence in the storage memory.

Assignments (13)