IP Library Granted Patent US 10,587,807
Granted Patent B2
US 10,587,807 · App. 15/987,786 · Granted Mar 10, 2020

Systems and methods for stabilizing videos

Inventors: Thomas Derbanne (Paris, FR); César Douady (Les Molières, FR); Maxim Karpushin (San Mateo, CA)
Assignee: GoPro, Inc.
H04N5/23258H04N5/23267H04N5/2252
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,587,807
App. No.
15/987,786
Granted
Mar 10, 2020
Kind
B2
Abstract

Images with an optical field of view may be captured by an image capture device. An observed trajectory of the image capture device reflecting the positions of the image capture device at different moments may be determined. A capture trajectory of the image capture device may be determined based on a subsequent portion of the observed trajectory such that a portion of the capture trajectory corresponding to the one portion of the observed trajectory may be determined based on a subsequent portion of the observed trajectory. Orientations of a capture field of view for the images with respect to the optical field of view of the images may be determined based on the capture trajectory. Video content may be generated based on visual content of the images within the capture field of view.

Claims (44)

1. A system that stabilizes videos, the system comprising:

a housing;

an optical element carried by the housing and configured to guide light within an optical field of view to an image sensor, the optical field of view being greater than a capture field of view for generating video content;

the image sensor carried by the housing and configured to generate an image output signal based on light that becomes incident thereon during a capture duration, the image output signal conveying image information that defines images with the optical field of view;

a position sensor carried by the housing and configured to generate a position output signal based on positions of the housing during the capture duration, the position output signal conveying position information that characterizes positions of the housing at different moments within the capture duration; and

one or more physical processors configured by machine-readable instructions to:

determine an observed trajectory of the housing during the capture duration based on the position information, the observed trajectory reflecting the positions of the housing at the different moments within the capture duration, the observed trajectory including a first portion corresponding to a first moment within the capture duration and a second portion corresponding to a second moment subsequent to the first moment within the capture duration;

determine a capture trajectory of the housing based on a look ahead of the observed trajectory, the capture trajectory reflecting actual and/or virtual positions of the housing from which orientations of the capture field of view are determined, the look head of the observed trajectory including use of a subsequent portion of the observed trajectory to determine a preceding portion of the capture trajectory such that a portion of the capture trajectory corresponding to the first portion of the observed trajectory, which corresponds to the first moment within the capture duration, is determined based on the second portion of the observed trajectory, which corresponds to the second moment within the capture duration, the capture trajectory having smoother changes than the observed trajectory, wherein the capture trajectory of the housing is determined to minimize inter-frame motion while preserving intra-frame motion, wherein the intra-frame motion is preserved based on intra-frame motion of the housing during frame exposure such that the capture trajectory follows the intra-frame motion;

determine the orientations of the capture field of view for the images with respect to the optical field of view of the images based on the capture trajectory of the housing; and

generate the video content based on visual content of the images within the capture field of view.

2. The system of claim 1 , wherein the positions of the housing include rotational positions of the housing.

3. The system of claim 2 , wherein the positions of the housing further include translational positions of the housing.

4. The system of claim 1 , wherein the position sensor includes a gyroscope, an accelerometer, and/or an inertial measurement unit, and the position information is determined independent of the image information.

5. The system of claim 1 , wherein the capture trajectory having smoother changes than the observed trajectory is characterized by the capture trajectory having less jitters than the observed trajectory.

6. The system of claim 1 , wherein the capture trajectory of the housing is determined based on minimization of a rotational velocity of the housing and a rotational acceleration of the housing while respecting a set of constraints.

7. The system of claim 6 , wherein the set of constraints includes a margin constraint determined based on a difference between the optical field of view and the capture field of view.

8. The system of claim 7 , wherein the set of constraints includes a trajectory constraint determined based on the subsequent portion of the observed trajectory.

9. The system of claim 8 , wherein the set of constraints includes a target constraint determined based on positions of a target in the images.

10. A method for stabilizing videos, the method performed by a system including a housing and one or more processors, the housing carrying an optical element, an image sensor, and a position sensor, the optical element configured to guide light within an optical field of view to the image sensor, the optical field of view being greater than a capture field of view for generating video content, the method comprising:

generating, by the image sensor, an image output signal based on light that becomes incident thereon during a capture duration, the image output signal conveying image information that defines images with the optical field of view;

generating, by the position sensor, a position output signal based on positions of the housing during the capture duration, the position output signal conveying position information that characterizes positions of the housing at different moments within the capture duration;

determining, by the one or more processors, an observed trajectory of the housing during the capture duration based on the position information, the observed trajectory reflecting the positions of the housing at the different moments within the capture duration, the observed trajectory including a first portion corresponding to a first moment within the capture duration and a second portion corresponding to a second moment subsequent to the first moment within the capture duration;

determining, by the one or more processors, a capture trajectory of the housing based on a look ahead of the observed trajectory, the capture trajectory reflecting actual and/or virtual positions of the housing from which orientations of the capture field of view are determined, the look head of the observed trajectory including use of a subsequent portion of the observed trajectory to determine a preceding portion of the capture trajectory such that a portion of the capture trajectory corresponding to the first portion of the observed trajectory, which corresponds to the first moment within the capture duration, is determined based on the second portion of the observed trajectory, which corresponds to the second moment within the capture duration, the capture trajectory having smoother changes than the observed trajectory, wherein the capture trajectory of the housing is determined to minimize inter-frame motion while preserving intra-frame motion, wherein the intra-frame motion is preserved based on intra-frame motion of the housing during frame exposure such that the capture trajectory follows the intra-frame motion;

determining, by the one or more processors, the orientations of the capture field of view for the images with respect to the optical field of view of the images based on the capture trajectory of the housing; and

generating, by the one or more processors, the video content based on visual content of the images within the capture field of view.

11. The method of claim 10 , wherein the positions of the housing include rotational positions of the housing.

12. The method of claim 11 , wherein the positions of the housing further include translational positions of the housing.

13. The method of claim 10 , wherein the position sensor includes a gyroscope, an accelerometer, and/or an inertial measurement unit, and the position information is determined independent of the image information.

14. The method of claim 10 , wherein the capture trajectory having smoother changes than the observed trajectory is characterized by the capture trajectory having less jitters than the observed trajectory.

15. The method of claim 10 , wherein the capture trajectory of the housing is determined based on minimization of a rotational velocity of the housing and a rotational acceleration of the housing while respecting a set of constraints.

16. The method of claim 15 , wherein the set of constraints includes a margin constraint determined based on a difference between the optical field of view and the capture field of view.

17. The method of claim 16 , wherein the set of constraints includes a trajectory constraint determined based on the subsequent portion of the observed trajectory.

18. The method of claim 17 , wherein the set of constraints includes a target constraint determined based on positions of a target in the images.

19. A system that stabilizes videos, the system comprising:

a housing;

an optical element carried by the housing and configured to guide light within an optical field of view to an image sensor, the optical field of view being greater than a capture field of view for generating video content;

the image sensor carried by the housing and configured to generate an image output signal based on light that becomes incident thereon during a capture duration, the image output signal conveying image information that defines images with the optical field of view;

a position sensor carried by the housing and configured to generate a position output signal based on positions of the housing during the capture duration, the positions of the housing including rotational positions and/or translational positions of the housing, the position output signal conveying position information that characterizes positions of the housing at different moments within the capture duration, the position information determined independent of the image information; and

one or more physical processors configured by machine-readable instructions to:

determine an observed trajectory of the housing during the capture duration based on the position information, the observed trajectory reflecting the positions of the housing at the different moments within the capture duration, the observed trajectory including a first portion corresponding to a first moment within the capture duration and a second portion corresponding to a second moment subsequent to the first moment within the capture duration;

determine a capture trajectory of the housing based on look ahead of the observed trajectory, the capture trajectory reflecting actual and/or virtual positions of the housing from which orientations of the capture field of view are determined, the look head of the observed trajectory including use of a subsequent portion of the observed trajectory to determine a preceding portion of the capture trajectory such that a portion of the capture trajectory corresponding to the first portion of the observed trajectory, which corresponds to the first moment within the capture duration, is determined based on the second portion of the observed trajectory, which corresponds to the second moment within the capture duration, the capture trajectory having smoother changes than the observed trajectory, wherein the capture trajectory of the housing is determined to minimize inter-frame motion while preserving intra-frame motion, wherein the intra-frame motion is preserved based on intra-frame motion of the housing during frame exposure such that the capture trajectory follows the intra-frame motion;

determine the orientations of the capture field of view for the images with respect to the optical field of view of the images based on the capture trajectory of the housing; and

generate the video content based on visual content of the images within the capture field of view.

20. The system of claim 19 , wherein the capture trajectory of the housing is determined based on minimization of a rotational velocity of the housing and a rotational acceleration of the housing while respecting a set of constraints, the set of constraints including a margin constraint determined based on a difference between the optical field of view and the capture field of view and a trajectory constraint determined based on the subsequent portion of the observed trajectory.

Assignments (6)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY INTEREST Recorded Oct 19, 2020
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 054113/0594 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 18, 2018
From: KARPUSHIN, MAXIM; DERBANNE, THOMAS; DOUADY, CÉSAR
To: GOPRO, INC.
Reel/Frame 047810/0965 →
SECURITY INTEREST Recorded Sep 5, 2018
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 047016/0417 →
Continuity (2)
Provisional Application 62673388 · May 18, 2018
Related Publication 20190356856A1 · Nov 21, 2019
Cited By (4)
US 12,256,147 US 12,284,439 US 12,289,522 US 12,289,523