IP Library Granted Patent US 12,302,000
Granted Patent B2
US 12,302,000 · App. 17/744,949 · Granted May 13, 2025

Systems and methods for horizon leveling videos

Inventors: Daryl Stimm (Encinitas, CA); Kyler William Schwartz (Valley Center, CA); Jonathan Leland Thorn (San Leandro, CA)
Assignee: GoPro, Inc.
H04N23/695G06T3/40G06T5/80H04N23/53H04N23/632H04N23/635
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,302,000
App. No.
17/744,949
Granted
May 13, 2025
Kind
B2
Abstract

A video may be captured by an image capture device in motion. A horizon-leveled view of the video may be generated by providing a punchout of the video. The punchout of the video may compensate for rotation of the image capture device during capture of the video. The placement of the punchout of the video may be changed based on different rotational positions of to provide a view in which a horizon depicted within the video is leveled.

Claims (54)

1. An image capture device for horizon leveling videos, the image capture device comprising:

a housing;

an optical element carried by the housing and configured to guide light to an image sensor;

the image sensor carried by the housing and configured to generate a visual output signal based on light that becomes incident thereon during a capture duration, the visual output signal conveying visual information that defines visual content having a field of view;

a position sensor carried by the housing and configured to generate a position output signal, the position output signal conveying rotational position information that characterizes rotational positions of the image capture device during the capture duration; and

one or more physical processors carried by the housing and configured by machine-readable instructions to:

during the capture duration, determine placement of a viewing window within the field of view of the visual content based on the rotational positions of the image capture device during the capture duration, the viewing window defining extents of the visual content within the field of view; and

generate horizon-leveled video frames of a video to include the extents of the visual content defined by the viewing window, wherein inclusion of the extents of the visual content defined by the viewing window within the horizon-leveled video frames effectuates horizon leveling of the visual content by the image capture device during capture of the visual content.

2. The image capture device of claim 1 , wherein the viewing window has a one-by-one aspect ratio, the one-by-one aspect ratio reducing impact of an off-axis horizon depicted within the horizon-leveled video frames.

3. The image capture device of claim 1 , wherein the visual content includes a distortion such that a straight line within a scene depicted within the visual content appears as a curved line, the distortion of the visual content reducing impact of an off-axis horizon depicted within the horizon-leveled visual content.

4. The image capture device of claim 3 , wherein the visual content includes the distortion based on capture of the visual content through a wide field of view optical element.

5. The image capture device of claim 1 , wherein:

the visual content is originally captured by the image capture device non- distorted such that a straight line within a scene depicted within the originally captured visual content appears straight; and

a distortion is applied by the image capture device to the originally captured visual content to cause the straight line within the scene to appear curved, wherein application of the distortion reduces the impact of an off-axis horizon depicted within the distorted visual content.

6. The image capture device of claim 1 , wherein the determination of the placement of the viewing window includes determination of a size or a shape of the viewing window, further wherein separate sizes or separate shapes of the viewing window are determined for the visual content captured at different moments during the capture duration.

7. The image capture device of claim 6 , wherein the separate sizes or the separate shapes of the viewing window determined for the visual content captured at the different moments during the capture duration include different sizes or different shapes of the viewing window.

8. The image capture device of claim 1 , wherein the rotational position information characterizing the rotational positions of the image capture device includes the rotational position information including information on direction of gravity on the image capture device.

9. The image capture device of claim 1 , wherein the placement of the viewing window is determined further based on rotation of the visual content performed by electronic image stabilization.

10. The image capture device of claim 1 , further comprising a display, wherein:

a preview of the visual content captured by the image capture device is presented on the display; and

an outline of the placement of the viewing window within the field of view of the visual content is presented within the preview of the visual content presented on the display.

11. The image capture device of claim 1 , wherein the rotational position information is synchronized to video frames captured by the image capture device.

12. The image capture device of claim 11 , wherein the rotational position information is synchronized to the video frames captured by the image capture device by including information on the rotational positions of the image capture device with information on time stamps per the visual content captured at different moments during the capture duration.

13. The image capture device of claim 1 , wherein the placement of the viewing window for the visual content captured at a moment during the capture duration is determined further based on the placement of the viewing window for the visual content captured at neighboring moments during the capture duration.

14. A method for horizon leveling videos, the method performed by a an image capture device including an optical element, an image sensor, a position sensor, and one or more processors, the optical element configured to guide light to the image sensor, the image sensor configured to generate a visual output signal based on light that becomes incident thereon during a capture duration, the visual output signal conveying visual information that defines visual content having a field of view, the position sensor configured to generate a position output signal, the position output signal conveying rotational position information that characterizes rotational positions of the image capture device during the capture duration, the method comprising:

during the capture duration, determining, by the one or more processors, placement of a viewing window within the field of view of the visual content based on the rotational positions of the image capture device during the capture duration, the viewing window defining extents of the visual content within the field of view; and

generating, by the one or more processors, horizon-leveled video frames of a video to include the extents of the visual content defined by the viewing window, wherein inclusion of the extents of the visual content defined by the viewing window within the horizon-leveled video frames effectuates horizon leveling of the visual content by the image capture device during capture of the visual content.

15. The method of claim 14 , wherein the viewing window has a one-by-one aspect ratio, the one-by-one aspect ratio reducing impact of an off-axis horizon depicted within the horizon-leveled video frames.

16. The method of claim 14 , wherein the visual content includes a distortion such that a straight line within a scene depicted within the visual content appears as a curved line, the distortion of the visual content reducing impact of an off-axis horizon depicted within the horizon-leveled visual content.

17. The method of claim 16 , wherein the visual content includes the distortion based on capture of the visual content through a wide field of view optical element.

18. The method of claim 14 , wherein:

the visual content is originally captured by the image capture device non-distorted such that a straight line within a scene depicted within the originally captured visual content appears straight; and

a distortion is applied by the image capture device to the originally captured visual content to cause the straight line within the scene to appear curved, wherein application of the distortion reduces the impact of an off-axis horizon depicted within the distorted visual content.

19. The method of claim 14 , wherein determining the placement of the viewing window includes determining a size or a shape of the viewing window, further wherein separate sizes or separate shapes of the viewing window are determined for the visual content captured at different moments during the capture duration.

20. The method of claim 19 , wherein the separate sizes or the separate shapes of the viewing window determined for the visual content captured at the different moments during the capture duration include different sizes or different shapes of the viewing window.

21. The method of claim 14 , wherein the rotational position information characterizing the rotational positions of the image capture device includes the rotational position information including information on direction of gravity on the image capture device.

22. The method of claim 14 , wherein the placement of the viewing window is determined further based on rotation of the visual content performed by electronic image stabilization.

23. The method of claim 14 , wherein:

the image capture device further comprises a display;

a preview of the visual content captured by the image capture device is presented on the display; and

an outline of the placement of the viewing window within the field of view of the visual content is presented within the preview of the visual content presented on the display.

24. The method of claim 14 , wherein the rotational position information is synchronized to video frames captured by the image capture device.

25. The method of claim 24 , wherein the rotational position information is synchronized to the video frames captured by the image capture device by including information on the rotational positions of the image capture device with information on time stamps per the visual content captured at different moments during the capture duration.

26. The method of claim 14 , wherein the placement of the viewing window for the visual content captured at a moment during the capture duration is determined further based on the placement of the viewing window for the visual content captured at neighboring moments during the capture duration.

27. An image capture device for horizon leveling videos, the image capture device comprising:

a housing;

an optical element carried by the housing and configured to guide light to an image sensor;

the image sensor carried by the housing and configured to generate a visual output signal based on light that becomes incident thereon during a capture duration, the visual output signal conveying visual information that defines visual content having a field of view;

a position sensor carried by the housing and configured to generate a position output signal, the position output signal conveying rotational position information that characterizes rotational positions of the image capture device during the capture duration, wherein the rotational position information characterizing the rotational positions of the image capture device includes the rotational position information including information on direction of gravity on the image capture device; and

one or more physical processors carried by the housing and configured by machine-readable instructions to:

one or more physical processors carried by the housing and configured by machine-readable instructions to:

during the capture duration, determine placement of a viewing window within the field of view of the visual content based on the rotational positions of the image capture device during the capture duration and based on rotation of the visual content performed by electronic image stabilization, the viewing window defining extents of the visual content within the field of view; and

generate horizon-leveled video frames of a video to include the extents of the visual content defined by the viewing window, wherein inclusion of the extents of the visual content defined by the viewing window within the horizon-leveled video frames effectuates horizon leveling of the visual content by the image capture device during capture of the visual content.

28. The image capture device of claim 27 , wherein the determination of the placement of the viewing window includes determination of a size or a shape of the viewing window further wherein separate sizes or separate shapes of the viewing window are determined for the visual content captured at different moments during the capture duration.

Assignments (4)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
CORRECTIVE ASSIGNMENT TO CORRECT THE SPELLING OF THE THIRD INVENTOR'S NAME PREVIOUSLY RECORDED ON REEL 059916 FRAME 0413. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded May 27, 2022
From: STIMM, DARYL; SCHWARTZ, KYLER WILLIAM; THORN, JONATHAN LELAND
To: GOPRO, INC.
Reel/Frame 060204/0740 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 16, 2022
From: STIMM, DARYL; SCHWARTZ, KYLER WILLIAM; THORN, JONATHAN LELAN
To: GOPRO, INC.
Reel/Frame 059916/0413 →
Continuity (3)
Continuation 17006536 · Aug 28, 2020
Provisional Application 62894649 · Aug 30, 2019
Related Publication 20220279128A1 · Sep 1, 2022
References Cited (58)
US 7567752B2 · Klinghult · 2009 [cited by applicant]
US 9055222B2 · Lin · 2015 [cited by applicant]
US D789435S · Nguyen · 2017 [cited by applicant]
US 9787902B1 · Beysserie · 2017 [cited by applicant]
US 10015413B1 · Woodman · 2018 [cited by applicant]
US 10341564B1 · Derbanne · 2019 [cited by applicant]
US 10462370B2 · Shi · 2019 [cited by applicant]
US 10529052B2 · Newman · 2020 [cited by applicant]
US 10574894B2 · Derbanne · 2020 [cited by applicant]
US 10735653B1 · Huang · 2020 [cited by applicant]
US 10778896B2 · Watanabe · 2020 [cited by applicant]
US 10958840B2 · Douady · 2021 [cited by applicant]
US 20040257384A1 · Park · 2004 [cited by examiner]
US 20050168583A1 · Thomason · 2005 [cited by applicant]
US 20150130963A1 · Tanaka · 2015 [cited by applicant]
US 20150181123A1 · Pacurariu · 2015 [cited by applicant]
US 20150215532A1 · Jafarzadeh · 2015 [cited by examiner]
US 20150254871A1 · MacMillan · 2015 [cited by examiner]
US 20160088287A1 · Sadi · 2016 [cited by examiner]
US 20160283097A1 · Voss · 2016 [cited by examiner]
US 20170118458A1 · Grönholm · 2017 [cited by examiner]
US 20170142337A1 · Kokaram · 2017 [cited by examiner]
US 20180160194A1 · Bayliss · 2018 [cited by examiner]
US 20190189160A1 · Huang · 2019 [cited by examiner]
US 20240348927A1 · Stimm · 2024 [cited by applicant]
CN 102845053 · 2012 [cited by applicant]
JP 2018006996A · 2018 [cited by applicant]
WO 2015038180A1 · 2015 [cited by applicant]
WO 2015183636A1 · 2015 [cited by applicant]
WO 2018053785A1 · 2018 [cited by applicant]
Video Retrieval Methods Using Geographic Information in Windows Azure Cloud (Year: 2014). [cited by applicant]
B. Mayo, “Horizon for iPhone Always Records Landscape Video, Independent of Device Orientation,” https://9to5mac.com/2014/01/14/horizon-for-iphone-always-records-landscape-video-independent-of-device-orientation/, 9to5M… [cited by applicant]
Declaration of Dr. Aggelos Katsaggelos, dated Jun. 20, 2024, 276 pages. [cited by applicant]
Exemplary Infringement Claim Chart for U.S. Pat. No. 11,336,832, In the Matter of Certain Cameras, Camera Systems, and Accessories Used Therewith, Inv. No. 337-TA-1400, Mar. 29, 2024. 18 pages. [cited by applicant]
K. Vanhemert, “Horizon App Solves the Dumbest Thing About Smartphone Video,” https://www.wired.com/2014/01/camera-app-solves-single-dumbest-thing-smartphone-video/, Wired (Jan. 22, 2014.) 3 pages. [cited by applicant]
Mar. 29, 2024 Letter to U.S. International Trade Commission Letter providing Complainant's (GoPro's) Initial Statement on the Public Interest, Complaint Under Section 337 of the Tariff Act of 1930, as Amended, and other… [cited by applicant]
Mayo, Benjamin, Horizon for iPhone always records landscape video, independent of device oreientation, Jan. 14, 2014, 2 pages. [cited by applicant]
Petition for Inter Partes Review Under 25 U.S.C. Sections 311-319 and 37 C.F.R. Section 42.100 et seq., dated Jun. 20, 2024, 103 pages. [cited by applicant]
Petitioner's Power of Attorney Under 37 C.F.R. Section 42.10(b), dated Jun. 20, 2024, 3 pages. [cited by applicant]
Prosecution History of U.S. Pat. No. 11,336,832, 179 pages. [cited by applicant]
Ben Claremont, Insta360 Go: The Bite Sized steady-cam!, YouTube (Aug. 28, 2019), https://www.youtube.com/watch?v=Hm0TYd3JXo8. [cited by applicant]
Daven Mathies, GoPro Hero (2018) Review, Digital Trends (Apr. 19, 2018), https://www.digitaltrends.com/photography/gopro-hero-2018-review/. [cited by applicant]
Daven Mathies, Insta360 Go review: Everyday life becomes a (boring) movie, DigitalTrends (Aug. 29, 2019), https://www.digitaltrends.com/photography/insta360-go-review/. [cited by applicant]
GoPro Fusion /GoPro Fusion Studio (e.g., Photoxels, GoPro: Fusion Spherical Camera Captures Immersive 5.2K Spherical Content, (Sep. 29, 2017), PHOTOXELS https://www.photoxels.com/gopro-fusion-spherical-camera-captures-i… [cited by applicant]
GoPro HERO (2018 Edition) (e.g., GoPro, GoPro Launches Entry-Level Hero Camera 2018, GoPRo (Mar. 29, 2018), https://gopro.com/en/US/news/gopro-launches-entry-level-hero-camera-2018. [cited by applicant]
GoPro HERO5 Black / Ambarella (e.g., HERO5Black_UM_ENG_REVD_Web.pdf; GoPro Hero 5 with Image Stabilization—Does it Work for Mountain Biking?, https://www.youtube.com/watch?v=5EIOX8Ptlz8; https://gethypoxic.com/blogs/tec… [cited by applicant]
GoPro HERO6 Black (e.g., HERO6Black Um Eng REVB.pdf, GoPro Hero 6 review_ Slow-mo, stabilization and subtle refinements.pdf; GoPro Hero 6 Stabilization // Enough to leave the gimbal at home?, https://www.youtube.com/wat… [cited by applicant]
GoPro, Introducing Fusion Studio, GoPRo (Nov. 21, 2017), https://gopro.com/en/au/news/introducing-fusion-studio. [cited by applicant]
Horizon App (e.g., INSTA360 1400-PA00001830, INSTA360 1400-PA00001776, INSTA360 1400-PA00001790, INSTA360 1400-PA00001817). [cited by applicant]
Insta360 Go (1.2.0) (e.g., Insta360, Downloads | Insta360 GO, https://www.insta360.com/download/insta360-go. [cited by applicant]
Insta360 Nano (e.g., INSTA360 1400-PA00002724, INSTA360 1400-PA00002722, INSTA360 1400-PA00002729). [cited by applicant]
Insta360 One (e.g., INSTA360 1400-PA00002777, INSTA360 1400-PA00002779, INSTA360 1400-PA00002799, INSTA360 1400-PA00002792). [cited by applicant]
Jim Martin, GoPro Hero Review (2018), Tech Advisor (May 18, 2018), https://www.techadvisor.com/article/719094/gopro-hero-revi ew-2018.html). [cited by applicant]
Michael Zhang, ReelSteady Aims to Take Video Stabilization Software to the Next Level, PETAPIXEL (Aug. 27, 2015), https://petapixel.com/2015/08/27/reelsteady-aims-to-take-video-stabilization-software-to-the-next-level/). [cited by applicant]
Mike McCarthy, Review GoPro Fusion 360 camera, POSTPERSPECTIVE (Nov. 30, 2017), https://postperspective.com/review-gopro-fusion-360-camera/ (“McCarthy”)). [cited by applicant]
ReelSteady (e.g., ReelSteady, Version 1.0 out now!, REELSTEADY (Apr. 28, 2016), https://www.reelsteady.com/blogs/news/105201414-version-1-0-out-now. [cited by applicant]
Reelsteady, New version!, REELSTEADY (Mar. 7, 2017), https://www.reelsteady.com/blogs/news/tagged/build89. [cited by applicant]
V. Renee, Meet ReelSteady, a Video Stabilization Plugin That Really Does Its Job, NOFILMSCHOOL (Aug. 15, 2015), https://nofilmschool.com/2015/08/meet-reelsteady-video-stabilization-plugin-really-does-its-job. [cited by applicant]
Cited By (1)
US 12,531,960