IP Library Granted Patent US 11,195,253
Granted Patent B2
US 11,195,253 · App. 16/871,762 · Granted Dec 7, 2021

Equatorial stitching of hemispherical images in a spherical image capture system

Inventors: Joseph Steel (London, GB); Timothy Macmillan (La Honda, CA)
G06T3/0018G06T3/0043
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,195,253
App. No.
16/871,762
Granted
Dec 7, 2021
Kind
B2
Abstract

Hyper-hemispherical images may be combined to generate a rectangular projection of a spherical image having an equatorial stitch line along of a line of lowest distortion in the two images. First and second circular images are received representing respective hyper-hemispherical fields of view. A video processing device may project each circular image to a respective rectangular image by mapping an outer edge of the circular image to a first edge of the rectangular image and mapping a center point of the circular image to a second edge of the first rectangular image. The rectangular images may be stitched together along the edges corresponding to the outer edge of the original circular image.

Claims (62)

1. An imaging apparatus configured to generate a planar projection of a spherical image, the imaging apparatus comprising:

a processing apparatus;

a first camera facing a first direction;

a second camera facing a second direction different from the first direction; and

a non-transitory computer-readable apparatus comprising a storage medium, the storage medium comprising computer instructions, the computer instructions configured to, when executed by the processing apparatus, enable the imaging apparatus to:

obtain a first non-planar image via the first camera;

obtain a second non-planar image via the second camera;

map an outer portion associated with the first non-planar image to a bottom portion associated with a first planar image;

map an outer portion associated with the second non-planar image to a top portion associated with a second planar image;

generate the planar projection of the spherical image based at least on a combination of the bottom portion associated with the first planar image with the top portion associated with the second planar image;

divide the planar projection of the spherical image into a first portion associated with a right half of the first camera and a second portion associated with a left half of the first camera; and

rotate and combine the first portion and the second portion such that the right half and the left half form a centered view of the spherical image.

2. The imaging apparatus of claim 1 , wherein the computer instructions are further configured to, when executed by the processing apparatus, enable the imaging apparatus to:

map a center portion associated with the first non-planar image to a top portion associated with the first planar image; and

map a center portion associated with the second non-planar image to a bottom portion associated with the second planar image.

3. The imaging apparatus of claim 2 , wherein the top portion associated with the first planar image and the bottom portion associated with the second planar image each comprises a single pixel.

4. The imaging apparatus of claim 2 , wherein the computer instructions are further configured to, when executed by the processing apparatus, enable the imaging apparatus to:

map one or more inner portions associated with the first non-planar image to one or more rows associated with the first planar image, the one or more inner portions being disposed between the outer and the center portions associated with the first non-planar image, the one or more rows being disposed between the bottom and the top portions associated with the first planar image; and

map another one or more inner portions associated with the second non-planar image to another one or more rows associated with the second planar image, the another one or more inner portions being disposed between the outer and the center portions associated with the second non-planar image, the another one or more rows being disposed between the bottom and the top portions associated with the second planar image.

5. The imaging apparatus of claim 1 , wherein the generation of the planar projection of the spherical image comprises generating a two-dimensional projection associated with a rectangular projection.

6. The imaging apparatus of claim 1 , wherein:

the first camera is configured to capture a first hyper-hemispherical field of view;

the second camera is configured to capture a second hyper-hemispherical field of view; and

the first and second cameras are configured such that first and second hyper-hemispherical fields of view overlap in at least one region by a prescribed degree.

7. The imaging apparatus of claim 1 , wherein the combination comprises (i) alignment of the bottom portion associated with the first planar image with respect to the top portion associated with the second planar image, and (ii) performance of a stitching algorithm on the bottom portion associated with the first planar image and the top portion associated with the second planar image.

8. A method of performing equatorial stitching, comprising:

capturing a first hemispherical video via a first camera of a spherical capture device and a second hemispherical video via a second camera of the spherical capture device, where the first camera and the second camera are characterized by an equatorial region; and

for each frame of the first hemispherical video and the second hemispherical video:

mapping a first hemispherical image to a first rectangular image and a second hemispherical image to a second rectangular image within a processor of the spherical capture device;

stitching the first rectangular image and the second rectangular image into a stitched rectangular image using a stitching algorithm designed for rectangular images, where stitching occurs at the equatorial region;

dividing the stitched rectangular image into a first portion associated with a right half of the first camera and a second portion associated with a left half of the first camera;

rotating and combining the first portion and the second portion such that the right half and the left half form a re-oriented stitched rectangular image comprising a centered view of a spherical image; and

generating a rectangular video from a plurality of re-oriented stitched rectangular images.

9. The method of claim 8 , wherein capturing the first hemispherical video and the second hemispherical video comprises capturing hyper-hemispherical videos that overlap at the equatorial region.

10. The method of claim 9 , further comprising:

mapping the equatorial region of the first hemispherical image to a first bottom edge of the first rectangular image;

mapping a first center point of the first hemispherical image to a first top edge of the first rectangular image;

mapping the equatorial region of the second hemispherical image to a second top edge of the second rectangular image; and

mapping a second center point of the second hemispherical image to a second bottom edge of the second rectangular image.

11. The method of claim 8 , where the stitched rectangular image is further re-oriented by:

combining the first portion and the second portion such that the right half and the left half are contiguous.

12. The method of claim 11 , where the rectangular video is generated according to a desired field of view identified by user control inputs.

13. The method of claim 12 , further comprising transmitting the rectangular video to another device for display.

14. A method for generating a planar projection of a spherical image, the method comprising:

obtaining a first non-planar image via a first camera;

obtaining a second non-planar image via a second camera;

mapping a first longitudinal portion associated with the first non-planar image to a first equatorial portion associated with a first planar image;

mapping a second longitudinal portion associated with the second non-planar image to a second equatorial portion associated with a second planar image;

stitching the first equatorial portion of the first planar image to the second equatorial portion of the second planar image to generate the planar projection;

divide the planar projection into a right half and a left half; and

rotate and combine the right half and the left half to form a centered view of the planar projection.

15. The method of claim 14 , where the first non-planar image is obtained from a first non-planar video and the second non-planar image is obtained from a second non-planar video.

16. The method of claim 14 , further comprising:

mapping a first pixel to a top portion of the first planar image; and

mapping a second pixel to a bottom portion of the second planar image.

17. The method of claim 14 , where mapping the first longitudinal portion comprises a first f-theta projection; and

mapping the first longitudinal portion comprises a second f-theta projection.

18. The method of claim 17 , further comprising calibrating at least one of the first camera and the second camera for a physical offset; and

offsetting at least one center or orientation of the first f-theta projection or the second f-theta projection.

19. The method of claim 14 , where the first longitudinal portion and the second longitudinal portion are characterized by a prescribed degree of overlap.

20. The method of claim 14 , where the first non-planar image comprises a first hyper-hemispherical field of view in a first direction; and

the second non-planar image comprises a second hyper-hemispherical field of view in a second direction different from the first direction.

Assignments (4)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY INTEREST Recorded Oct 19, 2020
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 054113/0594 →
Continuity (5)
Continuation 16450672 · Jun 24, 2019
Continuation 16056370 · Aug 6, 2018
Continuation 15234869 · Aug 11, 2016
Provisional Application 62204290 · Aug 12, 2015
Related Publication 20200372604A1 · Nov 26, 2020