IP Library Granted Patent US 10,484,664
Granted Patent B2
US 10,484,664 · App. 16/208,327 · Granted Nov 19, 2019

Mapping of spherical image data into rectangular faces for transport and decoding across networks

Inventors: Timothy Macmillan (La Honda, CA); Adeel Abbas (Carlsbad, CA)
Assignee: GoPro, Inc.
H04N13/243G06K9/4609G06K9/52G06K9/6201G06K9/6212G06T3/0043G06T3/0068G06T3/20G06T7/0002G06T7/38G06T7/55G06T7/593G06T7/73G06T11/60G06T15/20H04N5/23238H04N5/265H04N13/10H04N13/282H04N13/302H04N13/334H04N13/344H04N19/136H04N19/44G06T2200/04G06T2207/20221H04N13/161
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,484,664
App. No.
16/208,327
Granted
Nov 19, 2019
Kind
B2
Abstract

A system captures a first hemispherical image and a second hemispherical image, each hemispherical image including an overlap portion, the overlap portion capturing a same field of view, the two hemispherical images collectively comprising a spherical FOV and separated along a longitudinal plane. The system maps a modified first hemispherical image to a first portion of the 2D projection of a cubic image, the modified first hemispherical image including a non-overlap portion of the first hemispherical image, and maps a modified second hemispherical image to a second portion of the 2D projection of the cubic image, the modified second hemispherical image also including a non-overlap portion. The system maps the overlap portions of the first hemispherical image and the second hemispherical image to the 2D projection of the cubic image, and encodes the 2D projection of the cubic image to generate an encoded image representative of the spherical FOV.

Claims (75)

1. A method for mapping spherical images to a two-dimensional projection of a cubic representation of a spherical field of view (FOV), the method comprising:

capturing a first hyper-hemispherical image and a second hyper-hemispherical image, each of the first hyper-hemispherical image and the second hyper-hemispherical image comprising an overlap portion, the overlap portions capturing a same area within the spherical FOV;

modifying the first hyper-hemispherical image, the modifying comprising removing the overlap portion of the first hyper-hemispherical image, the modifying of the first hyper-hemispherical image resulting in a modified first hemispherical image;

modifying the second hyper-hemispherical image, the modifying comprising removing the overlap portion of the second hyper-hemispherical image, the modifying of the second hyper-hemispherical image resulting in a modified second hemispherical image;

mapping a first portion of the modified first hemispherical image into first, second and third facets of the two-dimensional projection of the cubic representation, the first, the second and the third facets of the two-dimensional projection of the cubic representation defining a first extent of contiguous space within the spherical FOV;

mapping a first portion of the modified second hemispherical image into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV;

mapping a first portion of the overlap portions into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV; and

encoding the two-dimensional projection of the cubic representation of the spherical FOV to generate an encoded image representative of the spherical FOV.

2. The method of claim 1 , further comprising:

mapping a second portion of the modified first hemispherical image into a fourth facet of the two-dimensional projection of the cubic representation;

mapping a second portion of the modified second hemispherical image into the fourth facet of the two-dimensional projection of the cubic representation; and

mapping a remainder portion of the overlap portions into the fourth facet of the two-dimensional projection of the cubic representation.

3. The method of claim 2 , further comprising arranging the fourth facet of the two-dimensional projection of the cubic representation with the first, the second, and the third facets of the two-dimensional projection of the cubic representation, the arranging defining a second extent of contiguous space within the spherical FOV, the second extent of contiguous space including the first extent of contiguous space within the spherical FOV.

4. The method of claim 3 , further comprising mapping a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation;

mapping a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation; and

arranging the fifth and the sixth facet of the two-dimensional projection of the cubic representation with the second facet of the two-dimensional projection of the cubic representation such that the second, the fifth, and the sixth facets are defining a third extent of contiguous space within the spherical FOV.

5. The method of claim 2 , further comprising rotating the fourth facet of the two-dimensional projection of the cubic representation; and

arranging the rotated fourth facet of the two-dimensional projection of the cubic representation adjacent the second facet of the two-dimensional projection of the cubic representation, a boundary between the rotated fourth facet and the second facet comprising a non-contiguous extent of space within the spherical FOV.

6. The method of claim 5 , further comprising mapping a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation; and

mapping a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation.

7. The method of claim 6 , further comprising arranging the fifth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation; and

arranging the sixth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation such that the fourth, the fifth and the sixth facets of the two-dimensional projection of the cubic representation comprises a second extent of contiguous space within the spherical FOV.

8. A computer readable apparatus comprising a storage medium adapted to store a computer program thereon, the computer program being configured to, when executed, cause a computerized apparatus to:

retrieve a first hyper-hemispherical image and a second hyper-hemispherical image, each of the first hyper-hemispherical image and the second hyper-hemispherical image comprising an overlap portion, the overlap portions capturing a same area within a spherical field of view (FOV);

modify the first hyper-hemispherical image, the modification comprising removal of the overlap portion of the first hyper-hemispherical image, the modification of the first hyper-hemispherical image resulting in a modified first hemispherical image;

modify the second hyper-hemispherical image, the modification comprising removing the overlap portion of the second hyper-hemispherical image, the modification of the second hyper-hemispherical image resulting in a modified second hemispherical image;

map a first portion of the modified first hemispherical image into first, second and third facets of a two-dimensional projection of a cubic representation, the first, the second and the third facets of the two-dimensional projection of the cubic representation defining a first extent of contiguous space within the spherical FOV;

map a first portion of the modified second hemispherical image into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV;

map a first portion of the overlap portions into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV; and

encode the two-dimensional projection of the cubic representation of the spherical FOV to generate an encoded image representative of the spherical FOV.

9. The computer readable apparatus of claim 8 , wherein the computer program, when executed, is further configured to cause the computerized apparatus to:

map a second portion of the modified first hemispherical image into a fourth facet of the two-dimensional projection of the cubic representation;

map a second portion of the modified second hemispherical image into the fourth facet of the two-dimensional projection of the cubic representation; and

map a remainder portion of the overlap portions into the fourth facet of the two-dimensional projection of the cubic representation.

10. The computer readable apparatus of claim 9 , wherein the computer program is further configured to, when executed, cause the computerized apparatus to:

arrange the fourth facet of the two-dimensional projection of the cubic representation with the first, the second, and the third facets of the two-dimensional projection of the cubic representation, the arrangement defining a second extent of contiguous space within the spherical FOV, the second extent of contiguous space including the first extent of contiguous space within the spherical FOV.

11. The computer readable apparatus of claim 10 , wherein the computer program is further configured to, when executed, cause the computerized apparatus to:

map a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation;

map a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation; and

arrange the fifth and the sixth facet of the two-dimensional projection of the cubic representation with the second facet of the two-dimensional projection of the cubic representation such that the second, the fifth, and the sixth facets are defining a third extent of contiguous space within the spherical FOV.

12. The computer readable apparatus of claim 9 , wherein the computer program is further configured to, when executed, cause the computerized apparatus to:

rotate the fourth facet of the two-dimensional projection of the cubic representation; and

arrange the rotated fourth facet of the two-dimensional projection of the cubic representation adjacent the second facet of the two-dimensional projection of the cubic representation, a boundary between the rotated fourth facet and the second facet comprising a non-contiguous extent of space within the spherical FOV.

13. The computer readable apparatus of claim 12 , wherein the computer program is further configured to, when executed, cause the computerized apparatus to:

map a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation; and

map a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation.

14. The computer readable apparatus of claim 13 , wherein the computer program is further configured to, when executed, cause the computerized apparatus to:

arrange the fifth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation; and

arrange the sixth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation such that the fourth, the fifth and the sixth facets of the two-dimensional projection of the cubic representation comprises a second extent of contiguous space within the spherical FOV.

15. An image processing system for mapping spherical images to a two-dimensional projection of a cubic representation of a spherical field of view (FOV), the image processing system configured to:

retrieve a first hyper-hemispherical image and a second hyper-hemispherical image from memory, each of the first hyper-hemispherical image and the second hyper-hemispherical image comprising an overlap portion, the overlap portions capturing a same area within a spherical field of view (FOV);

modify the first hyper-hemispherical image, the modification comprising removal of the overlap portion of the first hyper-hemispherical image, the modification of the first hyper-hemispherical image resulting in a modified first hemispherical image;

modify the second hyper-hemispherical image, the modification comprising removing the overlap portion of the second hyper-hemispherical image, the modification of the second hyper-hemispherical image resulting in a modified second hemispherical image;

map a first portion of the modified first hemispherical image into first, second and third facets of the two-dimensional projection of the cubic representation, the first, the second and the third facets of the two-dimensional projection of the cubic representation defining a first extent of contiguous space within the spherical FOV;

map a first portion of the modified second hemispherical image into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV;

map a first portion of the overlap portions into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV; and

encode the two-dimensional projection of the cubic representation of the spherical FOV to generate an encoded image representative of the spherical FOV.

16. The image processing system of claim 15 , the image processing system further configured to:

map a second portion of the modified first hemispherical image into a fourth facet of the two-dimensional projection of the cubic representation;

map a second portion of the modified second hemispherical image into the fourth facet of the two-dimensional projection of the cubic representation; and

map a remainder portion of the overlap portions into the fourth facet of the two-dimensional projection of the cubic representation.

17. The image processing system of claim 16 , the image processing system further configured to:

arrange the fourth facet of the two-dimensional projection of the cubic representation with the first, the second, and the third facets of the two-dimensional projection of the cubic representation, the arrangement defining a second extent of contiguous space within the spherical FOV, the second extent of contiguous space including the first extent of contiguous space within the spherical FOV.

18. The image processing system of claim 17 , the image processing system further configured to:

map a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation;

map a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation; and

arrange the fifth and the sixth facet of the two-dimensional projection of the cubic representation with the second facet of the two-dimensional projection of the cubic representation such that the second, the fifth, and the sixth facets are defining a third extent of contiguous space within the spherical FOV.

19. The image processing system of claim 16 , the image processing system further configured to:

rotate the fourth facet of the two-dimensional projection of the cubic representation; and

arrange the rotated fourth facet of the two-dimensional projection of the cubic representation adjacent the second facet of the two-dimensional projection of the cubic representation, a boundary between the rotated fourth facet and the second facet comprising a non-contiguous extent of space within the spherical FOV.

20. The image processing system of claim 19 , the image processing system further configured to:

map a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation;

map a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation;

arrange the fifth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation; and

arrange the sixth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation such that the fourth, the fifth and the sixth facets of the two-dimensional projection of the cubic representation comprises a second extent of contiguous space within the spherical FOV.

Assignments (5)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY INTEREST Recorded Oct 19, 2020
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 054113/0594 →
SECURITY INTEREST Recorded Mar 5, 2019
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 048508/0728 →