IP Library Granted Patent US 10,652,558
Granted Patent B2
US 10,652,558 · App. 16/234,214 · Granted May 12, 2020

Apparatus and methods for video compression using multi-resolution scalable coding

Inventors: Adeel Abbas (Carlsbad, CA); Balineedu Chowdary Adsumilli (San Francisco, CA); David Newman (San Diego, CA)
Assignee: GoPro, Inc.
H04N19/30H04N19/146H04N19/167H04N19/187H04N19/29H04N19/59
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,652,558
App. No.
16/234,214
Granted
May 12, 2020
Kind
B2
Abstract

Apparatus and methods for digital video data compression via a scalable, multi-resolution approach. In one embodiment, the video content may be encoded using a multi-resolution and/or multi-quality scalable coding approach that reduces computational and/or energy load on a client device. In one implementation, a low fidelity image is obtained based on a first full resolution image. The low fidelity image may be encoded to obtain a low fidelity bitstream. A second full resolution image may be obtained based on the low fidelity bitstream. A portion of a difference image obtained based on the second full resolution image and the first full resolution may be encoded to obtain a high fidelity bitstream. The low fidelity bitstream and the high fidelity bitstream may be provided to e.g., a receiving device.

Claims (58)

1. A method of obtaining an image responsive to a change of a viewport, the method comprising:

obtaining a difference image based at least on a first full-resolution image and a second full-resolution image;

encoding a portion of the difference image to produce an encoded portion, the encoded portion corresponding to a position of the viewport; and

providing, via a communications interface to a client device, at least the encoded portion corresponding to the position of the viewport.

2. The method of claim 1 , further comprising:

obtaining the first full-resolution image from a sequence of images, the first full-resolution having a first frame resolution;

downsampling the first full-resolution image to obtain a downsampled image having a second frame resolution, the second frame resolution being lower than the first frame resolution; and

encoding the downsampled image into a reduced-fidelity bitstream.

3. The method of claim 2 , further comprising selecting an image resolution of the second frame resolution of the downsampled image based at least on a scaling factor ranging from two (2) to sixteen (16), inclusive.

4. The method of claim 2 , further comprising:

decoding the reduced-fidelity bitstream into a reduced-fidelity image; and

upsampling the reduced-fidelity image to obtain the second full-resolution image;

wherein the encoding of the portion of the difference image comprises generating an increased-fidelity bitstream; and

wherein the providing the at least encoded portion of the difference image comprises providing at least the increased-fidelity bitstream to the client device.

5. The method of claim 3 , wherein the providing the at least encoded portion of the difference image comprises providing the reduced-fidelity bitstream to the client device along with the increased-fidelity bitstream.

6. The method of claim 1 , wherein the obtaining of the difference image comprises subtracting values of pixels of the second full-resolution image from pixels of the first full-resolution image for each of one or more image components.

7. The method of claim 1 , further comprising modifying a bitrate for the at least encoded portion of the difference image based on a scalability parameter.

8. A non-transitory computer-readable apparatus comprising a storage medium, the storage medium including a computer program comprising a plurality of instructions which are configured to, when executed by one or more processor apparatus, cause a computerized apparatus to:

obtain a residual image based on a first full-resolution image and a second full-resolution image;

detect a portion of the obtained residual image, the portion being viewed by a user;

encode the portion of the obtained residual image;

produce a high-fidelity bitstream based at least on the encode of the portion of the obtained residual image; and

provide at least the produced high-fidelity bitstream to a client device of the user.

9. The non-transitory computer-readable apparatus of claim 8 , wherein the plurality of instructions are further configured to, when executed by the one or more processor apparatus, cause the computerized apparatus to:

downsample the first full-resolution image to obtain a downsampled image having a resolution lower than that of the first full-resolution image; and

encode the downsampled image into a low-fidelity bitstream;

wherein a number of bits per pixel of the high-fidelity bitstream is greater than that of the low-fidelity bitstream.

10. The non-transitory computer-readable apparatus of claim 9 , wherein the plurality of instructions are further configured to, when executed by the one or more processor apparatus, cause the computerized apparatus to:

determine the frame resolution of the downsampled image based at least on a downsampling factor selected from a range of two (2) and sixteen (16), inclusive.

11. The non-transitory computer-readable apparatus of claim 8 , wherein the plurality of instructions are further configured to, when executed by the one or more processor apparatus, cause the computerized apparatus to:

decode the low-fidelity bitstream into a low-fidelity image; and

upsample the low-fidelity image to obtain the second full-resolution image.

12. The non-transitory computer-readable apparatus of claim 11 , wherein the upsample of the low-fidelity image comprises a reciprocation of a downsampling factor, the downsampling factor being selected from a range of two (2) and sixteen (16), inclusive.

13. The non-transitory computer-readable apparatus of claim 8 , wherein the residual image is obtained based on a subtraction of values of pixels of the second full-resolution image from pixels of the first full-resolution image for each of one or more image components.

14. The non-transitory computer-readable apparatus of claim 8 , wherein the plurality of instructions are further configured to, when executed by the one or more processor apparatus, cause the computerized apparatus to:

obtain a plurality of source images from one or more capture devices; and

combine the plurality of source images to obtain the first full-resolution image.

15. A computerized apparatus for providing video content, the computerized apparatus comprising:

a processing apparatus;

a network interface in data communication with the processing apparatus;

a non-transitory computer-readable apparatus comprising a storage medium, the storage medium including a computer program comprising a plurality of instructions which are configured to, when executed by the processing apparatus, cause a computerized apparatus to:

derive a difference image based at least on a first image and a second image;

encode a portion of the derived difference image to generate an encoded portion, the encoded portion corresponding to a position of a viewport as reflected in data received from at least one client device via the network interface; and

provide, to the network interface for distribution to the at least one client device, at least the encoded portion of the difference image corresponding to the position of the viewport.

16. The computerized apparatus of claim 15 , wherein the plurality of instructions are further configured to, when executed by the processing apparatus, cause the computerized apparatus to:

downsample the first image to obtain a downsampled image having a second frame resolution, the second frame resolution being lower than a first frame resolution of the first image; and

encode the downsampled image into a first bitstream.

17. The computerized apparatus of claim 16 , wherein the plurality of instructions are further configured to, when executed by the processing apparatus, cause the computerized apparatus to:

decode the first bitstream into a reduced-fidelity image; and

upsample the reduced-fidelity image to obtain the second image;

wherein the encode of the portion of the difference image comprises generation of a second bitstream; and

wherein the provision the at least encoded portion of the difference image comprises provision of at least the high-fidelity bitstream to the at least one client device.

18. The computerized apparatus of claim 17 , wherein the provision the at least encoded portion of the difference image comprises provision of the first bitstream along with the second bitstream to the at least one client device.

19. The computerized apparatus of claim 15 , further comprising a storage device;

wherein the plurality of instructions are further configured to, when executed by the processing apparatus, cause the computerized apparatus to:

cause storage of the at least encoded portion of the difference image on at least one of (i) the storage device of the computerized apparatus or (ii) a networked remote storage device.

20. The computerized apparatus of claim 15 , wherein the plurality of instructions are further configured to, when executed by the processing apparatus, cause the computerized apparatus to:

obtain the first image based on one or more source images acquired from one or more capture devices.

Assignments (5)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY INTEREST Recorded Oct 19, 2020
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 054113/0594 →
SECURITY INTEREST Recorded Mar 5, 2019
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 048508/0728 →
Continuity (5)
Continuation 15799837 · Oct 31, 2017
Division 15250748 · Aug 29, 2016
Provisional Application 62289105 · Jan 29, 2016
Provisional Application 62310550 · Mar 18, 2016
Related Publication 20190246126A1 · Aug 8, 2019
Cited By (1)
US 12,389,013