IP Library Granted Patent US 9,967,302
Granted Patent B2
US 9,967,302 · App. 14/076,998 · Granted May 8, 2018

Method and system for complexity adaptive streaming

Inventors: Youngkwon Lim (Allen, TX); Zhan Ma (Dallas, TX); Kong Posh Bhat (Plano, TX); Felix C. A. Fernandes (Plano, TX)
Assignee: Samsung Electronics Co., Ltd.
H04L65/4084H04L65/602H04L65/80
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,967,302
App. No.
14/076,998
Granted
May 8, 2018
Kind
B2
Abstract

A method includes calculating a complexity value for each segment or version of multimedia content. The multimedia content has multiple versions associated with common content and different encoding parameters, and each version is divided into segments. The method also includes adaptively streaming the multimedia content to a client device based on at least some of the calculated complexity values. The complexity value for each segment or version is calculated by summing multiple operation complexity values defining complexities of operations for decoding the segment or version. The operation complexity values could include values for a bit parsing operation, a buffer initialization operation, an intra-prediction operation, an inverse quantization and transform operation, a reference block fetch operation, a fractional interpolation operation, and in-loop deblocking operation, a sample adaptive offset correction operation, an adaptive loop filtering operation, and a de-ringing filtering operation.

Claims (152)

1. A method comprising:

calculating a complexity value for each segment or version of multimedia content, the multimedia content having multiple versions associated with common content and different encoding parameters, each version divided into segments; and

adaptively streaming the multimedia content to a client device based on at least some of the calculated complexity values, wherein adaptively streaming the multimedia content comprises:

streaming a first version of the multimedia content to the client device; and

switching to streaming a second version of the multimedia content to the client device based on a power status of the client device;

wherein the complexity value for each segment or version is calculated by summing multiple operation complexity values defining complexities of operations for decoding the segment or version.

2. The method of claim 1 , wherein:

the versions of the multimedia content include different versions associated with different encoding standards; and

the operations for decoding vary between at least some of the encoding standards.

3. The method of claim 2 , wherein the different versions of the multimedia content have different bitrates and require different computational powers to decode.

4. The method of claim 1 , wherein:

all versions are associated with bit parsing, buffer initialization, inverse quantization and transform, and reference block fetch operations;

a first subset of versions are also associated with intra-prediction, fractional interpolation, and in-loop deblocking operations;

a second subset of versions are also associated with intra-prediction, fractional interpolation, in-loop deblocking, and sample adaptive offset correction operations; and

a third subset of versions are also associated with intra-prediction, fractional interpolation, in-loop deblocking, sample adaptive offset correction, adaptive loop filtering, and de-ringing filtering operations.

5. The method of claim 4 , wherein the complexity value for each segment or version is calculated as:

C total =C bit _ parsing +C buf _ init +C intra +C itrans +C ref _ fetch +C intp +C dblk +C sao +C alf +C drf   (1)

where:

C total represents the complexity value for the segment or version;

C bit _ parsing represents the operation complexity value of the bit parsing operation;

C buf _ init represents the operation complexity value of the buffer initialization operation;

C intra represents the operation complexity value of the intra-prediction operation;

C itrans represents the operation complexity value of the inverse quantization and transform operation;

C ref _ fetch represents the operation complexity value of the reference block fetch operation;

C intp represents the operation complexity value of the fractional interpolation operation;

C dblk represents the operation complexity value of the in-loop deblocking operation;

C sao represents the operation complexity value of the sample adaptive offset correction operation;

C alf represents the operation complexity value of the adaptive loop filtering operation; and

C drf represents the operation complexity value of the de-ringing filtering operation.

6. The method of claim 5 , wherein the operation complexity values are calculated as:

C bit _ parsing =k bit ×n bit

C buf _ init =k buf _ init ×n total

C intra =k intra ×n intra

C itrans =k itrans ×( n total −n zero )

C ref _ fetch =k ref _ fetch ×n total

C intp =k intp ×n intp

C dblk =k dblk ×n dblk

C sao =k sao ×n sao

C alf =k alf ×n alf

C drf =k drf ×n drf

where:

n bit represents a number of bits;

n total represents a number of blocks;

n intra represents an intra-block number;

n zero represents a number of zero blocks

n intp represents a number of interpolations;

n dblk represents a number of pixels requiring deblocking;

n sao represents a number of pixels requiring sample adaptive offset correction;

n alf represents a number of blocks requiring adaptive loop filtering;

n drf represents a number of blocks requiring de-ringing filtering; and

k bit , k buf _ init , k intra , k itrans , k ref _ fetch , k intp , k dblk , k sao , k alf , and k drf denote unit operation complexity values.

7. The method of claim 1 , wherein:

streaming the first version comprises streaming the first version in response to a first request from the client device; and

switching to streaming the second version comprises selecting the second version in response to a second request from the client device.

8. The method of claim 1 , further comprising:

providing a list of the different versions of the multimedia content to the client device.

9. An apparatus comprising:

at least one network interface configured to communicate with a client device over a network; and

at least one processing device configured to:

calculate a complexity value for each segment or version of multimedia content, the multimedia content having multiple versions associated with common content and different encoding parameters, each version divided into segments; and

adaptively stream the multimedia content via the at least one network interface to the client device based on at least some of the calculated complexity values, wherein the at least one processing device is configured to adaptively stream the multimedia content by:

streaming a first version of the multimedia content to the client device; and

switching to streaming a second version of the multimedia content to the client device based on a power status of the client device;

wherein the at least one processing device is configured to calculate the complexity value for each segment or version by summing multiple operation complexity values defining complexities of operations for decoding the segment or version.

10. The apparatus of claim 9 , wherein:

the versions of the multimedia content include different versions associated with different encoding standards; and

the operations for decoding vary between at least some of the encoding standards.

11. The apparatus of claim 9 , wherein:

all versions are associated with bit parsing, buffer initialization, inverse quantization and transform, and reference block fetch operations;

a first subset of versions are also associated with intra-prediction, fractional interpolation, and in-loop deblocking operations;

a second subset of versions are also associated with intra-prediction, fractional interpolation, in-loop deblocking, and sample adaptive offset correction operations; and

a third subset of versions are also associated with intra-prediction, fractional interpolation, in-loop deblocking, sample adaptive offset correction, adaptive loop filtering, and de-ringing filtering operations.

12. The apparatus of claim 11 , wherein the at least one processing device is configured to calculate the complexity value for each segment or version as:

C total =C bit _ parsing +C buf _ init +C intra +C itrans +C ref _ fetch +C intp +C dblk +C sao +C alf +C drf

where:

C total represents the complexity value for the segment or version;

C bit _ parsing represents the operation complexity value of the bit parsing operation;

C buf _ init represents the operation complexity value of the buffer initialization operation;

C intra represents the operation complexity value of the intra-prediction operation;

C itrans represents the operation complexity value of the inverse quantization and transform operation;

C ref _ fetch represents the operation complexity value of the reference block fetch operation;

C intp represents the operation complexity value of the fractional interpolation operation;

C dblk represents the operation complexity value of the in-loop deblocking operation;

C sao represents the operation complexity value of the sample adaptive offset correction operation;

C alf represents the operation complexity value of the adaptive loop filtering operation; and

C drf represents the operation complexity value of the de-ringing filtering operation.

13. The apparatus of claim 12 , wherein the at least one processing device is configured to calculate the operation complexity values as:

C bit _ parsing =k bit ×n bit

C buf _ init =k buf _ init ×n total

C intra =k intra ×n intra

C itrans =k itrans ×( n total −n zero )

C ref _ fetch =k ref _ fetch ×n total

C intp =k intp ×n intp

C dblk =k dblk ×n dblk

C sao =k sao ×n sao

C alf =k alf ×n alf

C drf =k drf ×n drf

where:

n bit represents a number of bits;

n total represents a number of blocks;

n intra represents an intra-block number;

n zero represents a number of zero blocks

n intp represents a number of interpolations;

n dblk represents a number of pixels requiring deblocking;

n sao represents a number of pixels requiring sample adaptive offset correction;

n alf represents a number of blocks requiring adaptive loop filtering;

n drf represents a number of blocks requiring de-ringing filtering; and

k bit , k buf _ init , k intra , k itrans , k ref _ fetch , k intp , k dblk , k sao , k alf , and k drf denote unit operation complexity values.

14. The apparatus of claim 9 , wherein the at least one processing device is further configured to provide a list of the different versions of the multimedia content to the client device.

15. A method comprising:

receiving a list of available versions of multimedia content at a client device, the multimedia content having multiple versions associated with common content and different encoding parameters, each version divided into segments;

identifying whether the client device has a specified power status;

when the client device has the specified power status, selecting or switching to a segment or version that consumes less power for decoding; and

receiving and rendering the segment or version of multimedia content;

wherein the selecting or switching comprises selecting or switching to a segment or version having a lower complexity value, the complexity value comprising a sum of multiple operation complexity values defining complexities of operations for decoding the segment or version.

16. The method of claim 15 , wherein the selecting or switching occurs without discontinuing use of a streaming service.

17. The method of claim 15 , wherein the complexity value for each segment or version is defined as:

C total =C bit _ parsing +C buf _ init +C intra +C itrans +C ref _ fetch +C intp +C dblk +C sao +C alf +C drf

where:

C total represents the complexity value for the segment or version;

C bit _ parsing represents the operation complexity value of a bit parsing operation;

C buf _ init represents the operation complexity value of a buffer initialization operation;

C intra represents the operation complexity value of an intra-prediction operation;

C itrans represents the operation complexity value of an inverse quantization and transform operation;

C ref _ fetch represents the operation complexity value of a reference block fetch operation;

C intp represents the operation complexity value of a fractional interpolation operation;

C dblk represents the operation complexity value of an in-loop deblocking operation;

C sao represents the operation complexity value of a sample adaptive offset correction operation;

C alf represents the operation complexity value of an adaptive loop filtering operation; and

C drf represents the operation complexity value of a de-ringing filtering operation.

18. The method of claim 17 , wherein the operation complexity values are defined as:

C bit _ parsing =k bit ×n bit

C buf _ init =k buf _ init ×n total

C intra =k intra ×n intra

C itrans =k itrans ×( n total −n zero )

C ref _ fetch =k ref _ fetch ×n total

C intp =k intp ×n intp

C dblk =k dblk ×n dblk

C sao =k sao ×n sao

C alf =k alf ×n alf

C drf =k drf ×n drf

where:

n bit represents a number of bits;

n total represents a number of blocks;

n intra represents an intra-block number;

n zero represents a number of zero blocks

n intp represents a number of interpolations;

n dblk represents a number of pixels requiring deblocking;

n sao represents a number of pixels requiring sample adaptive offset correction;

n alf represents a number of blocks requiring adaptive loop filtering;

n drf represents a number of blocks requiring de-ringing filtering; and

k bit , k buf _ init , k intra , k itrans , k ref _ fetch , k intp , k dblk , k sao , k alf , and k drf denote unit operation complexity values.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 17, 2014
From: LIM, YOUNGKWON; MA, ZHAN; BHAT, KONG POSH; FERNANDES, FELIX C.A.
To: SAMSUNG ELECTRONICS CO., LTD.
Reel/Frame 031999/0001 →
Continuity (2)
Provisional Application 61726343 · Nov 14, 2012
Related Publication 20140136727A1 · May 15, 2014