IP Library Granted Patent US 10,178,399
Granted Patent B2
US 10,178,399 · App. 15/156,126 · Granted Jan 8, 2019

Systems and methods of encoding multiple video streams for adaptive bitrate streaming

Inventors: Sam Orton-Jay (Kent, GB); Ivan Vladimirovich Naletov (Tomsk, RU)
Assignee: Sonic IP, Inc.
H04N19/436H04N19/119H04N19/124H04N19/146H04N19/176H04N19/194H04N19/147
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,178,399
App. No.
15/156,126
Granted
Jan 8, 2019
Kind
B2
Abstract

Systems and methods for encoding multiple video streams in accordance with embodiments of the invention are disclosed. In one embodiment, a source encoder configured to encode source video as a number of alternative video streams includes a parallel processing system configured to collect statistics on source video data and write the statistics to shared memory in a first pass through the received multimedia content, determine initial encoding information for source video data and write the initial encoding information to shared memory during the first, encode the source video data in parallel using collected statistics and initial encoding information to produce a plurality of alternative video streams during a second pass, and parallel encoding processes are configured to reuse additional encoding information that has already been determined for a portion of video and to generate additional encoding information that has not already been determined for a portion of video.

Claims (66)

1. A source encoder configured to encode source video as a number of alternative video streams, the source encoder comprising:

memory containing a source encoder application;

a shared memory; and

a parallel processing system configured by the source encoding application to:

receive multimedia content, where the multimedia content comprises source video data having a primary resolution;

collect statistics on source video data and write the statistics to shared memory in a first pass through the received multimedia content;

determine initial encoding information for source video data and write the initial encoding information to shared memory during the first pass through the received multimedia content; and

encode the source video data in parallel using collected statistics and initial encoding information to produce a plurality of alternative video streams during a second pass through the received multimedia content with a plurality of parallel encoding processes, where the encoding of the source video utilizes additional encoding information, the parallel encoding processes are configured to reuse additional encoding information that has already been determined for a portion of video by another parallel encoding process and stored in the shared memory, and the parallel encoding processes are configured to generate additional encoding information that has not already been determined for a portion of video by another parallel encoding process and store the generated additional encoding information in the shared memory;

wherein the parallel encoding processes being configured to generate additional encoding information that has not already been determined for a portion of video by another parallel encoding process further comprises determining a Coding Tree Unit (CTU) size for encoding a portion of a frame of video in the source video data;

wherein determining a CTU size for encoding a portion of a frame of video in the source video data comprises:

selecting a portion of a frame of video in the source video data to encode as an output CTU in a first output stream;

checking if a size has been determined for a similar CTU previously encoded in a second output stream for the selected portion of a frame of video;

comparing the resolution of the first output stream to the resolution of the second output stream if a size has been determined for the similar CTU; and

scaling the CTU size of the similar CTU and applying the scaled CTU size to encoding the portion of the frame of video as the output CTU if the resolution of the first output stream is not the same resolution as the second output stream;

wherein the parallel processing system being configured to determine initial encoding information for source video data further comprises the parallel processing system configured to determine a mode distribution for at least one frame of video in at least one of the plurality of alternative video streams; and

wherein the parallel processing system being configured to encode the source video data in parallel using collected statistics and initial encoding information to produce a plurality of alternative video streams further comprises the parallel processing system being configured to:

maintain a count of blocks processed in a frame of video in an alternative video stream;

determine a threshold number of blocks based on the mode distribution; and

adjust criteria for block type decisions if the count of blocks meets the threshold number of blocks.

2. The source encoder of claim 1 wherein statistics on source video data comprises statistics selected from the group consisting of: average quantization parameter, size of header bits, size of texture bits, number of intra blocks, number of inter blocks, and number of skip blocks.

3. The source encoder of claim 1 wherein the parallel processing system being configured to determine initial encoding information for source video data further comprises the parallel processing system configured to calculate a frame complexity measure.

4. The source encoder of claim 1 wherein the parallel encoding processes being configured to reuse additional encoding information that has already been determined for a portion of video by another parallel encoding process and stored in the shared memory further comprises the parallel encoding processes being configured to:

determine whether a motion vector exists for a second corresponding block in a second alternative stream when encoding a first block in a video frame in a first alternative stream;

determine if the first alternative stream and the second alternative stream are the same resolution;

scale the motion vector if the first alternative stream and the second alternative stream are not the same resolution;

refine the motion vector; and

apply the motion vector in encoding the first block in the video frame in the first alternative stream.

5. The source encoder of claim 1 wherein initial encoding information further comprises header size, macroblock size, and the relative proportion of header size to macroblock size.

6. The source encoder of claim 1 wherein initial encoding information further comprises hypothetical reference decoder data.

7. The source encoder of claim 1 wherein each of the parallel encoding processes encodes at a different resolution.

8. The source encoder of claim 7 wherein each of the parallel encoding processes encodes one or more alternative video streams and each of the alternative video streams encoded by a parallel encoding process is a different bitrate.

9. The source encoder of claim 1 wherein each of the parallel encoding processes encodes a block from the source video data into each stream in a subset of the plurality of alternative video streams sequentially one after another.

10. The source encoder of claim 1 wherein additional encoding information comprises rate distortion information and quantization parameters.

11. A method for encoding source video as a number of alternative video streams, the method comprising:

receiving multimedia content, where the multimedia content comprises source video data having a primary resolution using a source encoder;

collecting statistics on source video data and writing the statistics to shared memory in a first pass through the received multimedia content using a source encoder;

determining initial encoding information for source video data and writing the initial encoding information to a shared memory during the first pass through the received multimedia content using a source encoder; and

encoding the source video data in parallel using collected statistics, initial encoding information, and additional encoding information to produce a plurality of alternative video streams during a second pass through the received multimedia content with a plurality of parallel encoding processes using a source encoder, where encoding of the source video further comprises:

reusing additional encoding information that has already been determined for a portion of video by another parallel encoding process and stored in the shared memory using at least one of the plurality of parallel encoding processes;

generating additional encoding information that has not already been determined for a portion of video by another of the plurality of parallel encoding processes; and

storing the generated additional encoding information in the shared memory using a parallel encoder process;

wherein generating additional encoding information that has not already been determined for a portion of video by another parallel encoding process further comprises determining a Coding Tree Unit (CTU) size for encoding a portion of a frame of video in the source video data; and

wherein determining a CTU size for encoding a portion of a frame of video in the source video data comprises:

selecting a portion of a frame of video in the source video data to encode as an output CTU in a first output stream;

checking if a size has been determined for a similar CTU previously encoded in a second output stream for the selected portion of a frame of video;

comparing the resolution of the first output stream to the resolution of the second output stream if a size has been determined for the similar CTU; and

scaling the CTU size of the similar CTU and applying the scaled CTU size to encoding the portion of the frame of video as the output CTU if the resolution of the first output stream is not the same resolution as the second output stream;

wherein determining initial encoding information for source video data further comprises determining a mode distribution for at least one frame of video in at least one of the plurality of alternative video streams; and

wherein encoding the source video data in parallel using collected statistics, initial encoding information, and additional encoding information to produce a plurality of alternative video streams further comprises:

maintaining a count of blocks processed in a frame of video in an alternative video stream;

determining a threshold number of blocks based on the mode distribution; and

adjusting criteria for block type decisions if the count of blocks meets the threshold number of blocks.

12. The method of claim 11 wherein statistics on source video data comprises statistics selected from the group consisting of: average quantization parameter, size of header bits, size of texture bits, number of intra blocks, number of inter blocks, and number of skip blocks.

13. The method of claim 11 wherein determining initial encoding information for source video data further comprises calculating a frame complexity measure.

14. The method of claim 11 wherein reusing additional encoding information that has already been determined for a portion of video by another parallel encoding process and stored in the shared memory further comprises:

determining whether a motion vector exists for a second corresponding block in a second alternative stream when encoding a first block in a video frame in a first alternative stream;

determining if the first alternative stream and the second alternative stream are the same resolution;

scaling the motion vector if the first alternative stream and the second alternative stream are not the same resolution;

refining the motion vector; and

applying the motion vector in encoding the first block in the video frame in the first alternative stream.

15. The method of claim 11 wherein initial encoding information further comprises header size, macroblock size, and the relative proportion of header size to macroblock size.

16. The method of claim 11 wherein initial encoding information further comprises hypothetical reference decoder data.

17. The method of claim 11 wherein each of the parallel encoding processes encodes at a different resolution.

18. The source encoder of claim 17 wherein each of the parallel encoding processes encodes one or more alternative video streams and each of the alternative video streams encoded by a parallel encoding process is a different bitrate.

19. The method of claim 11 wherein each of the parallel encoding processes encodes a block from the source video data into each stream in a subset of the plurality of alternative video streams sequentially one after another.

20. The method of claim 11 wherein additional encoding information comprises rate distortion information and quantization parameters.

Assignments (5)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 11, 2021
From: SONIC IP, LLC
To: NLD HOLDINGS I, LLC
Reel/Frame 055238/0786 →
ENTITY CONVERSION Recorded Jan 22, 2021
From: SONIC IP, INC.
To: SONIC IP, LLC
Reel/Frame 055086/0960 →
ENTITY CONVERSION Recorded Jan 22, 2021
From: SONIC IP, LLC
To: SONIC IP, LLC
Reel/Frame 055089/0821 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 30, 2017
From: ORTON-JAY, SAM; NALETOV, IVAN
To: DIVX, LLC
Reel/Frame 041801/0982 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 30, 2017
From: DIVX, LLC
To: SONIC IP, INC.
Reel/Frame 041802/0015 →
Continuity (3)
Continuation 13830914 · Mar 14, 2013
Provisional Application 61771024 · Feb 28, 2013
Related Publication 20160261879A1 · Sep 8, 2016