IP Library › Granted Patent US 11,343,492
Granted Patent B2
US 11,343,492 · App. 16/820,592 · Granted May 24, 2022

Method for generating video- and audience-specific encoding ladders

Inventors: Nick Chadwick (San Francisco, CA); Adam Brown (San Francisco, CA); Jon Dahl (San Francisco, CA); Masroor Hasan (San Francisco, CA); Ben Dodson (San Francisco, CA)
Assignee: Mux, Inc.
H04N19/115G06K9/6256G06V20/46H04N19/139H04N19/14H04N19/177
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,343,492
App. No.
16/820,592
Granted
May 24, 2022
Kind
B2
Abstract

A method including: extracting a set of video features representing properties of a video segment; generating a set of bitrate-resolution pairs based on the set of video features, each bitrate-resolution pair in the set of bitrate-resolution pairs defining a bitrate and defining a resolution estimated to maximize a quality score characterizing the video segment encoded at the bitrate; accessing a distribution of audience bandwidths; selecting a top bitrate-resolution pair in the set of bitrate-resolution pairs; selecting a bottom bitrate-resolution pair in the set of bitrate-resolution pairs; selecting a subset of bitrate-resolution pairs in the set of bitrate-resolution pairs based on the distribution of audience bandwidths, the subset of bitrate-resolution pairs defining bitrates less than the top bitrate and greater than the bottom bitrate; and generating an encoding ladder for the video segment comprising the top bitrate-resolution pair, the bottom bitrate-resolution pair, and the subset of bitrate-resolution pairs.

Claims (67)

1. A method for generating an encoding ladder comprising:

extracting a set of video features representing properties of a video segment;

generating a feature vector for the video segment based on the set of video features;

passing the feature vector into a convex hull estimation model to generate a set of bitrate-resolution pairs based on the set of video features, each bitrate-resolution pair in the set of bitrate-resolution pairs:

defining a bitrate; and

defining a resolution predicted to yield a quality score, approximating a maximum quality score, characterizing the video segment encoded at the bitrate and the resolution;

accessing a distribution of audience bandwidths representative of likely viewers of the video segment;

selecting a highest bitrate-resolution pair for the encoding ladder from the set of bitrate-resolution pairs, the highest bitrate-resolution pair defining a highest bitrate;

selecting a lowest bitrate-resolution pair for the encoding ladder from the set of bitrate-resolution pairs, the lowest bitrate-resolution pair defining a lowest bitrate;

selecting a subset of bitrate-resolution pairs from the set of bitrate-resolution pairs based on the distribution of audience bandwidths, the subset of bitrate-resolution pairs defining bitrates less than the highest bitrate and greater than the lowest bitrate; and

generating the encoding ladder, for the video segment, comprising the highest bitrate-resolution pair, the lowest bitrate-resolution pair, and the subset of bitrate-resolution pairs.

2. The method of claim 1 , further comprising:

encoding a set of renditions of the video segment, each rendition comprising an encoding of the video segment characterized by a bitrate and a resolution of a bitrate-resolution pair in the encoding ladder; and

publishing a manifest file representing the encoding ladder for an internet stream.

3. The method of claim 1 , further comprising:

accessing a set of training video segments;

for each training video segment in the set of training video segments:

extracting a set of training video features representing properties of the training video segment;

generating a training feature vector for the training video segment based on the set of training video features;

encoding a set of trial encodes of the training video segment, the trial encodes spanning a set of encoding bitrates;

calculating a quality score of each trial encode in the set of trial encodes according to a quality metric to generate a set of quality scores;

for each encoding bitrate in the set of encoding bitrates:

identifying a subset of trial encodes characterized by the encoding bitrate; and

selecting a trial encode for a convex hull of the training video from the subset of trial encodes, the trial encode characterized by a maximum quality score relative to the subset of trial encodes;

generating a training example comprising the training feature vector and the convex hull of the training video segment; and

storing the training example in a set of training examples; and

training the convex hull estimation model based on the set of training examples.

4. The method of claim 1 , wherein selecting the highest bitrate-resolution pair in the set of bitrate-resolution pairs comprises:

for each bitrate-resolution pair in the set of bitrate-resolution pairs, predicting a quality score characterizing the bitrate-resolution pair according to a quality metric;

identifying, in the set of bitrate-resolution pairs, a set of high-quality bitrate-resolution pairs characterized by quality scores greater than a threshold quality score; and

selecting the highest bitrate-resolution pair from the set of high-quality bitrate-resolution pairs, the highest bitrate-resolution pair defining a minimum bitrate in the set of high-quality bitrate resolution pairs.

5. The method of claim 1 , wherein selecting the lowest bitrate-resolution pair in the set of bitrate-resolution pairs comprises:

identifying a threshold percentile bandwidth in the distribution of audience bandwidths; and

selecting a first bitrate-resolution pair defining a bitrate greater than the threshold percentile bandwidth as the lowest bitrate-resolution pair.

6. The method of claim 1 , wherein selecting the subset of bitrate-resolution pairs in the set of bitrate-resolution pairs based on the distribution of audience bandwidths comprises selecting the subset of bitrate resolution pairs predicted to yield an aggregate quality score approximating a maximum aggregate quality score of the video segment viewed according to the distribution of audience bandwidths.

7. A method for generating an encoding ladder comprising:

accessing a set of training video segments;

for each training video segment in the set of training video segments:

extracting a set of training video features representing properties of the training video segment;

encoding a set of trial encodes of the training video segment, the trial encodes spanning a set of encoding bitrates;

calculating a quality score of each trial encode in the set of trial encodes according to a quality metric to generate a set of quality scores;

for each encoding bitrate in the set of encoding bitrates:

identifying a subset of trial encodes characterized by the encoding bitrate; and

selecting a trial encode for a convex hull of the training video from the subset of trial encodes, the trial encode characterized by a maximum quality score relative to the subset of trial encodes;

generating a training example comprising the set of training video features of the training video segment and the convex hull of the training video segment; and

storing the training example in a set of training examples;

training the convex hull estimation model based on the set of training examples;

extracting a set of video features representing properties of a video segment;

passing the set of video features into a convex hull estimation model to generate a set of bitrate-resolution pairs based on the set of video features, each bitrate-resolution pair in the set of bitrate-resolution pairs:

defining a bitrate; and

defining a resolution predicted to yield a quality score, approximating a maximum quality score, characterizing the video segment encoded at the bitrate and the resolution;

accessing a distribution of audience bandwidths representative of likely viewers of the video segment;

selecting a highest bitrate-resolution pair for the encoding ladder from the set of bitrate-resolution pairs, the highest bitrate-resolution pair defining a highest bitrate;

selecting a lowest bitrate-resolution pair for the encoding ladder from the set of bitrate-resolution pairs, the lowest bitrate-resolution pair defining a lowest bitrate;

selecting a subset of bitrate-resolution pairs from the set of bitrate-resolution pairs based on the distribution of audience bandwidths, the subset of bitrate-resolution pairs defining bitrates less than the highest bitrate and greater than the lowest bitrate; and

generating the encoding ladder, for the video segment, comprising the highest bitrate-resolution pair, the lowest bitrate-resolution pair, and the subset of bitrate-resolution pairs.

8. A method for generating an encoding ladder comprising:

extracting a set of video features representing properties of a video segment;

generating a feature vector for the video segment based on the set of video features;

generating a set of bitrate-resolution pairs via a convex hull estimation model and based on the feature vector for the video segment, each bitrate-resolution pair in the set of bitrate-resolution pairs:

defining a bitrate; and

defining a resolution predicted to yield a quality score, approximating a maximum quality score, characterizing the video segment encoded at the bitrate and the resolution;

accessing a distribution of audience bandwidths representative of likely viewers of the video segment;

selecting a highest bitrate-resolution pair for the encoding ladder from the set of bitrate-resolution pairs, the highest bitrate-resolution pair defining a highest bitrate;

selecting a lowest bitrate-resolution pair for the encoding ladder from the set of bitrate-resolution pairs, the lowest bitrate-resolution pair defining a lowest bitrate;

selecting a subset of bitrate-resolution pairs from the set of bitrate-resolution pairs based on the distribution of audience bandwidths, the subset of bitrate-resolution pairs defining bitrates less than the highest bitrate and greater than the lowest bitrate; and

generating the encoding ladder, for the video segment, comprising the highest bitrate-resolution pair, the lowest bitrate-resolution pair, and the subset of bitrate-resolution pairs.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 12, 2020
From: DAHL, JON; BROWN, ADAM; CHADWICK, NICK; DODSON, BEN; HASAN, MASROOR
To: MUX, INC.
Reel/Frame 054354/0169 →
Continuity (2)
Provisional Application 62819277 · Mar 15, 2019
Related Publication 20200296362A1 · Sep 17, 2020
Cited By (1)
US 12,267,376