IP Library Granted Patent US 11,089,359
Granted Patent B1
US 11,089,359 · App. 16/740,700 · Granted Aug 10, 2021

Systems and methods for persisting in-band metadata within compressed video files

Inventor: Ioannis Katsavounidis (Sane Jose, CA)
Assignee: Facebook, Inc.
H04N21/4402H04N21/434H04N21/4621
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,089,359
App. No.
16/740,700
Granted
Aug 10, 2021
Kind
B1
Abstract

The disclosed computer-implemented method may include (1) receiving, at a computing device, a compressed video bitstream that includes (A) video data and (B) in-band metadata that indicates a video quality of the compressed video bitstream, (2) extracting, at the computing device, the in-band metadata from the compressed video bitstream, (3) determining, based at least in part on the in-band metadata, the video quality of the compressed video bitstream, and then (4) making at least one video-processing decision in connection with the compressed video bitstream based at least in part on the video quality of the compressed video bitstream. Various other systems and methods are also disclosed.

Claims (99)

1. A computer-implemented method comprising:

receiving, at a computing device, a compressed video bitstream that includes:

video data; and

in-band metadata that indicates a video quality of the compressed video bitstream;

extracting, at the computing device, the in-band metadata from the compressed video bitstream, wherein the in-band metadata comprises a first-generation quality metric that represents the video quality of the compressed video bitstream upon being encoded by a first encoding operation;

performing, on the compressed video bitstream, a transcoding operation comprising:

a decoding operation that decodes the compressed video bitstream; and

a second encoding operation that re-encodes the compressed video bitstream;

producing a second-generation quality metric that represents the video quality of the compressed video bitstream upon being encoded by the second encoding operation;

retaining the first-generation quality metric in the in-band metadata included in the compressed video bitstream;

embedding the second-generation quality metric in the in-band metadata by prepending the first-generation and second-generation quality metrics to a beginning of the compressed video bitstream;

determining, based at least in part on the in-band metadata, the video quality of the compressed video bitstream; and

making at least one video-processing decision in connection with the compressed video bitstream based at least in part on the video quality of the compressed video bitstream.

2. The computer-implemented method of claim 1 , further comprising:

processing the video data included in the compressed video bitstream in accordance with the video-processing decision; and

providing the processed video data for visual presentation on a display device.

3. The computer-implemented method of claim 1 ,

further comprising forwarding the compressed video bitstream to an additional computing device.

4. The computer-implemented method of claim 1 , wherein embedding the second-generation quality metric in the in-band metadata comprises:

locating an in-band position of the compressed video bitstream that has been designated for the in-band metadata; and

inserting the second-generation quality metric along with the first-generation quality metric into the compressed video bitstream at the designated in-band position.

5. The computer-implemented method of claim 1 , wherein the video-processing decision comprises at least one of:

selecting at least one encoding parameter to apply as part of an encoding operation performed on the video data included in the compressed video bitstream;

selecting at least one decoding parameter to apply as part of a decoding operation performed on the video data included in the compressed video bitstream; or

selecting a certain bitrate at which the compressed video bitstream is delivered to a display device.

6. The computer-implemented method of claim 1 , wherein extracting the in-band metadata from the compressed video bitstream comprises:

searching a designated portion of the compressed video bitstream for the in-band metadata; and

during the search, identifying the in-band metadata at the designated portion of the compressed video bitstream.

7. The computer-implemented method of claim 1 , further comprising identifying at least one attribute of a display device associated with the computing device; and

wherein making the video-processing decision in connection with the compressed video bitstream comprises making the video-processing decision based at least in part on the video quality of the compressed video bitstream and the attribute of the display device.

8. The computer-implemented method of claim 7 , wherein:

the attribute of the display device comprises a resolution of the display device; and

the video-processing decision comprises selecting a certain bitrate at which the compressed video bitstream is delivered to the display device based at least in part on the resolution of the display device.

9. The computer-implemented method of claim 7 , wherein:

the attribute of the display device comprises a resolution of the display device; and

the video-processing decision comprises performing, in response to identifying the resolution of the display device, a scaling operation on the video data included in the compressed video bitstream to conform the video data to the resolution of the display device.

10. The computer-implemented method of claim 1 , further comprising identifying, within the in-band metadata, multiple generations of quality metrics that represent the video quality of the compressed video bitstream after undergoing different encoding or decoding operations; and

wherein determining the video quality of the compressed video bitstream comprises:

calculating an amount of degradation across the multiple generations of quality metrics; and

determining, based at least in part on the amount of degradation across the multiple generations of quality metrics, the video quality of the compressed video bitstream.

11. The computer-implemented method of claim 1 , further comprising identifying, within the in-band metadata, multiple generations of quality metrics that represent the video quality of the compressed video bitstream after undergoing different encoding or decoding operations; and

wherein determining the video quality of the compressed video bitstream comprises:

comparing the multiple generations of quality metrics with one another; and

determining, based at least in part on the comparison, the video quality of the compressed video bitstream.

12. The computer-implemented method of claim 1 , wherein the in-band metadata further indicates an impairment of the compressed video bitstream;

further comprising identifying, within the in-band metadata, the impairment of the compressed video bitstream; and

wherein making the video-processing decision in connection with the compressed video bitstream comprises making the video-processing decision based at least in part on the video quality of the compressed video bitstream and the impairment of the compressed video bitstream.

13. The computer-implemented method of claim 1 , wherein the in-band metadata comprises at least one of:

a name of a quality metric;

a value of the quality metric; or

a scale flag that indicates whether the quality metric is scaled or non-scaled.

14. The computer-implemented method of claim 1 , wherein the in-band metadata comprises at least one of:

a peak signal-to-noise ratio (PSNR) of the compressed video bitstream;

a structural similarity (SSIM) index of the compressed video bitstream;

a variance inflation factor (VIF) of the compressed video bitstream;

a detail loss metric (DLM) of the compressed video bitstream;

a video multimethod assessment fusion (VMAF) value of the compressed video bitstream;

a multi-scale SSIM (MS-SSIM) index of the compressed video bitstream;

a motion-tuned video integrity evaluation (MOVIE) index of the compressed video bitstream;

a video quality metric (VQM) of the compressed video bitstream;

a blind/referenceless image spatial quality evaluator (BRISQUE) value of the compressed video bitstream;

a feature maps based referenceless image quality evaluation engine (FRIQUEE) value of the compressed video bitstream; or

a blockiness indicator of the compressed video bitstream.

15. A system comprising:

a receiving module, stored in memory, that receives a compressed video bitstream that includes:

video data; and

in-band metadata that indicates a video quality of the compressed video bitstream;

an extraction module, stored in memory, that extracts the in-band metadata from the compressed video bitstream, wherein the in-band metadata comprises a first-generation quality metric that represents the video quality of the compressed video bitstream upon being encoded by a first encoding operation;

a processing module, stored in memory, that performs, on the compressed video bitstream, a transcoding operation comprising:

a decoding operation that decodes the compressed video bitstream; and

a second encoding operation that re-encodes the compressed video bitstream;

a generation module, stored in memory, that produces a second-generation quality metric that represents the video quality of the compressed video bitstream upon being encoded by the second encoding operation;

a metadata module, stored in memory, that:

retains the first-generation quality metric in the in-band metadata included in the compressed video bitstream; and

embeds the second-generation quality metric in the in-band metadata by prepending the first-generation and second-generation quality metrics to a beginning of the compressed video bitstream;

a determination module, stored in memory, that determines, based at least in part on the in-band metadata, the video quality of the compressed video bitstream;

a decision-making module, stored in memory, that makes at least one video-processing decision in connection with the compressed video bitstream based at least in part on the video quality of the compressed video bitstream; and

at least one physical processor configured to execute the receiving module, the extraction module, the processing module, the generation module, the metadata module, the determination module, and the decision-making module.

16. The system of claim 15 , wherein the processing module processes the video data included in the compressed video bitstream in accordance with the video-processing decision; and

further comprising a presentation module, stored in memory, that providing the processed video data for visual presentation on a display device; and

wherein the physical processor is further configured to execute the processing module and the presentation module.

17. The system of claim 15 ,

further comprising a forwarding module, stored in memory, that forwards the compressed video bitstream to an additional computing device.

18. The system of claim 15 , wherein, to embed the second-generation quality metric in the in-band metadata, the metadata module:

locates an in-band position of the compressed video bitstream that has been designated for the in-band metadata; and

inserts the second-generation quality metric along with the first-generation quality metric into the compressed video bitstream at the designated in-band position.

19. A non-transitory, computer-readable medium comprising computer-readable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

receive a compressed video bitstream that includes:

video data; and

in-band metadata that indicates a video quality of the compressed video bitstream;

extract the in-band metadata from the compressed video bitstream, wherein the in-band metadata comprises a first-generation quality metric that represents the video quality of the compressed video bitstream upon being encoded by a first encoding operation;

perform, on the compressed video bitstream, a transcoding operation comprising:

a decoding operation that decodes the compressed video bitstream; and

a second encoding operation that re-encodes the compressed video bitstream;

produce a second-generation quality metric that represents the video quality of the compressed video bitstream upon being encoded by the second encoding operation;

retain the first-generation quality metric in the in-band metadata included in the compressed video bitstream;

embed the second-generation quality metric in the in-band metadata by prepending the first-generation and second-generation quality metrics to a beginning of the compressed video bitstream;

determine, based at least in part on the in-band metadata, the video quality of the compressed video bitstream; and

make at least one video-processing decision in connection with the compressed video bitstream based at least in part on the video quality of the compressed video bitstream.

Assignments (2)
CHANGE OF NAME Recorded Jan 27, 2022
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058871/0336 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 28, 2020
From: KATSAVOUNIDIS, IOANNIS
To: FACEBOOK, INC.
Reel/Frame 051966/0365 →