IP Library › Granted Patent US 11,503,099
Granted Patent B2
US 11,503,099 · App. 16/835,587 · Granted Nov 15, 2022

Methods, systems, and media for selecting video formats for adaptive video streaming

Inventors: Yeping Su (Cupertino, CA); Balu Adsumilli (San Francisco, CA)
Assignee: Google LLC
H04L65/80H04L65/75H04N7/01H04N21/442H04L67/61
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,503,099
App. No.
16/835,587
Granted
Nov 15, 2022
Kind
B2
Abstract

Methods, systems, and media for selecting video formats for adaptive video streaming are provided. In some embodiments, the method comprises: receiving an indication of a video to be presented on a user device; identifying a group of quality metrics for each of a plurality of segments of the video, wherein each quality metric includes values for a particular segment and for a particular format of a group of available formats for the video; selecting a first format for a first segment of the video; causing the first segment of the video to be presented on the user device; identifying a quality of a network connection between the user device and a server that hosts the video; identifying a second format for a second segment of the video based on the quality of the network connection; determining whether a format of the video is to be changed from the first format to the second format based at least on the group of quality metrics for the second segment of the video; and, in response to determining that the format of the video is to be changed from the first format to the second format, causing the second segment having the second format to be presented by the user device.

Claims (55)

1. A method for selecting video formats for adaptive video streaming, the method comprising:

receiving, at a server including a hardware processor, an indication of a video to be presented on a user device;

identifying, at the server, information indicating quality metrics corresponding to a plurality of segments of the video;

generating, at the server, by inputting video features for each segment of the video into a first trained model, a group of quality metrics for each segment from the plurality of segments and for each format from a group of available formats for the video, wherein each quality metric for a subsequent segment of the video corresponds to a difference between a quality metric for the subsequent segment and the quality metric for a preceding segment;

generating, at the server, by inputting the video features for each segment of the video into a second trained model, a group of predicted viewership scores for each format from the group of available formats for the video;

selecting, at the server, a first format for a first segment of the video based on the group of quality metrics for the first segment and based on the group of predicted viewership scores for the first segment;

causing, at the server, the first segment of the video to be presented on the user device;

identifying, at the server, a quality of a network connection between the user device and the server that hosts the video during the presentation of the first segment of the video on the user device;

identifying, at the server, a second format for a second segment of the video based on the quality of the network connection;

determining, at the server, whether a format of the video is to be changed from the first format to the second format based at least on the group of quality metrics for the second segment of the video; and

in response to determining that the format of the video is to be changed from the first format to the second format, causing, at the server, the second segment having the second format to be presented by the user device.

2. The method of claim 1 , wherein the group of quality metrics includes data from a sensor used to capture the video, and wherein determining whether the format of the video is to be changed from the first format to the second format is based on the data from the sensor of the camera.

3. The method of claim 2 , wherein the data from the sensor includes motion information that indicates positioning of the camera during capture of a corresponding segment of the video.

4. The method of claim 1 , wherein the group of quality metrics includes analysis data from processing of the video after capture of the video.

5. The method of claim 4 , wherein the analysis data includes activity information associated with content included in a corresponding segment of the video.

6. The method of claim 1 , wherein at least one quality metric in the group of quality metrics includes Just Noticeable Difference (JND) scores that indicate a minimum difference in a value of the quality metric from the first segment of the video to the second segment of the video that is perceptible to a viewer of the video.

7. The method of claim 6 , wherein determining that the format of the video is to be changed from the first format to the second format is based on JND scores for the first format and the second format.

8. The method of claim 1 , wherein determining that the format of the video is to be changed from the first format to the second format comprises is based on the first trained model that takes the group of quality metrics and the quality of the network connection as inputs and generates an output indicating that the format of the video is to be changed.

9. A system for selecting video formats for adaptive video streaming, the system comprising:

a server including a hardware processor that is configured to:

receive, at the server, an indication of a video to be presented on a user device;

identify, at the server, information indicating quality metrics corresponding to a plurality of segments of the video;

generate, at the server, by inputting video features for each segment of the video into a first trained model, a group of quality metrics for each segment from the plurality of segments and for each format from a group of available formats for the video, wherein each quality metric for a subsequent segment of the video corresponds to a difference between a quality metric for the subsequent segment and the quality metric for a preceding segment;

generate, at the server, by inputting the video features for each segment of the video into a second trained model, a group of predicted viewership scores for each format from the group of available formats for the video;

select, at the server, a first format for a first segment of the video based on the group of quality metrics for the first segment and based on the group of predicted viewership scores for the first segment;

cause, at the server, the first segment of the video to be presented on the user device;

identify, at the server, a quality of a network connection between the user device and the server that hosts the video during the presentation of the first segment of the video on the user device;

identify, at the server, a second format for a second segment of the video based on the quality of the network connection;

determine, at the server, whether a format of the video is to be changed from the first format to the second format based at least on the group of quality metrics for the second segment of the video; and

in response to determining that the format of the video is to be changed from the first format to the second format, cause, at the server, the second segment having the second format to be presented by the user device.

10. The system of claim 9 , wherein the group of quality metrics includes data from a sensor used to capture the video, and wherein determining whether the format of the video is to be changed from the first format to the second format is based on the data from the sensor of the camera.

11. The system of claim 10 , wherein the data from the sensor includes motion information that indicates positioning of the camera during capture of a corresponding segment of the video.

12. The system of claim 9 , wherein the group of quality metrics includes analysis data from processing of the video after capture of the video.

13. The system of claim 12 , wherein the analysis data includes activity information associated with content included in a corresponding segment of the video.

14. The system of claim 9 , wherein at least one quality metric in the group of quality metrics includes Just Noticeable Difference (JND) scores that indicate a minimum difference in a value of the quality metric from the first segment of the video to the second segment of the video that is perceptible to a viewer of the video.

15. The system of claim 14 , wherein determining that the format of the video is to be changed from the first format to the second format is based on JND scores for the first format and the second format.

16. The system of claim 9 , wherein determining that the format of the video is to be changed from the first format to the second format comprises is based on the first trained model that takes the group of quality metrics and the quality of the network connection as inputs and generates an output indicating that the format of the video is to be changed.

17. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for selecting video formats for adaptive video streaming, the method comprising:

receiving, at a server including a hardware processor, an indication of a video to be presented on a user device;

identifying, at the server, information indicating quality metrics corresponding to a plurality of segments of the video;

generating, at the server, by inputting video features for each segment of the video into a first trained model, a group of quality metrics for each segment from the plurality of segments and for each format from a group of available formats for the video, wherein each quality metric for a subsequent segment of the video corresponds to a difference between a quality metric for the subsequent segment and the quality metric for a preceding segment;

generating, at the server, by inputting the video features for each segment of the video into a second trained model, a group of predicted viewership scores for each format from the group of available formats for the video;

selecting, at the server, a first format for a first segment of the video based on the group of quality metrics for the first segment and based on the group of predicted viewership scores for the first segment;

causing, at the server, the first segment of the video to be presented on the user device;

identifying, at the server, a quality of a network connection between the user device and the server that hosts the video during the presentation of the first segment of the video on the user device;

identifying, at the server, a second format for a second segment of the video based on the quality of the network connection;

determining, at the server, whether a format of the video is to be changed from the first format to the second format based at least on the group of quality metrics for the second segment of the video; and

in response to determining that the format of the video is to be changed from the first format to the second format, causing, at the server, the second segment having the second format to be presented by the user device.

18. The non-transitory computer-readable medium of claim 17 , wherein the group of quality metrics includes data from a sensor used to capture the video, and wherein determining whether the format of the video is to be changed from the first format to the second format is based on the data from the sensor of the camera.

19. The non-transitory computer-readable medium of claim 18 , wherein the data from the sensor includes motion information that indicates positioning of the camera during capture of a corresponding segment of the video.

20. The non-transitory computer-readable medium of claim 17 , wherein the group of quality metrics includes analysis data from processing of the video after capture of the video.

21. The non-transitory computer-readable medium of claim 20 , wherein the analysis data includes activity information associated with content included in a corresponding segment of the video.

22. The non-transitory computer-readable medium of claim 17 , wherein at least one quality metric in the group of quality metrics includes Just Noticeable Difference (JND) scores that indicate a minimum difference in a value of the quality metric from the first segment of the video to the second segment of the video that is perceptible to a viewer of the video.

23. The non-transitory computer-readable medium of claim 22 , wherein determining that the format of the video is to be changed from the first format to the second format is based on JND scores for the first format and the second format.

24. The non-transitory computer-readable medium of claim 17 , wherein determining that the format of the video is to be changed from the first format to the second format comprises is based on the first trained model that takes the group of quality metrics and the quality of the network connection as inputs and generates an output indicating that the format of the video is to be changed.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 31, 2020
From: SU, YEPING; ADSUMILLI, BALU
To: GOOGLE LLC
Reel/Frame 052271/0095 →
Continuity (1)
Related Publication 20210306401A1 · Sep 30, 2021