IP Library Granted Patent US 10,904,591
Granted Patent B2
US 10,904,591 · App. 15/710,062 · Granted Jan 26, 2021

Scheme for zero-copy adaptive bitrate video streaming

Inventor: Brian Schmidt (Sunnyvale, CA)
Assignee: Google LLC
H04N21/234381H04N19/132H04N19/146H04N19/184H04N21/2187H04N21/2368H04N21/23406H04N21/24H04N21/6373
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,904,591
App. No.
15/710,062
Granted
Jan 26, 2021
Kind
B2
Abstract

Implementations disclose zero-copy adaptive bitrate video streaming. A method includes capturing, by a user device, a first video frame of a plurality of video frames of a video item to be transmitted as a livestream; delivering the first video frame to an encoder of the user device; capturing a second video frame of the plurality of video frames, the second video frame being captured after the first video frame; delivering the second video frame to the encoder of the user device; in response to determining that the first video frame did not enter the encoder prior to the second video frame arriving at the encoder, discarding the first video frame; determining, by the user device, a frequency of discarded video frames comprising the first video frame; and adjusting, by a processing device of the user device, quality of the video item transmitted as the livestream based on the frequency.

Claims (67)

1. A method comprising:

capturing, by a user device, a first video frame of a plurality of video frames of a video item to be transmitted as a livestream;

delivering the first video frame to a queue for an encoder of the user device;

capturing, by the user device at a frame rate of the user device and a resolution of the user device, a second video frame of the plurality of video frames of the video item to be transmitted as the livestream, the second video frame being captured after the first video frame;

delivering the second video frame to the queue for the encoder of the user device;

in response to determining that the first video frame did not enter the encoder prior to the second video frame arriving at the queue for the encoder, discarding the first video frame without the first video frame having entered the encoder;

in response to determining the second video frame entered the encoder prior to a third video frame arriving at the queue for the encoder, determining that the second video frame is not to be discarded by the user device, and encoding, by the encoder at an encoder bitrate of the encoder, the second video frame to generate an encoded video frame, the third video frame being captured after the second video frame;

determining, by the user device, a frequency of discarded video frames having been discarded without having entered the encoder, the discarded video frames comprising the first video frame; and

responsive to the frequency exceeding a threshold frequency, reducing, by a processing device of the user device, quality of the video item transmitted as the livestream based on the frequency by reducing one or more of the frame rate of the user device, the resolution of the user device, or the encoder bitrate of the encoder.

2. The method of claim 1 , wherein:

the first video frame did not enter the encoder responsive to an initial frame being in the encoder when the second video frame arrived at the queue for the encoder;

the second video frame entered the encoder responsive to the initial frame having been encoded by the encoder prior to the third video frame arriving at the queue for the encoder; and

the method further comprises multiplexing, by the user device, the encoded video frame with encoded audio data to generate multiplexed data, wherein the video item transmitted as the livestream comprises the multiplexed data.

3. The method of claim 2 further comprising:

responsive to the frequency being below a second threshold frequency, increasing the quality of the livestream by increasing one or more of the frame rate of the user device, the resolution of the user device, or the encoder bitrate of the encoder.

4. The method of claim 2 further comprising:

monitoring the plurality of video frames, the encoded audio data, and processing overhead of the user device;

creating metrics based on the monitoring; and

combining the metrics into a unified value, wherein adjusting the quality of the livestream is further based on the unified value.

5. The method of claim 1 , wherein the livestream is an adaptive bitrate (ABR) video stream.

6. The method of claim 1 , wherein transmitting of the livestream and adjusting the livestream is without use of a buffer.

7. A user device comprising:

a memory;

an encoder; and

a processing device coupled to the memory and the encoder, wherein the processing device is to:

capture a first video frame of a plurality of video frames of a video item to be transmitted as a livestream;

deliver the first video frame to a queue for the encoder;

capture, at a frame rate of the user device and a resolution of the user device, a second video frame of the plurality of video frames of the video item to be transmitted as the livestream, the second video frame being captured after the first video frame;

deliver the second video frame to the queue for the encoder of the user device;

discard the first video frame without the first video frame having entered the encoder in response to determining that the first video frame did not enter the encoder prior to the second video frame arriving at the queue for the encoder;

in response to determining the second video frame entered the encoder prior to a third video frame arriving at the queue for the encoder, determine that the second video frame is not to be discarded by the user device, wherein the encoder is to encode the second video frame at an encoder bitrate of the encoder to generate an encoded video frame, the third video frame being captured after the second video frame;

determine a frequency of discarded video frames having been discarded without having entered the encoder, the discarded video frames comprising the first video frame; and

in response to the frequency exceeding a threshold frequency, reduce quality of the video item transmitted as the livestream based on the frequency by reducing one or more of the frame rate of the user device, the resolution of the user device, or the encoder bitrate of the encoder.

8. The user device of claim 7 , wherein:

the first video frame did not enter the encoder responsive to an initial frame being in the encoder when the second video frame arrived at the queue for the encoder;

the second video frame entered the encoder responsive to the initial frame having been encoded by the encoder prior to the third video frame arriving at the queue for the encoder; and

the processing device is further to multiplex the encoded video frame with encoded audio data to generate multiplexed data, wherein the video item transmitted as the livestream comprises the multiplexed data.

9. The user device of claim 8 , wherein the processing device is further to:

responsive to the frequency being below a second threshold frequency, increase the quality of the livestream by increasing one or more of the frame rate of the user device, the resolution of the user device, or the encoder bitrate of the encoder.

10. The user device of claim 8 , wherein the processing device is further to:

monitor the plurality of video frames, the encoded audio data, and processing overhead of the user device to create metrics;

and

combine the metrics into a unified value, wherein adjusting the quality of the livestream is further based on the unified value.

11. The user device of claim 7 , wherein the livestream is an adaptive bitrate (ABR) video stream.

12. The user device of claim 7 , wherein transmitting of the livestream and adjusting the livestream is without use of a buffer.

13. A non-transitory machine-readable storage medium storing instructions which, when executed cause a processing device of a user device to perform operations comprising:

capturing a first video frame of a plurality of video frames of a video item to be transmitted as a livestream;

delivering the first video frame to a queue for an encoder of the user device;

capturing, at a frame rate of the user device and a resolution of the user device, a second video frame of the plurality of video frames of the video item to be transmitted as the livestream, the second video frame being captured after the first video frame;

delivering the second video frame to the queue for the encoder of the user device;

in response to determining that the first video frame did not enter the encoder prior to the second video frame arriving at the queue for the encoder, discarding the first video frame without the first video frame having entered the encoder;

in response to determining the second video frame entered the encoder prior to a third video frame arriving at the queue for the encoder, determining that the second video frame is not to be discarded by the user device, and encoding, by the encoder at an encoder bitrate of the encoder, the second video frame to generate an encoded video frame, the third video frame being captured after the second video frame;

determining, by the user device, a frequency of discarded video frames having been discarded without having entered the encoder, the discarded video frames comprising the first video frame; and

responsive to the frequency exceeding a threshold frequency, reducing, by the processing device, quality of the video item transmitted as the livestream based on the frequency by reducing one or more of the frame rate of the user device, the resolution of the user device, or the encoder bitrate of the encoder.

14. The non-transitory machine-readable storage medium of claim 13 , wherein:

the first video frame did not enter the encoder responsive to an initial frame being in the encoder when the second video frame arrived at the queue for the encoder;

the second video frame entered the encoder responsive to the initial frame having been encoded by the encoder prior to the third video frame arriving at the queue for the encoder; and

the operations further comprising multiplexing, by the user device, the encoded video frame with encoded audio data to generate multiplexed data, wherein the video item transmitted as the livestream comprises the multiplexed data.

15. The non-transitory machine-readable storage medium of claim 14 , wherein the operations further comprise:

responsive to the frequency being below a second threshold frequency, increasing the quality of the livestream by increasing one or more of the frame rate of the user device, the resolution of the user device, or the encoder bitrate of the encoder.

16. The non-transitory machine-readable storage medium of claim 14 , the operations further comprising:

monitoring the plurality of video frames, the encoded audio data, and processing overhead of the user device;

creating metrics based on the monitoring; and

combining the metrics into a unified value, wherein adjusting the quality of the livestream is further based on the unified value.

17. The non-transitory machine-readable storage medium of claim 13 , wherein:

the livestream is an adaptive bitrate (ABR) video stream; and

transmitting of the livestream and adjusting the livestream is without use of a buffer.

Assignments (2)
CHANGE OF NAME Recorded Oct 20, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044567/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 20, 2017
From: SCHMIDT, BRIAN
To: GOOGLE INC.
Reel/Frame 043643/0014 →
Continuity (2)
Provisional Application 62439618 · Dec 28, 2016
Related Publication 20180184142A1 · Jun 28, 2018