Methods, systems, and apparatuses for adaptive bitrate ladder construction based on dynamically adjustable neural networks
Methods, systems, and apparatuses are described for streaming content. The embodiments described herein may apply to ABR streaming and output an optimized ABR ladder enabling a playback device to maximize the perceived video quality of received content, provide substantially constant visual quality, and/or minimize rebuffering content. The playback device may request an ABR representation associated with an ABR ladder that has been optimized based on one or more neural networks. The one or more neural networks may be pre-trained based on one or more properties and characteristics/features of the video content. The one or more neural networks may process the video content to dynamically determine parameters such as the bitrates of each ABR representation for the content. The parameters may be used to output the encoded ABR representations.
1 . A method comprising:
receiving content to be displayed by a computing device;
determining one or more conditions associated with a network;
selecting, based on the one or more conditions, one or more neural networks from a plurality of neural networks pre-trained to output parameters used to optimize an encoding bitrate for a representation of the content; and
encoding, based on one or more parameters output by the selected one or more neural networks, the representation of the content.
2 . The method of claim 1 , wherein the selecting comprises adjusting the selected one or more neural networks, wherein the adjustment comprises one or more of:
removing one or more neural network layers of the selected one or more neural networks,
adjusting one or more weights associated with one or more neurons of the selected one or more neural networks,
removing one or more neurons from one or more layers of the selected one or more neural networks,
adding one or more neurons to one or more neural network layers of the selected one or more neural networks,
adjusting an activation function associated with the selected one or more neural networks,
adjusting a loss function associated with the selected one or more neural networks, or
adding one or more neural network layers to the selected one or more neural networks.
3 . The method of claim 1 , wherein the selecting comprises adjusting a structure or one or more hyperparameters of the selected one or more neural networks.
4 . The method of claim 1 , wherein the one or more parameters comprise one or more parameters for encoding each segment of the representation of the content.
5 . The method of claim 1 , further comprising:
receiving data associated with the content, wherein the data comprises at least one of: a target bitrate, a content type, visual quality of the content, a resolution, or a frame rate.
6 . The method of claim 1 , further comprising:
extracting, from the content, one or more features associated with the content.
7 . The method of claim 6 , wherein the extracted one or more features comprise at least one of: temporal and spatial information, edges, corners, textures, pixel luma and chroma values, region of interests, motion or optical flow information, backgrounds, foregrounds, patterns, or spatial low or high frequencies.
8 . The method of claim 1 , wherein the one or more conditions comprise at least one of: available bandwidth, quality of service (QOS), latency, packet loss ratio, rebuffering time, or quality of experience (QoE).
9 . The method of claim 1 , wherein the one or more parameters achieve at least one of:
maximizing perceptual visual quality of the content,
maintaining a substantially constant perceptual visual quality of the content, or
minimizing latency associated with playback of the content.
10 . The method of claim 9 , wherein the substantially constant perceptual visual quality is indicated by at least one of: peak signal-to-noise ratio (PSNR), multiscale structural similarity index measure (MS-SSIM), or video multimethod assessment fusion (VMAF).
11 . The method of claim 9 , wherein the minimized latency associated with playback of the content causes minimizing rebuffering at the computing device.
12 . The method of claim 1 , wherein each neural network of the plurality of neural networks is trained, based on one or more of:
content type, compressed content, uncompressed content, content comprising artifacts, temporal and spatial information, edges, corners, textures, pixel luma and chroma values, region of interests, motion or optical flow information, backgrounds, foregrounds, patterns, or spatial low or high frequencies.
13 . The method of claim 1 , further comprising:
determining a change to the one or more conditions associated with the network; and
based on the determined change, repeating:
the selecting the one or more neural networks from the plurality of neural networks, and
the encoding the representation of the content.
14 . A method comprising:
receiving content to be displayed by a computing device;
selecting, based on one or more conditions associated with a network for delivering encoded content to the computing device, one or more neural networks from a plurality of neural networks pre-trained to output parameters used to optimize an encoding bitrate for a representation of the content; and
encoding, based on one or more parameters output by the selected one or more neural networks, the representation of the content.
15 . The method of claim 14 , wherein the selecting comprises adjusting the selected one or more neural networks, wherein the adjustment comprises one or more of:
removing one or more neural network layers of the selected one or more neural networks,
adjusting one or more weights associated with one or more neurons of the selected one or more neural networks,
removing one or more neurons from one or more layers of the selected one or more neural networks,
adding one or more neurons to one or more neural network layers of the selected one or more neural networks,
adjusting an activation function associated with the selected one or more neural networks,
adjusting a loss function associated with the selected one or more neural networks,
adding one or more neural network layers to the selected one or more neural networks, or
adjusting a structure or one or more hyperparameters of the selected one or more neural networks.
16 . The method of claim 14 , wherein the one or more conditions comprise at least one of:
available bandwidth, quality of service (QOS), latency, packet loss ratio, rebuffering time, or quality of experience (QoE).
17 . The method of claim 14 , further comprising:
receiving data associated with the content, wherein the data comprises at least one of: a target bitrate, a content type, visual quality of the content, a resolution, or a frame rate.
18 . The method of claim 14 , wherein the one or more parameters achieve at least one of:
maximizing perceptual visual quality of the content,
maintaining a substantially constant perceptual visual quality of the content, or
minimizing latency associated with playback of the content.
19 . A method comprising:
receiving data associated with content to be displayed by a computing device;
selecting, based on one or more conditions associated with a network for delivering encoded content to the computing device, one or more neural networks from a plurality of neural networks pre-trained to output parameters used to optimize an encoding bitrate for a representation of the content; and
encoding, based on the received data and one or more parameters output by the selected one or more neural networks, the representation of the content.
20 . The method of claim 19 , wherein the selecting comprises adjusting the selected one or more neural networks, wherein the adjustment comprises one or more of:
removing one or more neural network layers of the selected one or more neural networks,
adjusting one or more weights associated with one or more neurons of the selected one or more neural networks,
removing one or more neurons from one or more layers of the selected one or more neural networks,
adding one or more neurons to one or more neural network layers of the selected one or more neural networks,
adjusting an activation function associated with the selected one or more neural networks,
adjusting a loss function associated with the selected one or more neural networks,
adding one or more neural network layers to the selected one or more neural networks, or
adjusting a structure or one or more hyperparameters of the selected one or more neural networks.