Encoding output for streaming applications based on client upscaling capabilities
In various examples, the decoding and upscaling capabilities of a client device are analyzed to determine encoding parameters and operations used by a content streaming server to generate encoded video streams. The quality of the upscaled content of the client device may be monitored by the streaming servers such that the encoding parameters may be updated based on the monitored quality. In this way, the encoding operations of one or more streaming servers may be more effectively matched to the decoding and upscaling abilities of one or more client devise such that an increased number of client devices may be served by the streaming servers.
1. A method comprising:
receiving, from a client device, feedback data corresponding to one or more upscaled versions of one or more first encoded frames transmitted using one or more streams of video data to the client device, the feedback data being generated using the client device, wherein the one or more first encoded frames are generated by applying one or more encoding parameters to one or more video frames;
determining, using the feedback data, one or more indicators of visual quality of the one or more upscaled versions of the one or more first encoded frames relative to the one or more video frames, the one or more indicators of visual quality corresponding to a level of visual similarity between the one or more upscaled versions of the one or more first encoded frames and the one or more video frames;
determining, based at least on the one or more indicators of visual quality, one or more updated encoding parameters comprising at least one updated parameter value corresponding to at least one encoding parameter of the one or more encoding parameters; and
transmitting one or more second encoded frames in the one or more streams of video data to the client device to cause the client device to generate one or more upscaled versions of the one or more second encoded frames, the one or more second encoded frames being encoded using the one or more updated encoding parameters.
2. The method of claim 1 , wherein the feedback data includes one or more portions of the one or more upscaled versions of the one or more first encoded frames, and
the determining the one or more indicators of visual quality includes comparing the one or more portions of the one or more upscaled versions of the one or more first encoded frames to one or more corresponding portions of the one or more video frames.
3. The method of claim 1 , wherein the applying the one or more encoding parameters includes downscaling the one or more video frames to a first resolution indicated by the one or more encoding parameters to generate the one or more first encoded frames having the first resolution, and
wherein the one or more updated encoding parameters indicate a second resolution for the one or more second encoded frames based at least on the one or more indicators of visual quality.
4. The method of claim 1 , wherein the one or more streams of video data include one or more streams of output frames from an executing application instance, wherein the one or more video frames are included in the output frames.
5. The method of claim 1 , further comprising, based at least on the determining the one or more updated encoding parameters, applying the one or more updated encoding parameters to one or more second video frames to generate the one or more second encoded frames.
6. The method of claim 1 , wherein the one or more updated encoding parameters control a frame resolution for the one or more streams of video data.
7. A system comprising:
one or more processing units to execute operations including:
receiving, from a client device, feedback data corresponding to one or more upscaled versions of one or more first frames, the one or more first frames being decoded and upscaled using the client device and based on one or more streams of video data;
determining, using the feedback data, one or more indicators of visual quality of the one or more upscaled versions of the one or more first frames relative to one or more video frames used to generate the one or more first frames, the one or more indicators of visual quality corresponding to a level of visual similarity between the one or more upscaled versions of the one or more first frames and the one or more video frames;
selecting one or more encoding parameters based at least on the one or more indicators of visual quality; and
generating one or more second frames encoded using the one or more encoding parameters.
8. The system of claim 7 , wherein the feedback data includes one or more portions of the one or more upscaled versions of the one or more first frames, and
the determining the one or more indicators of visual quality includes comparing the one or more portions of the one or more upscaled versions of the one or more first frames to one or more corresponding portions of the one or more video frames.
9. The system of claim 7 , wherein the one or more first frames have a first resolution prior to being upscaled using the client device, and
the one or more encoding parameters define a second resolution for the one or more second frames based at least on the one or more indicators of visual quality.
10. The system of claim 7 , wherein the one or more streams of video data include one or more streams of output frames from an executing application instance, the one or more video frames included in the output frames.
11. The system of claim 7 , wherein the operations further include, based at least on the selecting the one or more encoding parameters, applying the one or more encoding parameters to one or more second video frames to generate the one or more second frames.
12. The system of claim 7 , wherein the one or more encoding parameters control a frame resolution for the one or more streams of video data.
13. The system of claim 7 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
14. A processor comprising:
one or more circuits to generate one or more encoded frames in one or more streams of video data using one or more encoding parameters, the one or more encoding parameters based at least on one or more indicators of visual quality, the one or more indicators of visual quality determined based at least on:
receiving, from a client device, feedback data corresponding to one or more upscaled versions of one or more frames decoded and upscaled from the one or more streams of video data using the client device; and
comparing, using the feedback data, the one or more upscaled versions of the one or more frames to one or more video frames used to generate the one or more frames, wherein the one or more indicators of visual quality correspond to a level of visual similarity between the one or more upscaled versions of the one or more frames and the one or more video frames.
15. The processor of claim 14 , wherein the feedback data includes one or more portions of the one or more upscaled versions of the one or more frames.
16. The processor of claim 14 , wherein the one or more frames have a first resolution prior to being upscaled using the client device, and
the one or more encoding parameters define a second resolution for the one or more encoded frames based at least on the one or more indicators of visual quality.
17. The processor of claim 14 , wherein the one or more streams of video data include one or more streams of output frames from an executing application instance, the one or more video frames included in the output frames.
18. The processor of claim 14 , wherein the one or more circuits are further to, based at least on the one or more encoding parameters, apply the one or more encoding parameters to one or more second video frames to generate the one or more encoded frames.
19. The processor of claim 14 , wherein the one or more encoding parameters control a frame resolution for the one or more streams of video data.
20. The processor of claim 14 , wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.