Video compression at scene changes for low latency interactive experience
Methods and systems for video compression at scene changes provide an improved, low latency interactive experience in cloud computing environments. Exemplary use cases include all forms of cloud gaming including cloud-enabled interactive sporting events, e-sports, fantasy sports, gaming, and enhancements. Improvements in performance and experience are achieved with at least one of an extreme low latency rate controller, an extreme low latency rate controller method, frame partitioning at scene changes, preventive (relatively early) termination of encoding at scene changes, interactive signaling between a decoder and an encoder, or interactive signaling. Related apparatuses, devices, techniques, and articles are also described.
1 . A method for video compression, the method comprising:
estimating, with a rate controller operatively connected with an encoder, a quantization parameter value of at least a portion of a picture of a variable bitrate input;
determining an encoded picture size of the picture;
determining a required bitrate based on the encoded picture size and a frame rate of the encoder;
determining a capped variable bitrate of the variable bitrate input;
comparing the required bitrate with the capped variable bitrate;
based at least in part on determining that the required bitrate exceeds the capped variable bitrate, calculating an adjusted quantization parameter value equal to the quantization parameter value of the at least the portion of the picture plus an interval;
based at least on determining that the adjusted quantization parameter value is less than or equal to a maximum quantization parameter value supported by the encoder, calculating a new quantization parameter value equal to a ratio multiplied by a sum of the adjusted quantization parameter value and the maximum quantization parameter value supported by the encoder; and
encoding the picture using the new quantization parameter value.
2 . The method of claim 1 , wherein the interval is an integer, and the integer is 1.
3 . The method of claim 1 , further comprising comparing the adjusted quantization parameter value to the maximum quantization parameter value supported by the encode.
4 . The method of claim 1 , wherein the ratio is 1:2.
5 . The method of claim 1 , wherein encoding the picture using the new quantization parameter value comprises:
sending an intra picture generation request with the new quantization parameter value to the encoder; and
generating, with the encoder, an intra picture with the new quantization parameter value.
6 . The method of claim 1 , comprising
in response to determining that the adjusted quantization parameter value is not less than or equal to the maximum quantization parameter value supported by the encoder;
modifying the picture by at least one of changing a resolution of the picture, changing a framerate of the picture, or reducing the encoded picture size of the picture; and
transmitting the modified picture to the encoder.
7 . The method of claim 6 , wherein the modifying the picture by at least one of changing the resolution of the picture, changing the framerate of the picture, or reducing the encoded picture size of the picture comprises:
partitioning a frame at a scene change to re-distribute new picture data comprising intra coded macroblocks into a sequence of multiple frames in a row; and
encoding portions of the picture corresponding to repeat picture data from a previous scene using skipped macroblocks.
8 . The method of claim 6 , wherein the modifying the picture by at least one of changing the resolution of the picture, changing the framerate of the picture, or reducing the encoded picture size of the picture comprises:
terminating the encoding of the picture at a scene change based at least in part on determining that a number of accumulated bits for the picture greater than or equal to a calculated bit allowance; and
encoding a remaining portion of the picture after termination using skipped macroblocks or copied picture data.
9 . The method of claim 1 , further comprising:
determining a scene change will occur within a predetermined time period;
determining a probability of the picture being dropped at a client device due to the required bitrate exceeding the capped variable bitrate, wherein the determining is based on the picture being encoded at the scene change; and
based at least in part on determining that the probability of the picture being dropped exceeds a predetermined standard, generating an instantaneous decoder refresh (IDR) frame for a subsequent picture using the new quantization parameter value.
10 . A system for video compression, the system comprising:
a rate controller operatively connected with an encoder, the rate controller configured to:
estimate a quantization parameter value of at least a portion of a picture of a variable bitrate input;
determine an encoded picture size of the picture;
determine a required bitrate based on the encoded picture size and a frame rate of the encoder;
determine a capped variable bitrate of the variable bitrate input;
compare the required bitrate with the capped variable bitrate;
based at least in part on determining that the required bitrate exceeds the capped variable bitrate, calculate an adjusted quantization parameter value equal to the quantization parameter value of the at least the portion of the picture plus an interval;
based at least on determining that the adjusted quantization parameter value is less than or equal to a maximum quantization parameter value supported by the encoder, calculate a new quantization parameter value equal to a ratio multiplied by a sum of the adjusted quantization parameter value and the maximum quantization parameter value supported by the encoder; and
encode the picture using the new quantization parameter value.
11 . The system of claim 10 , wherein the interval is an integer, and the integer is 1.
12 . The system of claim 10 , wherein the rate controller is further configured to:
compare the adjusted quantization parameter value to the maximum quantization parameter value supported by the encode.
13 . The system of claim 10 , wherein the ratio is 1:2.
14 . The system of claim 10 , wherein the rate controller encodes the picture using the new quantization parameter value by:
sending an intra picture generation request with the new quantization parameter value to the encoder; and
generating, with the encoder, an intra picture with the new quantization parameter value.
15 . The system of claim 10 , wherein the rate controller configured to:
in response to determining that the adjusted quantization parameter value is not less than or equal to the maximum quantization parameter value supported by the encoder;
modify the picture by at least one of changing a resolution of the picture, changing a framerate of the picture, or reducing the encoded picture size of the picture; and
transmit the modified picture to the encoder.
16 . A non-transitory, computer-readable medium having non-transitory, computer-readable instructions encoded thereon for video compression with a rate controller operatively connected with an encoder, that, when executed:
estimate, with the rate controller operatively connected with the encoder, a quantization parameter value of at least a portion of a picture of a variable bitrate input;
determine an encoded picture size of the picture;
determine a required bitrate based on the encoded picture size and a frame rate of the encoder;
determine a capped variable bitrate of the variable bitrate input;
compare the required bitrate with the capped variable bitrate;
based at least in part on determining that the required bitrate exceeds the capped variable bitrate, calculate an adjusted quantization parameter value equal to the quantization parameter value of the at least the portion of the picture plus an interval;
based at least on determining that the adjusted quantization parameter value is less than or equal to a maximum quantization parameter value supported by the encoder, calculate a new quantization parameter value equal to a ratio multiplied by a sum of the adjusted quantization parameter value and the maximum quantization parameter value supported by the encoder; and
encode the picture using the new quantization parameter value.
17 . The non-transitory, computer-readable medium of claim 16 , wherein the interval is an integer, and the integer is 1.
18 . The non-transitory, computer-readable medium of claim 16 , wherein the ratio is 1:2.
19 . The non-transitory, computer-readable medium of claim 16 , wherein the instructions that cause encoding of the picture using the new quantization parameter value, further comprise instructions, that when executed:
send an intra picture generation request with the new quantization parameter value to the encoder; and
generate, with the encoder, an intra picture with the new quantization parameter value.
20 . The non-transitory, computer-readable medium of claim 16 , comprising instructions, that when executed:
in response to determining that the adjusted quantization parameter value is not less than or equal to the maximum quantization parameter value supported by the encoder:
modify the picture by at least one of changing a resolution of the picture, changing a framerate of the picture, or reducing the encoded picture size of the picture; and
transmitting the modified picture to the encoder.