Reinforcement learning based rate control
Implementations of the subject matter described herein provide a solution for rate control based on reinforcement learning. In this solution, an encoding state of a video encoder is determined, the encoding state being associated with encoding of a first video unit by the video encoder. An encoding parameter associated with rate control in the video encoder is determined by a reinforcement learning model and based on the encoding state of the video encoder. A second video unit different from the first video unit is encoded based on the encoding parameter. In this way, it is possible to achieve a better quality of experience (QOE) for real time communication with computation overhead being reduced.
1. A computer-implemented method, comprising:
determining an encoding state of a video encoder, the encoding state associated with encoding a first video unit by the video encoder;
determining, by a reinforcement learning model and based on the encoding state of the video encoder, an encoding parameter associated with rate control for the video encoder;
encoding a second video unit different from the first video unit based on the encoding parameter; and
training the reinforcement learning model according to a reward for the encoding parameter based on the encoding of the second video unit, the reward being configured to penalize buffer overshooting and to increase as the encoding parameter results in a higher visual quality, wherein the reward is based on a base reward that:
has a negative value if buffer overshooting occurs;
increases as the encoding parameter decreases if buffer overshooting does not occur; and
is scaled by a scaling factor to obtain the reward.
2. The method of claim 1 , wherein determining the encoding parameter comprises:
determining, by the reinforcement learning model, an action based on the encoding state of the video encoder; and
mapping the action to the encoding parameter.
3. The method of claim 1 , wherein the encoding state associated with the encoding the first video unit comprises:
a state representing an outcome for encoding at least the first video unit;
a state of a buffer configured to buffer video units encoded by the video encoder before transmission; and
a state associated with a status of a network for transmitting the encoded video units.
4. The method of claim 3 , wherein the outcome for encoding at least the first video unit comprises the encoding parameter from the encoding of the first video unit and a size of the encoded first video unit, the state of the buffer comprises the usage of the buffer, and the state associated with the status of the network comprises a target bits per pixel.
5. The method of claim 4 , wherein the usage of the buffer comprises at least one of:
a ratio of an occupied space to maximum space of the buffer; and
remaining space of the buffer measured in video units.
6. The method of claim 1 , wherein the scaling factor is based on a ratio of a bandwidth associated with the encoding the second video unit to maximum bandwidth of a transmission channel.
7. The method of claim 1 , wherein the reinforcement learning model is trained by:
determining an action associated with the encoding parameter based on the encoding state of the video encoder;
determining an evaluation value for encoding state for the encoding the second video unit;
determining a value loss based on the reward and the evaluation value;
determining a policy loss based on the action; and
updating the reinforcement learning model based on the value loss and the policy loss.
8. The method of claim 1 , wherein the reinforcement learning model comprises a neural network of an agent, and wherein the neural network comprises:
at least one input fully connected layer configured to extract features from the encoding state;
at least one recurrent neural network coupled to receive the extracted features; and
at least one output fully connected layer configured to decide an action for the agent.
9. The method of claim 8 , wherein the neural network is trained based on an actor-critic architecture, the actor being configured to generate the action based on the encoding state, and the critic being configured to generate an evaluation value for the encoding state; and
wherein the actor and the critic share a common portion of the neural network comprising the at least one input fully connected layer and the at least one recurrent neural network.
10. The method of claim 1 , wherein the encoding parameter comprises at least one of a quantization parameter and a lambda parameter.
11. The method of claim 1 , wherein the video encoder is configured to encode screen content for real-time communication.
12. A device comprising:
a processor; and
a memory having instructions stored thereon for execution by the processor, the instructions for causing, when executed by the processor, the device to perform acts including:
determining an encoding state of a video encoder, the encoding state associated with encoding a first video unit by the video encoder;
determining, by a reinforcement learning model and based on the encoding state of the video encoder, an encoding parameter associated with rate control in the video encoder, wherein the reinforcement learning model has been trained based on a reward for the encoding parameter, the reward being configured to penalize buffer overshooting and to increase as the encoding parameter results in a higher visual quality, and wherein the reward is based on a base reward that:
has a negative value if buffer overshooting occurs;
increases as the encoding parameter decreases if the buffer overshooting does not occur; and
is scaled by a scaling factor to obtain the reward; and
encoding a second video unit different from the first video unit based on the encoding parameter.
13. The device of claim 12 , wherein the encoding state associated with the encoding the first video unit comprises:
a state representing an outcome for encoding at least the first video unit;
a state of a buffer configured to buffer video units encoded by the video encoder before transmission; and
a state associated with a status of a network for transmitting the encoded video units.
14. The device of claim 13 , wherein the outcome for encoding at least the first video unit comprises the encoding parameter from the encoding of the first video unit and a size of the encoded first video unit, the state of the buffer comprises the usage of the buffer, and the state associated with the status of the network comprises a target bits per pixel.
15. The device of claim 12 , wherein the scaling factor is based on a ratio of a bandwidth associated with encoding one or more video units to maximum bandwidth of a transmission channel.
16. The device of claim 14 , wherein the reinforcement learning model comprises a neural network of an agent, and wherein the neural network comprises:
at least one input fully connected layer configured to extract features from the encoding state;
at least one recurrent neural network coupled to receive the extracted features; and
at least one output fully connected layer configure to decide an action for the agent.
17. The device of claim 12 , wherein the video encoder is configured to encode screen content for real-time communication.
18. A computer-readable storage medium having program instructions stored thereon, the program instructions being executable by a processor to cause the processor to perform acts comprising:
determining an encoding state of a video encoder, the encoding state associated with encoding a first video unit by the video encoder,
determining, by a reinforcement learning model and based on the encoding state of the video encoder, an encoding parameter associated with rate control in the video encoder, wherein the reinforcement learning model has been trained based on a reward for the encoding parameter, the reward being configured to penalize buffer overshooting and to increase as the encoding parameter results in a higher visual quality, and wherein the reward is based on a base reward that:
has a negative value if buffer overshooting occurs;
increases as the encoding parameter decreases if the buffer overshooting does not occur; and
is scaled by a scaling factor to obtain the reward; and
encoding a second video unit different from the first video unit based on the encoding parameter.
19. The computer-readable storage medium of claim 18 , wherein the encoding state associated with the encoding the first video unit comprises:
a state representing an outcome for encoding at least the first video unit;
a state of a buffer configured to buffer video units encoded by the video encoder before transmission; and
a state associated with a status of a network for transmitting the encoded video units.
20. The computer-readable storage medium of claim 19 , wherein the outcome for encoding at least the first video unit comprises the encoding parameter from the encoding of the first video unit and a size of the encoded first video unit, the state of the buffer comprises the usage of the buffer, and the state associated with the status of the network comprises a target bits per pixel.