Low-delay multi-pass frame-level rate control using a shared reference frame
A multi-pass encoding operation using a shared reference frame is implemented to encode one or more gaming frames into a game stream. The multi-pass encoding operation includes producing a shared reference frame based upon a second-pass reference frame used by a second pass encoding of the multi-pass encoding operation. The encoding operation also includes performing a first encoding pass on a current frame using the shared reference frame. As a result of the first encoding pass, an estimated complexity for the current frame is determined. A second pass encoding is then performed on the current frame according to the second-pass reference frame and the estimated complexity, resulting in an encoded frame. This encoded frame is then transmitted as part of a stream to a client system.
1 . A method, comprising:
comparing, by an encoder of a server, a first-pass resolution of a first pass encoding of a multi-pass encoding process to be performed by the encoder to a second-pass resolution of a second pass encoding of the multi-pass encoding process to be performed by the encoder;
producing a shared reference frame from a second-pass reference frame to be used in the second pass encoding based on the comparison;
performing, by the encoder, the first pass encoding on a first video frame of a set of video frames at the first-pass resolution using the shared reference frame to determine an estimated complexity of the first video frame; and
performing, by the encoder, the second pass encoding on the first video frame at the second-pass resolution using the second-pass reference frame and according to the estimated complexity to generate an encoded frame to be included in a video stream transmitted by the server.
2 . The method of claim 1 , further comprising:
retrieving the second-pass reference frame from a memory buffer associated with the second pass encoding.
3 . The method of claim 2 , wherein producing the shared reference frame comprises:
in response to the first-pass resolution being less than the second-pass resolution, downsampling the second-pass reference frame to the first-pass resolution and storing the downsampled reference frame in a memory buffer associated with the first pass encoding to produce the shared reference frame.
4 . The method of claim 1 , further comprising:
in response to the first-pass resolution being equal to the second-pass resolution, storing the second-pass reference frame in a memory buffer associated with the first pass encoding to produce the shared reference frame.
5 . The method of claim 1 , further comprising:
retrieving the second-pass reference frame according to a pointer stored in a memory buffer associated with the first pass encoding.
6 . The method of claim 5 , wherein the memory buffer associated with the first pass encoding includes a plurality of pointers each associated with a respective second-pass reference frame stored in a second memory buffer associated with the second pass encoding.
7 . A computer-implemented method, comprising:
producing, by an encoder of a server, a shared reference frame for use in a first pass encoding of a multi-pass encoding process to be performed by the encoder based on a second-pass reference frame for use in a second pass encoding of the multi-pass encoding process, wherein producing the shared reference frame comprises:
comparing a first-pass resolution of the first pass encoding to a second-pass resolution of the second pass encoding; and
downsampling, by the encoder, the second-pass reference frame based upon the comparison of the first-pass resolution to the second-pass resolution;
performing, by the encoder, the first pass encoding on a first video frame of a set of video frames at the first-pass resolution using the shared reference frame to determine an estimated complexity of the first video frame;
determining an estimated complexity of the first video frame based on the first pass encoding of the first video frame; and
performing, by the encoder, the second pass encoding on the first video frame at the second-pass resolution using the second-pass reference frame and the estimated complexity to generate an encoded frame to be included in a video stream transmitted by the server.
8 . The method of claim 7 , wherein producing the shared reference frame further comprises:
accessing a memory buffer associated with the second pass encoding according to a pointer in a second memory buffer associated with the first pass encoding.
9 . The method of claim 8 , wherein the second memory buffer includes a plurality of pointers each associated with a respective second-pass reference frame stored in the memory buffer associated with the second pass encoding.
10 . The method of claim 8 , wherein producing the shared reference frame further comprises:
in response to the first-pass resolution being less than the second-pass resolution, downsampling the second-pass reference frame to the first-pass resolution and storing the downsampled reference frame in the second memory buffer to produce the shared reference frame.
11 . The method of claim 8 , wherein producing the shared reference frame further includes:
in response to the first-pass resolution being equal to the second-pass resolution, storing the second-pass reference frame in the second memory buffer associated with the first pass encoding to produce the shared reference frame.
12 . The method of claim 1 , wherein the first pass encoding further comprises:
downsampling the first video frame to the first-pass resolution.
13 . The method of claim 1 , further comprising:
determining a quantization parameter for the first video frame.
14 . The method of claim 13 , wherein the second pass encoding is further performed based on the quantization parameter.
15 . The method of claim 1 , wherein performing the first pass encoding produces a first pass encoded frame having a bit size and wherein the estimated complexity is based on the bit size.
16 . The method of claim 1 , wherein the first video frame represents at least a portion of a virtual environment of a gaming session associated with a client system.
17 . The method of claim 16 , further comprising:
encoding the set of video frames according to the multi-pass encoding process so as to produce the video stream; and
transmitting the video stream to the client system.
18 . The method of claim 17 , further comprising:
decoding the video stream so as to produce a decoded set of video frames; and
displaying the decoded set of video frames.
19 . The method of claim 18 , further comprising:
rendering the set of video frames according to a client session associated with the client system.
20 . The method of claim 1 , wherein the first video frame on which the first pass and second pass encodings are performed is rendered.
21 . The method of claim 1 , further comprising encoding the first video frame according to a rate control scheme so that the encoded frame of the second pass encoding is compressed so as to comply with a predetermined target bit size.
22 . A cloud-based video system comprising:
one or more cloud-based servers communicatively coupled to one or more client systems and each including:
one or more processors; and
a memory coupled to the one or more processors and storing executable instructions configured to manipulate the one or more processors to:
compare a first-pass resolution of a first pass encoding of a multi-pass encoding process to a second-pass resolution of a second pass encoding of the multi-pass encoding process;
produce a shared reference frame from a second-pass reference frame used in the second pass encoding based on the comparison;
perform the first pass encoding on a first video frame of a set of video frames at the first-pass resolution using the shared reference frame to determine an estimated complexity of the first video frame; and
perform the second pass encoding on the first video frame at the second-pass resolution using the second-pass reference frame and according to the estimated complexity to generate an encoded frame.