Systems and methods for rendering and pre-encoded load estimation based encoder hinting
Systems and methods for hinting an encoder are disclosed in which a server monitors for information related to changes in frame rendering, calculates tolerance boundaries, rolling average frame time and/or short-term trends in frame time, and uses those calculations to identify a frame time peak. The server then hints a codec (encoder) to modulate the quality settings of frame output in proportion to the size of the frame time peak. In certain embodiments, a renderer records one or more playthroughs in a game environment, sorts a plurality of frames from one or more playthroughs into a plurality of cells on a heatmap, and collects the list of sorted frames. A codec may then encode one or more frames from the list of sorted frames to calculate an average encoded frame size for each cell in the heatmap, and associate each average encoded frame size with a per-cell normalized encoder quality setting.
1. A computer-implemented method for encoding data, comprising the steps of:
recording one or more playthroughs in a game environment;
sorting a plurality of frames from the one or more playthroughs into a plurality of cells on a heatmap, wherein said sorting results in a list of sorted frames associated with the heatmap;
collecting the list of sorted frames at a renderer;
encoding one or more frames from the list of sorted frames to calculate an average encoded frame size for each cell in the heatmap, wherein each average encoded frame size is associated with a per-cell normalized encoder quality setting; and
calculating an average frame size for the heatmap from the average encoded frame size of each cell,
wherein, during gameplay, the per-cell normalized encoder quality setting corresponding to the cell in the heatmap is used to hint an encoder to code a video sequence.
2. The method of claim 1 , wherein the one or more frames are coded into the video sequence in a single pass.
3. The method of claim 1 , further comprising the step of storing the per-cell normalized encoder quality settings at the renderer.
4. The method of claim 1 , wherein the one or more playthroughs are stored at a telemetry server.
5. The method of claim 1 , wherein the one or more playthroughs are comprised of a plurality of frames and a player location associated with each of the plurality of frames.
6. The method of claim 5 , wherein the player location is used to select the per-cell normalized encoder quality setting that hints the encoder.
7. The method of claim 1 , wherein the per-cell normalized encoder quality settings are calculated by the equation:
normalized
encoder
quality
setting
=
encoder
quality
setting
Σ
cells
encoder
quality
setting
number
of
cells
.
8. The method of claim 1 , further comprising the step of combining the per-cell normalized encoder quality settings from a spatially-related sequence and a temporally-related sequence.
9. A system for encoding data, comprising:
a renderer, wherein the renderer:
records one or more playthroughs in a game environment;
sorts a plurality of frames from the one or more playthroughs into a plurality of cells on a heatmap, wherein said sorting results in a list of sorted frames associated with the heatmap;
collects the list of sorted frames; and
calculates an average frame size for the heatmap from the average encoded frame size of each cell; and
an encoder, wherein the encoder:
encodes one or more frames from the list of sorted frames to calculate an average encoded frame size for each cell in the heatmap, wherein each average encoded frame size is associated with a per-cell normalized encoder quality setting, and
wherein, during gameplay, the per-cell normalized encoder quality setting corresponding to the cell in the heatmap is used to hint the encoder to code a video sequence.
10. The system of claim 9 , wherein the one or more frames are coded into the video sequence in a single pass.
11. The system of claim 9 , wherein the renderer stores the per-cell normalized encoder quality settings.
12. The system of claim 9 , wherein the one or more playthroughs are stored at a telemetry server.
13. The method of claim 9 , wherein the one or more playthroughs are comprised of a plurality of frames and a player location associated with each of the plurality of frames.
14. The method of claim 13 , wherein the player location is used to select the per-cell normalized encoder quality setting that hints the encoder.
15. The system of claim 9 , wherein the per-cell normalized encoder quality settings are calculated by the equation:
normalized
encoder
quality
setting
=
encoder
quality
setting
Σ
cells
encoder
quality
setting
number
of
cells
.
16. The system of claim 9 , wherein the per-cell normalized encoder quality settings are combined from a spatially-related sequence and a temporally-related sequence.
17. A system for encoding data, comprising:
a renderer, wherein the renderer records a video sequence comprised of a plurality of frames; and
an encoder, wherein the encoder codes the video sequence in a multi-pass mode that optimizes encoder quality settings against the first frame of the video sequence, and wherein the encoder records the encoder quality settings,
wherein the renderer normalizes the encoder quality settings to the first frame of the video sequence,
wherein the normalized encoder quality settings are used to hint the encoder to code the video sequence, and
wherein normalized encoder quality settings are calculated by the equation:
normalized
QP
=
frame
QP
first
frame
QP
.
18. The system of claim 17 , wherein the video sequence is temporally related to one or more other video sequences.
19. A system for encoding data, comprising:
a renderer, wherein the renderer records a video sequence comprised of a plurality of frames; and
an encoder, wherein the encoder codes the video sequence in a multi-pass mode that optimizes encoder quality settings against the first frame of the video sequence, and wherein the encoder records the encoder quality settings,
wherein the renderer normalizes the encoder quality settings to the first frame of the video sequence,
wherein the normalized encoder quality settings are used to hint the encoder to code the video sequence, and
wherein the normalized encoder quality settings are stored as an ordered list of floats.
20. A system for encoding data, comprising:
a renderer, wherein the renderer records a video sequence comprised of a plurality of frames; and
an encoder, wherein the encoder codes the video sequence in a multi-pass mode that optimizes encoder quality settings against the first frame of the video sequence, and wherein the encoder records the encoder quality settings,
wherein the renderer normalizes the encoder quality settings to the first frame of the video sequence,
wherein the normalized encoder quality settings are used to hint the encoder to code the video sequence, and
wherein the normalized encoder quality settings are multiplied by a runtime encoder quality setting, wherein the multiplied normalized encoder quality settings are used to hint the encoder to code the video sequence.
21. A computer-implemented method for encoding, comprising the steps of:
recording a video sequence comprised of a plurality of frames;
encoding the video sequence in a multi-pass mode that optimizes encoder quality settings against the first frame of the video sequence;
recording the encoder quality settings; and
normalizing the encoder quality settings to the first frame of the video sequence,
wherein the normalized encoder quality settings are used to hint an encoder to code the video sequence, and
wherein normalized encoder quality settings are calculated by the equation:
normalized
QP
=
frame
QP
first
frame
QP
.
22. The method of claim 21 , wherein the video sequence is temporally related to one or more other video sequences.
23. A computer-implemented method for encoding, comprising the steps of:
recording a video sequence comprised of a plurality of frames;
encoding the video sequence in a multi-pass mode that optimizes encoder quality settings against the first frame of the video sequence;
recording the encoder quality settings; and
normalizing the encoder quality settings to the first frame of the video sequence,
wherein the normalized encoder quality settings are used to hint an encoder to code the video sequence, and
wherein the normalized encoder quality settings are stored as an ordered list of floats.
24. A computer-implemented method for encoding, comprising the steps of:
recording a video sequence comprised of a plurality of frames;
encoding the video sequence in a multi-pass mode that optimizes encoder quality settings against the first frame of the video sequence;
recording the encoder quality settings; and
normalizing the encoder quality settings to the first frame of the video sequence,
wherein the normalized encoder quality settings are used to hint an encoder to code the video sequence, and
further comprising the step of multiplying the normalized encoder quality settings by a runtime encoder quality setting, wherein the multiplied normalized encoder quality settings are used to hint the encoder to code the video sequence.