Evaluating qualitative streaming experience using session performance metadata
A technique for evaluating qualitative streaming experience using session performance metadata is disclosed herein. A pipeline of a streaming service can be adapted to collect metadata, such as timestamps, from various components of the pipeline. The metadata can then be analyzed to calculate an objective quality metric for each streaming session using weighted scores derived from the metadata for a plurality of different components including, but not limited to, stutter, latency, and/or picture quality. The quality metric is designed to have high correlation with subjective measures of quality by users of the streaming service, but provides dense data samples compared to typical sparse responses collected from user feedback (e.g., user surveys). The objective quality metric can be utilized to quickly adjust, either manually or automatically, the streaming service parameters to improve the quality of the streaming service due to changes in, e.g., streaming content.
1. A system for estimating a quality of a streaming session provided by a streaming service, the system comprising:
one or more processors configured to:
receive metadata for a plurality of frames included in the streaming session;
calculate, for each frame of the plurality of frames, a score and a weight for each component of a plurality of components derived from the metadata;
calculate a quality metric for the streaming session based on the scores and weights for the plurality of components; and
adjust at least one component of a pipeline implemented by the streaming service based on the quality metric.
2. The system of claim 1 , wherein the quality metric is calculated as a sum of a picture quality score and a combined stutter-latency score.
3. The system of claim 2 , wherein the combined stutter-latency score is calculated by:
calculating, for each frame of the plurality of frames, a rolling weighted score for each component of at least one latency component and at least one stutter component based on a first sliding window; and
calculating a rolling mean of a combined weighted sum of the rolling weighted scores for each of the at least one latency component and at least one stutter component, wherein the rolling mean of the combined weighted sum is based on a second sliding window.
4. The system of claim 3 , wherein the combined stutter-latency score is calculated as a difference between an aggregate weighted mean calculated based on a first set of parameters and a standard deviation based on a second set of parameters.
5. The system of claim 2 , wherein the picture quality score is calculated as a difference between a mean of a plurality of picture quality scores for the plurality of frames and a product of a coefficient and a standard deviation of the plurality of picture quality scores for the plurality of frames.
6. The system of claim 1 , wherein adjusting the at least one component of the pipeline implemented by the streaming service based on the quality metric comprises adjusting at least one of a quantization parameter, a target bit rate, and encoding parameter, or a target resolution for the frames generated by the pipeline.
7. The system of claim 1 , wherein the one or more processors are further configured to generate a visual representation of the quality metric for the streaming session.
8. The system of claim 1 , wherein the plurality of components include at least one of the following components:
a first stutter component based on a delta between a first timestamp corresponding to rendering a frame by an application and a second timestamp corresponding to capturing the frame by a first stage of the pipeline;
a second stutter component based on a delta between a first timestamp corresponding to presenting the frame on a client device and a second timestamp corresponding to presenting a previous frame on the client device;
a first latency component based on a round trip delay for the frame; and
a second latency component based on a stack latency for the frame.
9. A method for estimating a quality metric of a streaming session, the method comprising:
receiving metadata for a plurality of frames included in the streaming session, wherein the metadata includes information related to a number of stages in a pipeline implemented by a streaming service, and wherein the pipeline includes at least one stage implemented by a server device and at least one additional stage implemented by a client device connected to the server device via a network;
calculating, for each frame of the plurality of frames, a score and a weight for each component of a plurality of components derived from the metadata; and
calculating a quality metric for the streaming session based on the scores and weights for the plurality of components.
10. The method of claim 9 , further comprising:
generating a visual representation of the quality metric for one or more streaming sessions; or
adjusting at least one component of the pipeline based on the quality metric.
11. The method of claim 9 , wherein the quality metric is calculated as a sum of a picture quality score and a combined stutter-latency score.
12. The method of claim 11 , wherein the combined stutter-latency score is calculated by:
calculating, for each frame of the plurality of frames, a rolling weighted score for each component of at least one latency component and at least one stutter component based on a first sliding window; and
calculating a rolling mean of a combined weighted sum of the rolling weighted scores for each of the at least one latency component and at least one stutter component, wherein the rolling mean of the combined weighted sum is based on a second sliding window.
13. The method of claim 12 , wherein the combined stutter-latency score is calculated as a difference between an aggregate weighted mean calculated based on a first set of parameters and a standard deviation based on a second set of parameters.
14. The method of claim 11 , wherein the picture quality score is calculated as a difference between a mean of a plurality of picture quality scores for the plurality of frames and a product of a coefficient and a standard deviation of the plurality of picture quality scores for the plurality of frames.
15. The method of claim 9 , further comprising adjusting, in response to the quality metric, at least one of a quantization parameter, a target bit rate, and encoding parameter, or a target resolution for the frames generated by the pipeline.
16. The method of claim 9 , wherein the plurality of components include at least one of the following components:
a first stutter component based on a delta between a first timestamp corresponding to rendering a frame by an application and a second timestamp corresponding to capturing the frame by a first stage of the pipeline;
a second stutter component based on a delta between a first timestamp corresponding to presenting the frame on a client device and a second timestamp corresponding to presenting a previous frame on the client device;
a first latency component based on a round trip delay for the frame; and
a second latency component based on a stack latency for the frame.
17. A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising:
receiving metadata for a plurality of frames included in the streaming session, wherein the metadata includes information related to a number of stages in a pipeline implemented by a streaming service, and wherein the pipeline includes at least one stage implemented by a server device and at least one additional stage implemented by a client device connected to the server device via a network;
calculating, for each frame of the plurality of frames, a score and a weight for each component of a plurality of components derived from the metadata; and
calculating a quality metric for the streaming session based on the scores and weights for the plurality of components.
18. The computer-readable medium of claim 17 , the steps further comprising:
generating a visual representation of the quality metric for one or more streaming sessions; or
adjusting at least one component of the pipeline based on the quality metric.
19. The computer-readable medium of claim 17 , wherein:
the quality metric is calculated as a sum of a picture quality score and a combined stutter-latency score;
the combined stutter-latency score is calculated by:
calculating, for each frame of the plurality of frames, a rolling weighted score for each component of at least one latency component and at least one stutter component based on a first sliding window; and
calculating a rolling mean of a combined weighted sum of the rolling weighted scores for each of the at least one latency component and at least one stutter component, wherein the rolling mean of the combined weighted sum is based on a second sliding window; and
the picture quality score is calculated as a difference between a mean of a plurality of picture quality scores for the plurality of frames and a product of a coefficient and a standard deviation of the plurality of picture quality scores for the plurality of frames.
20. The computer-readable medium of claim 19 , wherein the combined stutter-latency score is calculated as a difference between an aggregate weighted mean calculated based on a first set of parameters and a standard deviation based on a second set of parameters.