Method and system for automatic user quality-of-experience measurement of streaming video
View Patent ↗Disclosed are system and method that automate measurement of end users' quality-of-experience (QoE) when perceiving the video being streamed to the users' viewing devices. The overall user QoE is measured and computed by combining the instantaneous presentation quality, the playback smoothness quality, and the interactions between them. Prediction accuracy is thus significantly improved. The instantaneous and end-of-process QoE measures created by the system and method described are suitable for the monitoring and optimization of media streaming systems and services.
1. A method of generating a quality-of-experience (QoE) measure of a streaming session of streaming video, the streaming video being transmitted from a video hosting server at server side to a user viewing device at receiver side over a network connection, the method comprising:
obtaining a presentation quality measure of the streaming video,
tracking occurrences of all stalling events during the streaming session,
obtaining a playback smoothness quality measure of the streaming video, the playback smoothness quality measure being assessed at the receiver side by combining contributions from all stalling events since the start of the streaming session, contribution from a stalling event being computed from the expression
S
k
(
t
)
=
{
Qd
(
t
)
,
during
stalling
event
Qd
(
i
k
+
l
k
f
)
m
(
t
)
,
after
stalling
event
0
,
prior
to
stalling
event
wherein the stalling event is the k-th stalling event that starts at i k and has a length l k , the stalling occurred during the period [i k ,i k +l k ], f is the frame rate of the streaming video in frames/second, d(t) is a quality decay function that monotonically decreases with time t, m(t) is a memory function that monotonically decreases with t and Q is a measure of presentation quality of the streaming video prior to the occurrence of the stalling event, and
generating an instantaneous QoE score by combining the presentation quality measure and the playback smoothness quality measure.
2. The method of claim 1 , further comprising:
cumulating instantaneous QoE scores generated at all time positions since the start of the streaming session to obtain an overall end-of-process QoE score of the streaming session.
3. The method of claim 1 , wherein the contribution of the stalling event is computed based on the presentation quality immediately prior to the occurrence of the stalling event.
4. The method of claim 1 , wherein a function monotonically decreasing with and saturating over time is used to represent the memory effect or the quality decay effect.
5. The method of claim 1 , wherein the playback smoothness quality measure is evaluated utilizing time positions and durations of initial buffering and playback stalling events.
6. The method of claim 5 , wherein the playback smoothness quality measure during a stalling event is evaluated utilizing the presentation quality measure of last rendered video frame before the start of the stalling event.
7. The method of claim 1 , wherein the quality decay function d(t) has the form
d
(
t
)
=
-
1
+
exp
{
-
(
tf
-
i
k
T
0
)
}
and the memory function m(t) has the form
m
(
t
)
=
exp
{
-
(
tf
-
i
k
-
l
k
T
1
)
}
for time t in the k-th stalling event, where T 0 and T 1 (T 1 >T 0 ) are two parameters selected to represent the rate of dissatisfaction and the relative strength of memory, respectively.
8. The method of claim 1 , wherein the playback smoothness quality measure is combined from the contributions due to all stalling events by adding the contributions from the individual stalling events.
9. The method of claim 8 , wherein the presentation quality measure and the playback smoothness quality measure are combined to generate the instantaneous QoE score by multiplying the two quality measures, by weighted summation of the two quality measures, or by taking the maximum or minimum of the two quality measures.
10. The method of claim 1 , wherein either the presentation quality measure is generated from a full-reference video quality assessment method, the full-reference video quality assessment method comparing quality of a test video generated from an original source of the streaming video with that of the original source video as a reference, or the presentation quality measure is generated from a no-reference video quality assessment method that requires no access to the original source video.
11. The method of claim 1 , wherein the presentation quality measure is either generated at the server side and transmitted to the receiver side over the network connection or generated at the receiver side, or is generated by a networked server.
12. The method of claim 11 , wherein the networked server receives parameters describing the user viewing device and viewing conditions of an end user and the presentation quality measure is generated using a video quality assessment method having viewing device and viewing condition adaptability.
13. The method of claim 1 , further comprising the step of measuring an end-of-process user QoE by cumulating the instantaneous QoE measures over the streaming session.
14. A system for measuring user quality-of-experience (QoE) of streaming video, the streaming video being transmitted from a video hosting server at server side to a user viewing device at receiver side over a network connection, the system comprising:
a presentation quality assessment unit, the presentation quality assessment unit generating or obtaining a presentation quality measure of the streaming video;
a playback smoothness quality assessment unit, the playback smoothness quality assessment unit tracking occurrences of all stalling events during a streaming session and generating a playback smoothness quality measure of playback smoothness quality perceived at the user viewing device playing back the streaming video, wherein the generation of the smoothness quality measure combines contributions from all stalling events since the start of the streaming session, contribution from a stalling event being computed from the expression
S
k
(
t
)
=
{
Qd
(
t
)
,
during
stalling
event
Qd
(
i
k
+
l
k
f
)
m
(
t
)
,
after
stalling
event
0
,
prior
to
stalling
event
wherein the stalling event is the k-th stalling event that starts at i k and has a length l k , the stalling occurred during the period [i k ,i k +l k ], f is the frame rate of the streaming video in frames/second, d(t) is a quality decay function that monotonically decreases with time t, m(t) is a memory function that monotonically decreases with t and Q is a measure of presentation quality of the streaming video prior to the occurrence of the stalling event, and
a QoE assessment unit, the QoE assessment unit combining the presentation quality measure and the playback smoothness quality measure into an instantaneous QoE score.
15. The system of claim 14 , further comprising an end-of-process QoE unit, the end-of-process QoE unit cumulating all instantaneous QoE scores and combining them into a single overall end-of-process QoE score of the streaming session.
16. The system of claim 14 , wherein the presentation quality assessment unit is integrated into the video hosting server and the presentation quality assessment unit computes contribution of the stalling event based on the presentation quality measure of a video frame prior to the occurrence of the stalling event.
17. The system of claim 14 , further comprising a networked server and wherein the presentation quality assessment unit or the playback smoothness quality assessment unit is integrated into the networked server.
18. The system of claim 17 , wherein the networked server is an edge sever at the receiver side to obtain the presentation quality measure transmitted from the server side or to generate the presentation quality measure at the receiver side.
19. The system of claim 17 , wherein the networked server receives parameters describing the user viewing device and viewing conditions of an end user and the presentation quality measure is generated using a video quality assessment method having viewing device and viewing condition adaptability.
20. The system of claim 14 , wherein the playback smoothness quality assessment unit uses a function monotonically decreasing with and saturating over time to represent the memory effect or to represent the quality decay effect.
21. The system of claim 14 , wherein the playback smoothness quality assessment unit evaluates the playback smoothness quality measure utilizing time positions and durations of initial buffering and playback stalling events.
22. The system of claim 14 , wherein Q is the presentation quality measure of the last fully rendered frame prior to the occurrence of the stalling event, P i k -1 .
23. The system of claim 14 , wherein the quality decay function d(t) has the form
d
(
t
)
=
-
1
+
exp
{
-
(
tf
-
i
k
T
0
)
}
and the memory function m(t) has the form
m
(
t
)
=
exp
{
-
(
tf
-
i
k
-
l
k
T
1
)
}
for time t in the k-th stalling event, where T 0 and T 1 (T 1 >T 0 ) are two parameters selected to represent the rate of dissatisfaction and the relative strength of memory, respectively.
24. The system of claim 14 , wherein the QoE assessment unit is configured to add the contributions from the individual stalling events to obtain the playback smoothness quality measure.
25. The system of claim 24 , wherein the QoE assessment unit is configured to add the presentation quality measure and the playback smoothness quality measure to generate the instantaneous QoE score.
26. The system of claim 24 , wherein the QoE assessment unit is configured to generate the instantaneous QoE score by multiplying the presentation quality measure and the playback smoothness quality measure, by weighted summation of the two quality measures, or by taking the maximum or minimum of the two quality measures.
27. The system of claim 14 , wherein the presentation quality assessment unit either is configured to generate the presentation quality measure from a full-reference video quality assessment method, the full-reference video quality assessment method comparing quality of a test video generated from an original source of the streaming video with that of the original source video as a reference, or is configured to generate the presentation quality measure from a no-reference video quality assessment method that requires no access to the original source video.
28. The system of claim 27 , wherein, where the full-reference video quality assessment method is used, the test video has spatial and/or temporal resolutions different from those of the reference video.
29. A non-transitory computer-readable medium having stored thereon computer readable code that when executed by a processor of a computing device, causes the computing device to perform a method of measuring user quality-of-experience of streaming video, according to claim 1 .
30. The non-transitory computer-readable medium of claim 29 , wherein the method of measuring user quality-of-experience of streaming video further comprises cumulating instantaneous QoE scores generated at all time positions since the start of the streaming session to obtain an overall end-of-process QoE score of the streaming session.
31. The non-transitory computer-readable medium of claim 29 , wherein the contribution of the stalling event is computed based on the presentation quality immediately prior to the occurrence of the stalling event.
32. The system of claim 14 , wherein the playback smoothness quality assessment unit is configured to compute the contribution of the stalling event based on the presentation quality immediately prior to the occurrence of the stalling event.
33. The system of claim 17 , wherein the networked server is a cloud server that obtains the presentation quality measure transmitted from the server side.