Quality of user experience testing for video transmissions
View Patent ↗Techniques for calculating quality of user experience (QoE) scores for video streams as played on receiving devices are described herein. Prior to calculating a QoE score, a device receives a video of a display screen of a receiving device captured while the receiving device plays a video stream. The device also receives transmission metrics from at least one device engaged in the transmission of the video stream to the receiving device. The device then calculates the QoE score for that received video based at least in part on a reference video, the transmission metrics, and one or more QoE models. Additionally, prior to receiving the video or the transmission metrics, the device may automate the transmission of the reference video as the video stream from a sending device to the receiving device over at least one network emulator, including providing network impairment settings to the network emulator.
1. A system comprising:
at least one network emulator component communicatively coupled to a sending device and a receiving device and configured to transmit a reference video as a video stream from the sending device to the receiving device;
a camera configured to capture a video of a display screen of the receiving device while the receiving device plays the video stream; and
a test controller device configured to:
automate the transmission of the video stream from the sending device to the receiving device, including providing network impairment settings to the at least one network emulator,
receive transmission metrics from at least one of the sending device, the receiving device, and the at least one network emulator,
receive the video from the camera, and
calculate a quality of user experience (QoE) score for the video based at least in part on the reference video, the transmission metrics, and one or more QoE models.
2. The system of claim 1 , wherein the at least one network emulator is one of a telecommunication network emulator or an Internet Protocol (IP) network emulator.
3. The system of claim 1 , wherein the sending device, the receiving device, the at least one network emulator, and the camera are each physically coupled to the test controller device.
4. The system of claim 1 , further comprising a video relay server that is connected to the at least one network emulator via an IP network and configured to receive the video stream from the at least one network emulator and to transmit the video stream back to the at least one network emulator.
5. The system of claim 4 , wherein the video relay server is configured to perform the transmission or receiving based on impairment settings.
6. A method comprising:
receiving from a camera of a testing system, by a test controller device of the testing system, a video of a display screen of a receiving device captured by the camera while the receiving device plays a video stream;
receiving, by the test controller device, transmission metrics from at least one device engaged in the transmission of a reference video as the video stream to the receiving device; and
calculating, by the test controller device, a quality of user experience (QoE) score for the video based at least in part on the reference video, the transmission metrics, and one or more QoE models.
7. The method of claim 6 , further comprising automating a transmission of the reference video as the video stream from a sending device to the receiving device over at least one network emulator, including providing network impairment settings to the at least one network emulator.
8. The method of claim 6 , wherein the receiving device receives the video stream from a remote video server.
9. The method of claim 6 , wherein the transmission metrics include at least one of a frame rate, a bit rate, a round trip time, a call establishment time, or an encoding rate.
10. The method of claim 6 , wherein the one or more QoE models include at least one of a QoE model for calculating a first QoE score for the video based on the reference video or QoE models for calculating second QoE scores for the video based on the transmission metrics.
11. The method of claim 10 , wherein the one or more QoE models include a QoE model for synthesizing the first and second QoE scores.
12. The method of claim 10 , wherein the one or more QoE models include the QoE model for calculating the first QoE score for the video based on the reference video and a QoE model for calculating QoE score adjustments based on the transmission metrics.
13. One or more non-transitory computer-readable media having stored thereon a plurality of executable instructions, the executable instructions configured to program a test controller device of a testing system to perform operations comprising:
automating a transmission of a reference video as a video stream from a sending device to a receiving device over at least one network emulator, including providing network impairment settings to the at least one network emulator;
receiving from a camera of the testing system a video of a display screen of the receiving device captured by the camera while the receiving device plays the video stream;
receiving transmission metrics from at least one device of the sending device, the receiving device, or the at least one network emulator; and
calculating a quality of user experience (QoE) score for the video based at least in part on the reference video, the transmission metrics, and one or more QoE models.
14. The one or more non-transitory computer-readable media of claim 13 , wherein the network impairment settings include at least one of radio frequency (RF) impairment settings, packet loss settings, transmission delay settings, data corruption settings, or throughput settings.
15. The one or more non-transitory computer-readable media of claim 13 , wherein the automating includes instructing the sending device to transmit the reference video as the video stream, instructing the at least one network emulator to transmit the video stream to the receiving device, and instructing the receiving device to accept and play the video stream.
16. The one or more non-transitory computer-readable media of claim 13 , wherein the automating includes providing the reference video to the sending device.
17. The one or more non-transitory computer-readable media of claim 13 , wherein the automating includes instructing the sending device to transmit the video stream to the receiving device in a peer-to-peer manner or in a client-server manner through a video relay server.
18. The one or more non-transitory computer-readable media of claim 13 , wherein the operations further comprise repeating the automating, the receiving the video, and the receiving the transmission metrics multiple times with the same network impairment settings and the same reference video and performing the calculating for the received videos based on the reference video, the received transmission metrics, and the one or more QoE models.
19. The one or more non-transitory computer-readable media of claim 18 , wherein the operations further comprise repeating the automating, the receiving the video, the receiving the transmission metrics, and the calculating with different network impairment settings.
20. The one or more non-transitory computer-readable media of claim 19 , wherein the operations further comprise producing a model associating QoE scores with one or both of network impairment settings or transmission metrics.