System and method for determining video codec performance in real-time communication over internet
A system for determining the performance of a video codec for real-time communication applications includes a video codec performance evaluation network model and video codec quality analyze. A set of network conditions is used to simulate certain typical real-world network conditions. A coded video stream is transmitted from the transmitting end to the receiving end under these conditions. The end-to-end latency and received video fluency are measured along with a set of existing video quality measures by a video codec performance evaluation system including a network model and a video codec quality analyzer. The decodable frame ratio, latency and video fluency are used as the performance metrics for real-time communication video quality evaluation. The video codec performance evaluation system does not send any video data over a network when it determines the real-time communication quality of the video codec.
1 . A video codec performance evaluation system for evaluating the performance of a video codec in real-time communication (RTC) use cases, said video codec performance evaluation system comprising:
(1) a video codec performance evaluation network model adapted to assess a video codec's performance relative to a reference video codec, said video codec including a video encoder and a video decoder, said video codec performance evaluation network model adapted to:
a. build a set of f-g lookup tables including packet loss ratio values, network latency values, retransmission values, arriving probability values and expected arriving delay values;
b. determine a set of quantization parameter values;
c. determine a set of seeds for random number generation; and
d. for each value within said set of quantization parameter values and each value within said set of seeds, determine a set of frame decoding data items for a video sequence, thereby forming a list of sets of frames decoding data items, said video sequence including a set of frames; and
(2) a video codec quality analyzer adapted to:
a. from each set of frame decoding data items within said list of sets of frame decoding data items,
i. determine a decodable frame ratio, thereby forming a list of decodable frame ratios;
ii. determine a delay of decodable frames, thereby forming a list of delays of decodable frames; and
iii. determine a video fluency, thereby forming a list of video fluencies;
b. from said list of decodable frame ratios, determine a final decodable frame ratio;
c. from said list of delays of decodable frames, determine a final delay of decodable frames; and
d. from said list of video fluencies, determine a final video fluency.
2 . The video codec performance evaluation system of claim 1 wherein said video codec performance evaluation network model determines a set of frame decoding data items by, for each frame within said set of frames:
(1) determining a sending timestamp;
(2) retrieving an encoded bitstream of said frame from said video encoder;
(3) splitting said encoded bitstream into a set of units;
(4) packetizing each unit into a set of packets;
(5) for each unit within said set of units,
a. determining an arriving probability value from said set of f-g lookup tables;
b. determining an expected arriving delay value from said set of f-g lookup tables;
c. generating a random number;
d. comparing said random number with said arriving probability value to determine said unit is received or lost; and
(6) determining a receiving timestamp;
(7) when all units of said frame are received and said frame is decodable, recording a frame decoding data item indicating said frame receiving timestamp and that said frame is decodable; and
(8) when not all units of said frame are received or said frame is not decodable, recording a frame decoding data item indicating said frame receiving timestamp and that said frame is not decodable.
3 . The video codec performance evaluation system of claim 2 wherein said video codec performance evaluation network model determines a set of frame decoding data items further by:
(1) from each set of frame decoding data items within said list of sets of frame decoding data items, determining a maximum delay of decodable frames, thereby forming a list of maximum delays of decodable frames; and
(2) from said list of maximum delays of decodable frames, determining a final maximum delay of decodable frames.
4 . The video codec performance evaluation system of claim 2 wherein said arriving probability values and said expected arriving delay values in said f-g lookup tables are determined using respective formulas below:
f
(
r
,
s
,
N
)
=
floor
(
10000
(
1
-
r
s
+
1
)
N
)
g
(
r
,
d
,
s
,
N
)
=
floor
(
(
∑
i
=
1
s
(
f
(
r
,
i
,
N
)
-
f
(
r
,
i
-
1
,
N
)
)
(
2
i
+
1
)
+
f
(
r
,
0
,
N
)
)
d
f
(
r
,
s
,
N
)
)
wherein r stands for a predetermined packet loss ratio, d stands for a predetermined network latency, s stands for a predetermined number of retransmissions, and N stands for the number of packets within said unit.
5 . The video codec performance evaluation system of claim 4 wherein:
(1) each frame decoding data item within said set of frame decoding data items indicates said receiving timestamp and whether said frame is decodable or not;
(2) said decodable frame ratio is ratio between the number of frames within said set of frames and the number frames within said set of frames that are decodable;
(3) said delay of decodable frames is determined using a formula below:
∑
i
=
1
N
D
(
t
r
k
i
-
t
s
k
i
)
N
D
wherein,
t
r
k
i
stands for the receiving timestamp of the frame k i ,
k
i
,
t
s
k
i
stands for the sending timestamp of the frame k i and N D stands for the number of decodable frames within the set of frames;
(4) said video fluency is determined using a formula below:
∑
i
=
1
N
D
-
1
(
t
r
k
i
+
1
-
(
t
r
k
i
+
1
fps
)
)
2
+
(
t
r
k
1
-
t
r
1
)
2
+
(
t
r
N
E
-
t
r
k
N
D
)
2
N
E
wherein fps stands for the number of frames per second and N E stands for the number of decodable frames within the set of frames;
(5) said receiving timestamp is determined using a formula below:
t
r
n
=
t
s
n
+
max
0
≤
m
<
M
g
(
r
,
d
,
s
,
N
m
)
wherein n stands for the index number of the frame within the set of frames.
6 . The video codec performance evaluation system of claim 5 wherein said video codec performance evaluation network model determines a set of frame decoding data items further by:
(1) retrieving a video frame decoding feedback of a processed frame;
(2) determining whether to forward said video frame decoding feedback to said video encoder that encodes said frame; and
(3) forwarding said video frame decoding feedback to said video encoder.
7 . The video codec performance evaluation system of claim 6 wherein said video frame decoding feedback is forwarded to said video encoder when
t
r
k
+
d
<
t
s
n
.
8 . The video codec performance evaluation system of claim 2 wherein said video codec performance evaluation network model determines a set of frame decoding data items further by:
(1) retrieving a video frame decoding feedback of a processed frame;
(2) determining whether to forward said video frame decoding feedback to said video encoder that encodes said frame; and
(3) forwarding said video frame decoding feedback to said video encoder.
9 . The video codec performance evaluation system of claim 5 wherein said video codec performance evaluation network model determines a set of frame decoding data items further by:
(1) from each set of frame decoding data items within said list of sets of frame decoding data items, determining a maximum delay of decodable frames, thereby forming a list of maximum delays of decodable frames; and
(2) from said list of maximum delays of decodable frames, determining a final maximum delay of decodable frames.
10 . The video codec performance evaluation system of claim 9 wherein said maximum delay of decodable frames is determined using a formula below:
max
1
≤
i
≤
N
D
(
t
r
k
i
-
t
s
k
i
)
.
11 . The video codec performance evaluation system of claim 5 wherein said final decodable frame ratio is an average of decodable frame ratios within said list of decodable frame ratios.
12 . The video codec performance evaluation system of claim 5 wherein said final delay of decodable frames is an average of delays of decodable frames within said list of delays of decodable frames.
13 . The video codec performance evaluation system of claim 5 wherein said final video fluency is an average of final video fluencies within said list of video fluencies.
14 . The video codec performance evaluation system of claim 2 wherein said final decodable frame ratio is an average of decodable frame ratios within said list of decodable frame ratios.
15 . The video codec performance evaluation system of claim 2 wherein said final delay of decodable frames is an average of delays of decodable frames within said list of delays of decodable frames.
16 . The video codec performance evaluation system of claim 2 wherein said final video fluency is an average of final video fluencies within said list of video fluencies.
17 . The video codec performance evaluation system of claim 8 wherein said video codec performance evaluation network model determines a set of frame decoding data items further by:
(1) from each set of frame decoding data items within said list of sets of frame decoding data items, determining a maximum delay of decodable frames, thereby forming a list of maximum delays of decodable frames; and
(2) from said list of maximum delays of decodable frames, determining a final maximum delay of decodable frames.