Visual complexity measure for playing videos adaptively
View Patent ↗A method plays frames of a video adaptively according to a visual complexity of the video. First a spatial frequency of pixel within frames of the video is measured, as well as a temporal velocity of corresponding pixels between frames of the video. The spatial frequency is multiplied by the temporal velocity to obtain a measure of the visual complexity of the frames of the video. The frames of the video are then played at a speed that corresponds to the visual complexity.
1. A method for playing frames of a video adaptively, comprising the steps of:
measuring a spatial frequency of pixel within frames of the video, wherein the spatial frequency is measured from discrete cosine transform coefficients of the pixels in the frames, and wherein basis functions of the discrete cosine transformation are in a form
cos
(
π
k
x
(
2
x
+
1
)
2
N
)
·
cos
(
π
k
y
(
2
y
+
1
)
2
N
)
=
cos
(
2
π
k
x
2
N
x
+
2
π
k
4
N
)
·
cos
(
2
π
k
y
2
N
y
+
2
π
k
4
N
)
,
where k x is a frequency f x in an x direction and k y is a frequency f y in a y direction in the frame represented as
cos
(
2
π
f
x
N
x
+
2
π
f
y
N
y
)
,
where N is 8 for DCT macro-blocks, and each DCT basis is a superimposition of two two dimensional sinusoids;
measuring a temporal velocity of corresponding pixels between frames of the video, wherein the temporal velocity is measured from motion vectors of corresponding pixels between the frames;
multiplying the spatial frequency by the temporal velocity to obtain a measure of visual complexity of the frames of the video; and
playing the frames of the video at a frame rate that corresponds to the measure of visual complexity.
2. The method of claim 1 wherein the video is compressed.
3. The method of claim 1 wherein each basis function is a superimposition of two 2D sinusoids, one with a spatial frequency
f
->
1
=
(
k
x
2
,
k
y
2
)
and another with a spatial frequency
f
->
2
=
(
k
x
2
,
k
y
2
)
.
4. The method of claim 3 wherein a particular motion vector is {right arrow over (v)}=(v x ,v y ).
5. The method of claim 4 wherein the visual complexity resulting from the discrete cosine coefficient and the motion vectors are
ω
1
=
f
→
1
·
v
→
1
=
k
x
2
v
x
+
k
y
2
v
y
,
and
ω
2
=
f
→
2
·
v
→
2
=
k
x
2
v
x
-
k
y
2
v
y
.
6. The method of claim 1 further comprising:
discarding motion vectors with a low texture;
median filtering the motion vectors; and
fitting a global motion model to the motion vectors to reduce spurious motion vectors.
7. The method of claim 1 wherein the compressed video includes I-frames and P-frames, and further comprising:
determined discrete cosine transformation coefficients of the P-frames by applying motion compensation; and
determining motion vectors for the I-frames by interpolating the motion vectors of the P-frames.
8. The method of claim 1 further comprising:
averaging the visual complexity over a set of frames to determine a complexity of a video segment.
9. The method of claim 1 further comprising:
applying motion blur while playing the video to reduce aliasing.
10. The method of claim 1 wherein the frame rate of playing is inversely proportional to the visual complexity.
11. The method of claim 1 further comprising:
applying coring to spatial filter the video while playing.
12. The method of claim 1 wherein the video is uncompressed.
13. The method of claim 1 , in which a temporal distortion of the video is minimized during playback.
14. The method of claim 13 , in which the minimizing uses a quantization of the visual complexity.
15. The method of claim 13 , in which the minimizing uses a smoothing and filtering of the visual complexity.
16. The method of claim 13 , in which the minimizing constructs a piece-wise linear approximation of the visual complexity so that the visual complexity is substantially linear.
17. The method of claim 13 , in which the minimizing assigns a constant visual complexity to a consistent temporal segment of the video.