Systems and methods for generating a motion attention model
View Patent ↗Systems and methods to generate a motion attention model of a video data sequence are described. In one aspect, a motion saliency map B is generated to precisely indicate motion attention areas for each frame in the video data sequence. The motion saliency maps are each based on intensity I, spatial coherence Cs, and temporal coherence Ct values. These values are extracted from each block or pixel in motion fields that are extracted from the video data sequence. Brightness values of detected motion attention areas in each frame are accumulated to generate, with respect to time, the motion attention model.
1. A method for generating a motion attention model of a video data sequence, the method comprising:
generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C s , and temporal coherence C t values from each location of a block MB ij in motion fields extracted from the video data sequence;
accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence; and
wherein I is a normalized magnitude of a motion vector that is calculated according to:
I
(
i
,
j
)
=
dx
i
,
j
2
+
dy
i
,
j
2
/
Max
Mag
,
wherein (dx i,j , dy i,j ) denotes two components of the motion vector in motion field, and MaxMag is the maximum magnitude of motion vectors.
2. A method for generating a motion attention model of a video data sequence, the method comprising:
generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C s , and temporal coherence C t values from each location of a block MB ij in motion fields extracted from the video data sequence;
accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence; and
wherein C s is calculated with respect to spatial widow w as follows:
Cs
(
i
,
j
)
=
-
∑
t
=
1
n
p
s
(
t
)
Log
(
p
s
(
t
)
)
,
p
s
(
t
)
=
SH
i
,
j
w
(
t
)
/
∑
k
=
1
n
SH
i
,
j
w
(
k
)
;
and
wherein SH w i,j (t) is a spatial phase histogram whose probability distribution function is p s (t), and n is a number of histogram bins.
3. A method for generating a motion attention model of a video data sequence, the method comprising:
generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C s , and temporal coherence C t values from each location of a block MB ij in motion fields extracted from the video data sequence;
accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence; and
wherein C t is calculated with respect to a sliding window of size L frames along time t axis as:
Ct
(
i
,
j
)
=
-
∑
t
=
1
n
p
t
(
t
)
Log
(
p
t
(
t
)
)
,
p
t
(
t
)
=
TH
i
,
j
L
(
t
)
/
∑
k
=
1
n
TH
i
,
j
L
(
k
)
;
and
wherein TH L i,j (t) is a temporal phase histogram whose probability distribution function is p t (t), and n is a number of histogram bins.
4. A method for generating a motion attention model of a video data sequence, the method comprising:
generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C s , and temporal coherence C t values from each location of a block MB ij in motion fields extracted from the video data sequence;
accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence; and
wherein the method further comprises instructions for generating the motion saliency map B according to
B=I×C t ×(1 I×C s ).
5. The method of claim 1 , wherein the video data sequence is in an MPEG data format.
6. The method of claim 1 wherein the motion field is a motion vector field, or an optical flow field.
7. The method of claim 1 , wherein the motion attention model is calculated according to:
M
motion
=
(
∑
r
∈
Λ
∑
q
∈
Ω
r
B
q
)
/
N
MB
,
B q being brightness of a block in the motion saliency map, Λ being the set of detected motion attention areas, Ω r denoting a set of blocks in each detected motion attention area, N MB being a number of blocks in a motion field; and
wherein an M motion value for each frame in the video data sequence represents a continuous motion attention curve with respect to time.
8. A computer-readable medium for generating a motion attention model of a video data sequence, the computer-readable medium comprising computer-program instructions executable by a processor for:
generating a motion saliency map B to precisely indicate motion attention areas for each frame in the video data sequence, the motion saliency map being based on intensity I, spatial coherence C s , and temporal coherence C t values from each location of a block MB ij in motion fields extracted from the video data sequence; and
accumulating brightness of detected motion attention areas to generate, with respect to time, a motion attention model for the video data sequence.
9. A computer-readable medium for generating a motion attention model of a video data sequence, the computer-readable medium comprising computer-program instructions executable by a processor for:
extracting a motion field between a current frame and a next frame of the video data sequence;
determining, at each location of a block MB ij , intensity I, spatial coherence C s , and temporal coherence C t values from the motion field;
integrating intensity I, spatial coherence C s , and temporal coherence C t to generate a motion saliency map B, the motion saliency map precisely indicating motion attention areas in the motion field; and
accumulating brightness of detected motion attention areas to indicate a motion attention degree for the current frame.
10. The computer-readable medium of claim 9 , wherein the video data sequence is in an MPEG data format.
11. The computer-readable medium of claim 10 , wherein the motion vector field is a motion vector field in MPEG stream, an optical flow field, or the result of block-based motion estimation.
12. The computer-readable medium of claim 9 , wherein the computer-program instructions for determining intensity I further comprise instructions for generating intensity I as a normalized magnitude of the motion vector according to:
I
(
i
,
j
)
=
dx
i
,
j
2
+
dy
i
,
j
2
/
Max
Mag
,
wherein (dx i,j , dy i,j ) denotes two components of the motion vector, and MaxMag is the maximum magnitude of motion vector.
13. The computer-readable medium of claim 9 , wherein the computer-program instructions for determining spatial coherence C s further comprise instructions for:
generating a phase histogram in a spatial window with the size of w×w (pixels) at each location of the block;
measuring phase distribution is measured by entropy as follows:
Ct
(
i
,
j
)
=
-
∑
t
=
1
n
p
t
(
t
)
Log
(
p
t
(
t
)
)
,
p
t
(
t
)
=
TH
i
,
j
L
(
t
)
/
∑
k
=
1
n
TH
i
,
j
L
(
k
)
;
and
wherein SH w i,j (t) is a spatial phase histogram whose probability distribution function is p s (t), and ii is a number of histogram bins.
14. The computer-readable medium of claim 9 , wherein the computer-program instructions for determining further comprise instructions for generating temporal coherence C t further comprise instructions for:
generating temporal coherence C t with respect to a sliding window of size L frames along time t axis, as:
Ct
(
i
,
j
)
=
-
∑
t
=
1
n
p
t
(
t
)
Log
(
p
t
(
t
)
)
,
p
t
(
t
)
=
TH
i
,
j
L
(
t
)
/
∑
k
=
1
n
TH
i
,
j
L
(
k
)
;
and
wherein TH L i,j (t) is a temporal phase histogram whose probability distribution function is p t (t), and n is a number of histogram bins.
15. The computer-readable medium of claim 9 , wherein the computer-program instructions for integrating further comprises instructions for generating the motion saliency map B according to B=I×C t ×(1−I×C s ).
16. The computer-readable medium of claim 9 , wherein the computer-program instructions for accumulating brightness further comprise instructions for detecting salient motion regions by employing histogram balance, median filtering, binarization, region growing, and/or region selection to the detected motion attention areas.
17. The computer-readable medium of claim 9 , further comprising computer-program instructions for executing the extracting, determining, integrating, and accumulating for each frame in the video data sequence to generate a motion attention model for all frames in the video data sequence with respect to time.
18. The computer-readable medium of claim 17 , wherein the motion attention model for all frames is calculated according to:
M
motion
=
(
∑
r
∈
Λ
∑
q
∈
Ω
r
B
q
)
/
N
MB
,
B q being brightness of a block in the motion saliency map, Λ being the set of detected motion attention areas, Ω r denoting a set of blocks in each detected motion attention area, N MB being a number of blocks in a motion field; and
wherein an M motion value for each frame in the video data sequence represents a continuous motion attention curve with respect to time.
19. A computing device for generating a motion attention model of a video data sequence that includes multiple frames, the computing device comprising:
a processor; and
a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for,
for each frame in the video data sequence:
(a) extracting a motion field between a current frame and a next frame of the video data sequence;
(b) determining, for each block MB ij represented by the motion field, intensity I, spatial coherence C s , and temporal coherence C t values to generate a motion saliency map B, the motion saliency map precisely indicating motion attention areas for each frame in the video data sequence; and
accumulating brightness of detected motion attention areas to generate the motion attention model with respect to time.
20. The computing device of claim 19 , wherein the video data sequence is in an MPEG data format.
21. The computing device of claim 19 , wherein the motion field is a motion vector field, or an optical flow field.
22. The computing device of claim 19 , wherein the computer-program instructions for determining intensity I farther comprise instructions for generating intensity I as a normalized magnitude of the motion vector according to:
I
(
i
,
j
)
=
dx
i
,
j
2
+
dy
i
,
j
2
/
MaxMag
,
wherein (dx i,j , dy i,j ) denotes two components of the motion vector, and MaxMag is the maximum magnitude of motion vector.
23. The computing device of claim 19 , wherein the computer-program instructions for determining spatial coherence C s further comprise instructions for:
generating a phase histogram in a spatial window with the size of w×w (pixels) at each location of the block;
measuring phase distribution is measured by entropy as follows:
Cs
(
i
,
j
)
=
-
∑
t
=
1
n
p
s
(
t
)
Log
(
p
s
(
t
)
)
,
p
s
(
t
)
=
SH
i
,
j
w
(
t
)
/
∑
k
=
1
n
SH
i
,
j
w
(
k
)
;
and
wherein SH w i,j (t) is a spatial phase histogram whose probability distribution function is p s (t), and n is a number of histogram bins.
24. The computing device of claim 19 , wherein the computer-program instructions for determining further comprise instructions for generating temporal coherence C t further comprise instructions for:
generating temporal coherence C t with respect to a sliding window of size L frames along time t axis, as:
Ct
(
i
,
j
)
=
-
∑
t
=
1
n
p
t
(
t
)
Log
(
p
t
(
t
)
)
,
p
t
(
t
)
=
TH
i
,
j
L
(
t
)
/
∑
k
=
1
n
TH
i
,
j
L
(
k
)
;
and
wherein TH L i,j (t) is a temporal phase histogram whose probability distribution function is p 1 (t), and n is a number of histogram bins.
25. The computing device of claim 19 , wherein the computer-program instructions for integrating further comprises instructions for generating the motion saliency map B according to B=I×C t ×(1−I×C s ).
26. The computing device of claim 19 , wherein the computer-program instructions for accumulating brightness further comprise instructions for detecting salient motion regions by employing histogram balance, median filtering, binarization, region growing, and/or region selection to the detected motion attention areas.
27. The computing device of claim 19 , wherein the motion attention model is calculated according to:
M
motion
=
(
∑
r
∈
Λ
∑
q
∈
Ω
r
B
q
)
/
N
MB
,
B q being brightness of a block in the motion saliency map, Λ being the set of detected motion attention areas, Ω r denoting a set of blocks in each detected motion attention area, N MB being a number of blocks in a motion field; and
wherein an M motion value for each frame in the video data sequence represents a continuous motion attention curve with respect to time.