Video object retrieval system and method
A video object retrieval method is provided. The method includes dividing an input video into a number of video shots and detecting target-class object occurrences and related-class object occurrences in each video shot. The method also includes pre-processing object bounding box images and performing automatic object extraction. Further, the method includes performing object-level feature extraction and integration and generating visual search results based on similarity scores.
1. A video object retrieval method, comprising:
dividing an input video into a number of video shots;
detecting target-class object occurrences and related-class object occurrences in each video shot;
pre-processing object bounding box images;
performing automatic object extraction via an automatic graph-cut algorithm:
performing object-level feature extraction and integration; and
generating visual search results based on similarity scores.
2. The method according to claim 1 , wherein detecting target-class object occurrences and related-class object occurrences in each video shot further including:
acquiring multimodal cue information for all the detected object occurrences;
combining an appearance cue, a spatio-temporal cue, and a topological cue to aid object detection in a plurality of video shots; and
applying dynamic programming to find optimal object paths.
3. The method according to claim 1 . wherein pre-processing object bounding box images further including:
performing histogram equalization to adjust global contrast of an image; and
carrying out image fusion to balance between image quality and global contrast level.
4. The method according to claim 1 , wherein performing automatic object extraction further including:
initializing a trimap T with object rectangle consisting of a foreground, region T F , a background region T B and an uncertain pixel region T U ;
initializing background (α i =0) for iεT B and foreground (α i =1) for iεT U ∪T F , where 1≦i≦N;
initializing the foreground and background Gaussian Mixture Models (GMMs) with k-means from sets α i =1 and α i =0, respectively;
assigning pixels to GMM components and learning GMM parameters from image data; and
estimating segmentation based on the graph-cut algorithm.
5. The method according to claim 1 , wherein performing object-level feature extraction and integration further including:
performing object-level feature extraction for both auto color correlogram (ACC) and color and edge directivity descriptor (CEDD) features; and
carrying out similarity fusion.
6. The method according to claim 5 , wherein:
provided that p and q are two feature vectors; ∥p−q∥ 1 is Manhattan distance between two feature vectors p and q, the ACC feature dissimilarity is defined by:
D
pq
=
p
-
q
1
max
{
D
pq
}
+
ξ
where ξ is a constant greater than zero for avoiding division by zero.
7. The method according to claim 5 , wherein:
provided that p and q are two feature vectors; p T is the transpose of the vector p; q T is the transpose of the vector q, the CEDD feature dissimilarity is defined by:
T
pq
=
p
T
q
p
T
p
+
q
T
q
-
p
T
q
wherein p T p+q T q−p T q is not zero.
8. The method according to claim 5 , wherein:
provided that p and q are two feature vectors; Sim ACC (.) is the ACC feature similarity; Sim CEDD (.) is the CEDD feature similarity, a similarity score is defined by:
Sim
f
(
p
,
q
)
=
a
·
Sim
ACC
(
p
,
q
)
+
β
·
Sim
CEDD
(
p
,
q
)
=
a
·
(
1
-
D
pq
)
+
β
·
(
1
-
T
pq
)
where α is the weight for the ACC feature; and β is the weight for the CEDD feature.
9. A video object retrieval system, comprising:
an object detection module configured to detect target-class object occurrences and related-class object occurrences in each video shot;
a preprocessing module configured to pre-process object bounding box images;
an object extraction module configured to perform automatic object extraction via an automatic graph-cut algorithm;
an object-level feature extraction module configured to perform object-level feature extraction;
a similarity computation module configured to compute auto color correlogram (ACC) feature similarity and color and edge directivity descriptor (CEDD) feature similarity; and
a similarity fusion and ranking module configured to carry out similarity fusion and generate final visual search results based on similarity scores.
10. The system according to claim 9 , wherein the object detection module is further configured to:
acquire multimodal cue information for all the detected object occurrences;
combine an appearance cue, a spatio-temporal cue, and a topological cue to aid object detection in a plurality of video shots; and
apply dynamic programming to find optimal object paths.
11. The system according to claim 9 , wherein the preprocessing module is further configured to:
perform histogram equalization to adjust global contrast of an image; and
carry out image fusion to balance between image quality and global contrast level.
12. The system according to claim 9 , wherein the object extraction module is further configured to:
initialize a trimap T with object rectangle consisting of a foreground region T F , a background region T B and an uncertain pixel region T U ;
initialize background (α i =0) for iεT B and foreground (α i =1) for iεT U ∪T F , where 1≦i≦N;
initialize the foreground and background Gaussian Mixture Models (GMMs) with k-means from sets α i =1 and α i =0, respectively;
assign pixels to GMM components and learn GMM parameters from image data; and
estimate segmentation based on the graph-cut algorithm.
13. The system according to claim 9 , wherein the object-level feature extraction module is further configured to:
perform object-level feature extraction for both auto color correlogram (ACC) and color and edge directivity descriptor (CEDD) features.
14. The system according to claim 13 , wherein:
provided that p and q are two feature vectors; ∥p−q∥ 1 is Manhattan distance between two feature vectors p and q, the ACC feature dissimilarity is defined by:
D
pq
=
p
-
q
1
max
{
D
pq
}
+
ξ
where ξ is a constant greater than zero for avoiding division by zero.
15. The system according to claim 13 , wherein:
provided that p and q are two feature vectors; p T is the transpose of the vector p; q T is the transpose of the vector q, the CEDD feature dissimilarity is defined by:
T
pq
=
p
T
q
p
T
p
+
q
T
q
-
p
T
q
wherein p T p+q T q−p T q is not zero.
16. The system according to claim 13 , wherein:
provided that p and q are two feature vectors; Sim ACC (.) is the ACC feature similarity; Sim CEDD (.) is the CEDD feature similarity, a similarity score is defined by:
Sim
f
(
p
,
q
)
=
a
·
Sim
ACC
(
p
,
q
)
+
β
·
Sim
CEDD
(
p
,
q
)
=
a
·
(
1
-
D
pq
)
+
β
·
(
1
-
T
pq
)
where α is the weight for the ACC feature; and β is the weight for the CEDD feature.