Semantic-aware co-indexing for near-duplicate image retrieval
View Patent ↗An image retrieval method includes learning multiple object category classifiers with a processor offline and generating classifications scores of images as the semantic attributes; performing vocabulary tree based image retrieval using local features with semantic-aware co-indexing to jointly embed two distinct cues offline for near-duplicate image retrieval; and identifying top similar or dissimilar images using multiple semantic attributes.
1. An image retrieval method, comprising:
learning multiple object category classifiers with a processor offline and generating classifications scores of images as the semantic attributes; and
performing vocabulary tree based image retrieval using local features with semantic-aware co-indexing to jointly embed two distinct cues offline for near-duplicate image retrieval; and
identifying top similar or dissimilar images using multiple semantic attributes;
performing semantic-aware online querying;
determining:
sim
(
q
,
d
)
=
.
1
S
q
S
d
∑
v
i
∈
T
(
x
i
)
,
v
j
∈
T
(
x
j
)
w
v
(
v
i
,
v
j
)
,
w
v
(
v
i
,
v
j
)
=
.
idf
2
(
v
)
1
(
v
i
=
v
j
)
=
log
2
(
M
M
v
)
,
where i and j are indices, x is tree node, S is a bag of local descriptions, q a query image, d a database image, T is a vocabulary tree, w is word, idf is an inverse document frequency of node v, M is a total number of database images and M v is a number of images containing at least one descriptor that quantizes to a node v.
2. The method of claim 1 , comprising inserting and removing images to and from the inverted indexes based on semantic attributes.
3. The method of claim 1 , wherein the online query employs only local features.
4. The method of claim 1 , comprising co-indexing local features and semantic attributes offline in inverted indexes for near-duplicate image retrieval.
5. The method of claim 1 , comprising extracting the semantic attributes based on multi-class object recognition offline for the database images.
6. The method of claim 1 , comprising using similar or dissimilar relations among images to alter the inverted indexes.
7. The method of claim 1 , comprising identifying local features and top similar images according to semantic attributes in the tf-idf (term frequency-inverse document frequency) voting in online retrieval.
8. The method of claim 1 , wherein q and d are the query and a database image, respectively, comprising determining a similarity between q and d as:
sim
^
(
q
,
d
)
=
.
sim
(
q
,
d
)
+
∑
d
g
∈
G
KNN
(
d
)
ω
×
sim
(
q
,
d
g
)
,
where ω is a weighting parameter of the contribution from semantic attributes, and G KNN (d) is the set of database images whose K semantic nearest images include d.
9. The method of claim 1 , comprising determining similarity sim(q, d) between q and d as an average TF-IDF (term frequency-inverse document frequency).
10. The method of claim 1 , comprising applying a Total Variance Distance (TVD) for two images d m and d n , to measure the semantic distance between their partial probability vectors p:
TVD
(
d
m
,
d
n
)
=
∑
c
=
1
C
p
c
(
d
m
)
-
p
c
(
d
n
)
.
11. The method of claim 1 , comprising performing sematic isolated image deletion.
12. The method of claim 1 , comprising performing semantic aware co-indexing off line.
13. An image retrieval system, comprising:
a processor;
a database of images coupled to the processor;
code for learning multiple object category classifiers with a processor offline and generating classifications scores of images as the semantic attributes; and
code for performing vocabulary tree based image retrieval using local features with semantic-aware co-indexing to jointly embed two distinct cues offline for near-duplicate image retrieval; and
code for identifying top similar or dissimilar images using multiple semantic attributes;
code for performing semantic-aware online querying;
code for determining:
sim
(
q
,
d
)
=
.
1
S
q
S
d
∑
v
i
∈
T
(
x
i
)
,
v
j
∈
T
(
x
j
)
w
v
(
v
i
,
v
j
)
,
w
v
(
v
i
,
v
j
)
=
.
idf
2
(
v
)
1
(
v
i
=
v
j
)
=
log
2
(
M
M
v
)
,
where i and j are indices, x is tree node, S is a bag of local descriptions, q a query image, d a database image, T is a vocabulary tree, w is word, idf is an inverse document frequency of node v, M is a total number of database images and M v is a number of images containing at least one descriptor that quantizes to a node v.
14. The method of claim 1 , comprising code for inserting and removing images to and from the inverted indexes based on semantic attributes.
15. The method of claim 1 , wherein the online query employs only local features.
16. The method of claim 1 , comprising code for co-indexing local features and semantic attributes offline in inverted indexes for near-duplicate image retrieval.
17. The method of claim 1 , comprising code for extracting the semantic attributes based on multi-class object recognition offline for the database images.
18. The method of claim 1 , comprising code for using similar or dissimilar relations among images to alter the inverted indexes.