System and method for image annotation and multi-modal image retrieval using probabilistic semantic models
View Patent ↗Systems and Methods for multi-modal or multimedia image retrieval are provided. Automatic image annotation is achieved based on a probabilistic semantic model in which visual features and textual words are connected via a hidden layer comprising the semantic concepts to be discovered, to explicitly exploit the synergy between the two modalities. The association of visual features and textual words is determined in a Bayesian framework to provide confidence of the association. A hidden concept layer which connects the visual feature(s) and the words is discovered by fitting a generative model to the training image and annotation words. An Expectation-Maximization (EM) based iterative learning procedure determines the conditional probabilities of the visual features and the textual words given a hidden concept class. Based on the discovered hidden concept layer and the corresponding conditional probabilities, the image annotation and the text-to-image retrieval are performed using the Bayesian framework.
1. A computer-implemented method of retrieving media, comprising:
(a) creating a probabilistic framework relating media types within a mixed media work to implicit concepts;
(b) creating an index of a set of media based on implicit concepts within the probabilistic framework;
(c) receiving a query expressed in the form of a media exemplar;
(d) determining a set of concepts expressed in the media exemplar;
(e) searching the index of the set of media for elements representing similar implicit concepts to those expressed in the media exemplar; and
(f) outputting at least one representation or identifier of the elements representing similar implicit concepts to those expressed in the media exemplar;
wherein said outputting comprises outputting a representation or identifiers of a plurality of elements, further comprising: ranking the plurality of elements based on at least a similarity of the respective element to implicit concepts expressed in the media exemplar;
wherein said determining comprises probabilistically determining a set of semantic concepts inherent in the media exemplar, based on correlations of features in respective multimedia works having predetermined semantic concepts associated therewith;
further comprising determining a concept vector for the media exemplar;
wherein the probabilistic framework comprises a Bayesian model for associating words with an image having visual features, comprising a hidden concept layer which connects a visual feature layer and a word layer which is discovered by fitting a generative model to a training set comprising images having the visual features and annotation words, wherein the conditional probabilities of the visual features and the annotation words given a hidden concept class are determined based on an Expectation-Maximization (EM) based iterative learning procedure.
2. The method according to claim 1 , wherein f 1 ,iε[1,N] denotes a visual feature vector of images in a training database, where N is the size of the database, w i ,jε[1,M] denotes the distinct textual words in a training annotation word set, where M is the size of an annotation vocabulary in the training database, the visual features of images in the database, f i =[f i 1 , f i 2 , . . . , f i L ],iε[1,N] are known i.i.d. samples from an unknown distribution, having a visual feature dimension L, the specific visual feature annotation word pairs (f i ,w j ), iε[1,N],jε[1,M] are known i.i.d. samples from an unknown distribution, associated with an unobserved semantic concept variable zεZ={z 1 . . . z k }, in which each observation of one visual feature fεF={f i , f 2 , . . . , f N } belongs to one or more concept classes z k and each observation of one word wεV={w 1 , w 2 , . . . , w M } in one image f belongs to one concept class, in which the observation pairs (f i ,w j ) are assumed to be generated independently, and the pairs of random variables (f i ,w j ) are assumed to be conditionally independent given the respective hidden concept z k , such that
P ( f i ,w j |z k )= p ℑ ( f i |z k ) P V ( w j |z k ).
3. The method according to claim 2 , in which the visual feature and word distribution is treated as a randomized data generation process, wherein a probability of a concept is represented as P z (z k ); a visual feature is selected f i εF with probability P ℑ (f i |z k ); and a textual word is selected w j εV with probability P V (w j |z k ), from which an observed pair (f i ,w j ) is obtained, such that a joint probability model is expressed as follows:
P
(
f
i
,
w
j
)
=
P
(
w
j
)
P
(
f
i
❘
w
j
)
=
P
(
w
j
)
∑
k
=
1
K
P
(
f
i
❘
z
k
)
P
(
z
k
❘
w
j
)
=
∑
k
=
1
K
P
z
(
z
k
)
P
(
f
i
❘
z
k
)
P
V
(
w
j
❘
z
k
)
,
and the visual features are generated from K Gaussian distributions, each one corresponding to a z k , such that for a specific semantic concept variable z k , the conditional probability density function of visual feature f i is expressed as:
P
(
f
i
❘
z
k
)
=
1
2
π
L
/
2
❘
∑
k
1
/
2
ⅇ
-
1
2
(
f
i
-
μ
k
)
T
∑
k
-
1
(
f
i
-
μ
k
)
where Σ k and μ k are the covariance matrix and mean of visual features belonging to z k , respectively.
4. The method according to claim 3 , in which word concept conditional probabilities P V (●|Z), i.e., P V (w j |z k ) for kε[1,K], are estimated through fitting the probabilistic model to the training set.
5. The method according to claim 4 , in which P ℑ (f i |z k ) is determined by maximization of the log-likelihood function:
log
∏
i
=
1
N
p
(
f
i
❘
Z
)
u
i
=
∑
i
=
1
N
u
i
log
(
∑
k
=
1
K
P
z
(
z
k
)
p
(
f
i
❘
z
k
)
)
where μ i is the number of annotation words for image f i , and P z (z k ) and P V (w j |z k ) are determined by maximization of the log-likelihood function:
=
log
P
(
F
,
V
)
=
∑
i
=
1
N
∑
j
=
1
M
n
(
w
i
j
)
log
P
(
f
i
,
w
j
)
where n(w i j ) denotes the weight of annotation word w j , i.e., occurrence frequency, for image f i .
6. The method according to claim 5 , wherein the model is resolved by applying the expectation-maximization (EM) technique, comprising: (i) an expectation (E) step where the posterior probabilities are computed for the hidden variable z k based on the current estimates of the parameters; and (ii) an maximization (M) step, where parameters are updated to maximize the expectation of the complete-data likelihood log P (F,V,Z) given the posterior probabilities computed in the preceding E-step, whereby the probabilities can be iteratively determined by fitting the model to the training image database and the associated annotations.
7. The method according to claim 6 , wherein Bayes' rule is applied to determine the posterior probability for z k under f i and (f i ,w j ):
p
(
z
k
❘
f
i
)
=
P
z
(
z
k
)
p
(
f
i
❘
z
k
)
∑
t
=
1
K
P
z
(
z
t
)
P
(
f
i
❘
z
t
)
P
V
(
w
j
❘
z
t
)
p
(
z
k
❘
f
i
,
w
j
)
=
P
z
(
z
k
)
P
z
(
f
i
❘
z
k
)
P
V
(
w
j
❘
z
k
)
∑
t
=
1
K
P
z
(
z
t
)
P
(
f
i
❘
z
t
)
P
V
(
w
j
❘
z
t
)
and expectation of the complete-data likelihood log P (F,V,Z) for the estimated log P(F|V,Z) is
∑
(
i
,
j
)
=
1
K
∑
i
=
1
N
∑
j
=
1
M
n
(
w
i
j
)
log
[
P
z
(
z
i
,
j
)
p
(
f
i
❘
z
i
,
j
)
P
V
(
w
j
❘
z
i
,
j
)
]
P
(
Z
❘
F
,
V
)
where
P
(
Z
❘
F
,
V
)
=
∏
s
=
1
N
∏
t
=
1
M
P
(
z
s
,
t
❘
f
s
,
w
t
)
and the notation z i,j is the concept variable that associates with the feature-word pair (f i ,w j ).
8. The method according to claim 6 , wherein the expectation of the likelihood log P(F,Z) for the estimated P(Z|F) may be expressed as:
∑
k
=
1
K
∑
i
=
1
N
log
(
P
z
(
z
k
)
p
(
f
i
❘
z
k
)
)
p
(
z
k
❘
f
i
)
and the expectation of log P(F,V,Z) and log P(F|V,Z) are maximized with Lagrange multipliers to P z (z l ), p ℑ (f u |z l ) and P V (w v |z l ), under the normalization constraints
∑
k
=
1
K
P
z
(
z
k
)
=
1
,
∑
k
=
1
K
P
(
z
k
❘
f
i
,
w
j
)
=
1
for any f i , w j , and z l , the parameters being determined as
μ
k
=
∑
i
=
1
N
u
i
f
i
p
(
z
k
❘
f
i
)
∑
s
=
1
N
u
s
p
(
z
k
❘
f
s
)
∑
k
=
∑
i
=
1
N
u
i
p
(
z
k
❘
f
i
)
(
f
i
-
μ
k
)
(
f
i
-
μ
k
)
T
∑
s
=
1
N
u
s
p
(
z
k
❘
f
s
)
P
z
(
z
k
)
=
∑
j
=
1
M
∑
i
=
1
N
u
(
w
i
j
)
P
(
z
k
❘
f
i
,
w
j
)
∑
j
=
1
M
∑
i
=
1
N
n
(
w
i
j
)
P
V
(
w
j
❘
z
k
)
=
∑
i
=
1
N
n
(
w
i
j
)
P
(
z
k
❘
f
i
,
w
j
)
∑
u
=
1
M
∑
V
=
1
N
n
(
w
V
u
)
P
(
z
k
❘
f
V
,
w
u
)
.
9. The method according to claim 8 , in which the number of concepts, K, is chosen to maximize
log
(
P
(
F
,
V
)
)
-
m
K
2
log
(
MN
)
where m K is the number of free parameters needed for a model with K mixture components.
10. The method according to claim 9 , in which
m K =( K− 1)+ K ( M− 1)+ K ( N− 1)+ L 2 =K ( M+N− 1)+ L 2 −1.
11. The method according to claim 10 , wherein a joint distribution P(w j ,z k ,f i ) is used to model the probability of an event that a word w j belonging to semantic concept z k is an annotation word of image f i :
P ( w j ,z k ,f i )= P z ( z k ) p ℑ ( f i |z k ) P V ( w j |z k )
or
P
(
w
j
❘
f
i
)
=
∫
P
v
(
w
j
❘
z
)
p
(
z
❘
f
i
)
ⅆ
z
=
∫
P
V
(
w
j
❘
z
)
p
(
f
i
❘
z
)
P
(
z
)
p
(
f
i
)
ⅆ
z
=
E
z
{
P
v
(
w
j
❘
z
)
p
(
f
i
❘
z
)
p
(
f
i
)
}
where
p(f i )=∫p ℑ (f i |z)P z (z)dz=E z {p ℑ (f i |z)}, and
E z {●} denotes the expectation over P(z k ).
12. The method according to claim 11 , wherein a Monte Carlo integration is used to derive
P
(
w
j
❘
f
i
)
≈
∑
k
=
1
K
P
V
(
w
j
❘
z
k
)
p
(
f
i
❘
z
k
)
∑
h
=
1
K
p
(
f
i
❘
z
h
)
=
∑
k
=
1
K
P
V
(
w
j
❘
z
k
)
x
k
where
x
k
=
p
(
f
i
❘
z
k
)
∑
h
=
1
K
p
(
f
i
❘
z
h
)
,
and wherein the words with the highest P(w j |f i ) are returned to annotate the image.
13. The method according to claim 10 , wherein images are retrieved for word queries by determining the conditional probability P(f i |w j )
P
(
f
i
❘
w
j
)
=
∫
P
(
f
i
❘
z
)
P
(
z
❘
w
j
)
ⅆ
z
=
∫
P
V
(
w
j
❘
z
)
p
(
f
i
❘
z
)
P
(
z
)
P
(
w
j
)
ⅆ
z
=
E
z
{
P
V
(
w
j
❘
z
)
p
(
f
i
❘
z
)
P
(
w
j
)
}
in which the expectation is estimated as follows:
P
(
f
i
❘
w
j
)
≈
∑
k
=
1
K
P
V
(
w
j
❘
z
k
)
p
(
f
i
❘
z
k
)
∑
h
=
1
K
P
V
(
w
j
❘
z
h
)
=
∑
k
=
1
K
p
(
f
i
❘
z
k
)
y
k
where
y
k
=
P
V
(
w
j
❘
z
k
)
∑
h
P
V
(
w
j
❘
z
h
)
,
and wherein the images in the database with the highest P(f i |w j ) are returned as the retrieval result for each query word.
14. An apparatus adapted to retrieve media, comprising:
(a) at least one memory, adapted for storing therein a probabilistic framework relating media types within a mixed media work to implicit concepts;
(b) at least one memory, adapted for storing an index of a set of media based on implicit concepts within the probabilistic framework
(c) an input, adapted to receive a query expressed in the form of a media exemplar;
(d) at least one processor, adapted to:
access the at least one memory adapted for storing a probabilistic framework
access the at least one memory adapted for storing an index of a set of media,
receive the query from the input,
determine a set of concepts expressed in the media exemplar, and
search the index of the set of media for elements representing similar implicit concepts to those expressed in the media exemplar; and
(e) an output adapted to present at least one of the elements representing similar implicit concepts to those expressed in the media exemplar;
wherein the probabilistic framework comprises a Bayesian model for associating words with an image having visual features, comprising a hidden concept layer which connects a visual feature layer and a word layer which is discovered by fitting a generative model to a training set comprising images having the visual features and annotation words, wherein the conditional probabilities of the visual features and the annotation words given a hidden concept class are determined based on an Expectation-Maximization (EM) based iterative learning procedure.
15. A non-transitory computer readable medium storing therein instructions for controlling a programmable processor to perform the steps of:
(a) creating a probabilistic framework relating media types within a mixed media work to implicit concepts;
(b) indexing a set of media based on implicit concepts within the probabilistic framework, and storing the index in a memory;
(c) receiving a query expressed as a media exemplar;
(d) determining a set of concepts expressed in the media exemplar;
(e) searching the stored index of the set of media for elements representing similar implicit concepts to the media exemplar; and
(f) outputting a representation or identifier of at least one of the elements representing similar implicit concepts to the media exemplar;
wherein the probabilistic framework comprises a Bayesian model stored in a memory for associating words with an image having visual features, comprising a hidden concept layer which connects a visual feature layer and a word layer which is discovered by fitting a generative model to a training set comprising images having the visual features and annotation words, wherein the conditional probabilities of the visual features and the annotation words given a hidden concept class are determined based on an Expectation-Maximization (EM) based iterative learning procedure.