Text based image search
Method and system for building a machine learning model for finding visual targets from text queries, the method comprising the steps of receiving a set of training data comprising text attribute labelled images, wherein each image has more than one text attribute label. Receiving a first vector space comprising a mapping of words, the mapping defining relationships between words. Generating a visual feature vector space by grouping images of the set of training data having similar attribute labels. Mapping each attribute label within the training data set on to the first vector space to form a second vector space. Fusing the visual feature vector space and the second vector space to form a third vector space. Generating a similarity matching model from the third vector space.
1. A method for building a machine learning model for finding visual targets from text queries, the method comprising the steps of:
receiving a set of training data comprising text attribute labelled images, wherein each image has more than one text attribute label;
receiving a first vector space comprising a mapping of words, the mapping defining relationships between words;
generating a visual feature vector space by grouping images of the set of training data having similar attribute labels;
mapping each attribute label within the training data set on to the first vector space to form a second vector space;
fusing the visual feature vector space and the second vector space to form a third vector space;
generating a similarity matching model from the third vector space; and
obtaining a global textual embedding, z glo , according to:
𝓏
g
l
o
=
f
(
{
𝓏
i
loc
}
i
=
1
N
att
)
=
Tanh
(
∑
i
=
1
N
att
(
w
2
i
·
Tanh
(
w
1
i
·
𝓏
i
loc
)
)
)
where w 1 and w 2 are learnable parameters and Tan h is a non-linear activation function of a neuron in a Convolutional Neural Network, CNN,
wherein mapping each attributed label within the training data set on to the first vector space to form a second vector space further comprises embedding each attribute label, z i loc , i∈{1, . . . , N att }.
2. The method of claim 1 , wherein the similarity matching model is generated using a mean square error loss function.
3. The method of claim 2 , wherein the mean square error loss function is:
_“mse”=1/N_“batch”Σ_( i =1){circumflex over ( )}(N_batch) ( y _ i −y⊥{circumflex over ( )}_ i _{circumflex over ( )}2
where y i and y⊥{circumflex over ( )}_i denote the ground-truth and predicted similarity of the i-th training pair, respectively and a mini-batch size is specified by N batch .
4. The method according to claim 1 , wherein the first vector space is based on a Wikipedia pre-trained word2vector model.
5. The method according to claim 1 , wherein the textual terms within the first vector space include the words of the text labels of the images within the training data set.
6. The method according to claim 1 , wherein generating the visual feature vector space by grouping images of the set of training data having similar attribute labels further comprises discriminative learning using a softmax Cross Entropy loss in a Deep Convolutional Neural Network, CNN, where each attribute label is treated as a separate classification task, cls , according to
ℒ
cls
=
-
1
N
b
a
t
c
h
∑
i
=
1
N
batch
∑
j
=
1
N
attr
log
(
p
ij
)
,
where p ij is a probability estimate of an i-th training sample on a j-th ground truth attribute.
7. The method of claim 1 , further comprising discriminative learning using a softmax Cross Entropy loss, where each attribute label is treated as a separate classification task, cls , according to
ℒ
cls
=
-
1
N
b
a
t
c
h
∑
i
=
1
N
batch
∑
j
=
1
N
attr
log
(
p
ij
)
,
where p ij is a probability estimate of an i-th training sample on a j-th ground truth attribute.
8. The method according to claim 1 , wherein generating the visual feature vector space by grouping images of the set of training data having similar attribute labels further comprises building local attribute-specific embedding:
( x i loc , i∈{ 1, . . . , N att })
based on a global part (x glo ) in a ResNet-50 CNN architecture.
9. The method according to claim 1 , wherein fusing the visual feature vector space and the second vector space to form the third vector space further comprises element-wise multiplication.
10. The method of claim 9 , wherein the element-wise multiplication is a Hadamard Product in CNN learning optimization.
11. The method of claim 10 , wherein for each attribute label a separate lightweight branch with two fully connected, FC, layers of a deep CNN are used.
12. The method of claim 10 , further comprising cross-modality global-level embedding s glo according to:
s glo =x glo ∘z glo
wherein ∘ specifies the Hadamard Product.
13. The method according to claim 1 , further comprising unlabelled images from a text query using the similarity matching model.
14. A method for building a machine learning model for finding visual targets from text queries, the method comprising the steps of:
receiving a set of training data comprising text attribute labelled images, wherein each image has more than one text attribute label;
receiving a first vector space comprising a mapping of words, the mapping defining relationships between words;
generating a visual feature vector space by grouping images of the set of training data having similar attribute labels;
mapping each attribute label within the training data set on to the first vector space to form a second vector space;
fusing the visual feature vector space and the second vector space to form a third vector space; and
generating a similarity matching model from the third vector space
estimating a per-attribute quality, P i loc , using minimum prediction scores on image and text as:
ρ i loc =min( p i vis ,p i tex ), i∈{ 1, . . . , N att }
where p i vis and P i tex denote ground-truth class posterior probability estimated by a corresponding classifier,
wherein fusing the visual feature vector space and the second vector space to form the third vector space further comprises forming per-attribute cross-modality embedding according to:
s i loc =x i loc ∘z i loc , i∈{ 1, . . . , N att }
wherein fusing the visual feature vector space and the second vector space to form the third vector space is based on a quality aware fusion algorithm.
15. The method of claim 14 , further comprising adaptively cross-attribute embedding according to:
s loc =f ({p i loc ·s i loc } i=1 N att ).
16. The method of claim 15 , further comprising forming a final cross-modality cross-level embedding according to:
s=f ({ s loc , s glo })
where the final embedding s is used to estimate an attribute matching result y.
17. One or more non-transitory computer readable media storing computer readable instructions which, when executed by a processor of a wireless communication device, cause the device to perform:
receiving a set of training data comprising text attribute labelled images, wherein each image has more than one text attribute label;
receiving a first vector space comprising a mapping of words, the mapping defining relationships between words;
generating a visual feature vector space by grouping images of the set of training data having similar attribute labels;
mapping each attribute label within the training data set on to the first vector space to form a second vector space;
fusing the visual feature vector space and the second vector space to form a third vector space; and
generating a similarity matching model from the third vector space; and
obtaining a global textual embedding, z glo , according to:
𝓏
g
l
o
=
f
(
{
𝓏
i
loc
}
i
=
1
N
att
)
=
Tanh
(
∑
i
=
1
N
att
(
w
2
i
·
Tanh
(
w
1
i
·
𝓏
i
loc
)
)
)
where w 1 and w 2 are learnable parameters and Tan h is a non-linear activation function of a neuron in a Convolutional Neural Network, CNN,
wherein mapping each attributed label within the training data set on to the first vector space to form a second vector space further comprises embedding each attribute label, z i loc , i∈{1, . . . , N att }.