Method and apparatus for real-world cross-modal retrieval problems
Broadly speaking, the present application generally relates to a method for training a machine learning, ML, model to perform real world cross-modal retrieval problems, and to a computer-implemented method and apparatus for performing real world cross-modal retrieval problems such as including text-based video retrieval, sketch-based image retrieval, and image-text retrieval using a trained machine learning, ML, model.
1. A computer-implemented method for training a machine learning ML model comprising a first feature extractor for extracting image features from an image and a second feature extractor, the method comprising:
obtaining a dataset comprising a plurality of pairs of data instances, with each pair comprising a plurality of first data instances having a first modality and a plurality of second data instances having a second modality, wherein the first data instance is an image;
evaluating, using the second feature extractor, at least some of the plurality of second data instances to extract a set of features for each of the plurality of second data instances;
assigning a first set of class labels to the plurality of first data instances based on the set of features for each of the plurality of second data instances;
training the first feature extractor using the assigned first set of class labels;
evaluating, using the first feature extractor, at least some of the plurality of first data instances to extract a set of image features for the plurality of first data instances;
assigning a second set of class labels to the plurality of second data instances based on the set of image features; and
training the second feature extractor using the assigned second set of class labels,
wherein the assigning of the first set of class labels comprises determining clusters of first data instances based on extracted set of features and assigning each pair of data instances in a cluster same class label, and
wherein the assigning of the second set of class labels comprises determining clusters of second data instances based on extracted set of image features and assigning each pair of data instances in a cluster same class label.
2. The method of claim 1 , further comprising:
iterating each of the evaluating, assigning and training steps for separate batches of data.
3. A non-transitory computer-readable recording medium having recorded thereon a program for executing, the method of claim 1 .
4. The method of claim 1 , wherein training the first feature extractor and training the second feature extractor uses cross-entropy minimization.
5. The method of claim 1 , wherein training the first and second feature extractors comprises defining a first linear classifier for the first modality, a second linear classifier for the second modality and a set of training parameters which are shared between the first modality and the second modality.
6. The method of claim 5 , wherein the first and second linear classifiers are denoted by p(y|x A ) and p(y|x B ) and are defined as,
p
(
y
=
j
|
x
M
)
=
exp
(
p
j
T
ϕ
M
(
x
)
/
τ
)
∑
l
exp
(
p
l
T
ϕ
M
(
x
)
/
τ
)
,
M
∈
{
A
,
B
where P={p 1 , . . . , p K } are trainable parameters that are shared between the first modality and the second modality, ϕ M represents the first and second feature extractors, x M represents the first and second feature data instances, y is the class label for class j, each p j may be regarded as prototype vector for class j that lies in shared feature space, and τ is temperature in softmax.
7. The method of claim 5 , further comprising:
using optimization to obtain the first and second linear classifiers.
8. The method of claim 7 , further comprising:
estimating a first surrogate for the first linear classifier using current second linear classifier; and
estimating a second surrogate for the second linear classifier using current first linear classifier.
9. The method of claim 8 , wherein training the first feature extractor comprises updating the first feature extractor using the estimated first and second surrogates.
10. The method of claim 8 , wherein training the second feature extractor comprises updating the second feature extractor using the estimated first and second surrogates.
11. The method of claim 8 ,
wherein the estimate for the first surrogate q(y|x A ) is found using an optimization defined as
Q
i
y
A
:
=
q
(
y
|
x
i
A
)
.
Min
Q
A
𝔼
i
∼
𝒟
[
𝔼
q
(
y
|
x
i
A
)
[
-
log
p
(
y
|
x
i
B
)
]
]
s
.
t
.
𝔼
i
∼
𝒟
[
q
(
y
|
x
i
A
)
]
=
1
/
K
,
∀
y
.
where Q A is an (N×K) matrix, N is a number of data instances x i A in dataset , K is a number of class labels that y can have, p(y|x i B ) is current estimate of probability for x i B that is paired with x i A of the class label having a value of y; and
wherein the estimate for the second surrogate q(y|x B ) is found using an optimization defined as
Q
i
y
B
:
=
q
(
y
|
x
i
B
)
.
min
Q
B
𝔼
i
∼
𝒟
[
𝔼
q
(
y
|
x
i
B
)
[
-
log
p
(
y
|
x
i
A
)
]
]
s
.
t
.
𝔼
i
∼
𝒟
[
q
(
y
|
x
i
B
)
]
=
1
K
,
∀
y
.
where Q B is an (N×K) matrix, N is a number of data instances x i B in the dataset , K is the number of class labels that y can have, p(y|x i A ) is current estimate of probability of the class label having a value of y for x i A that is paired with x i B .
12. The method of claim 11 , further comprising:
solving optimizations to estimate the first and second surrogates using Sinkhorn-Knopp (SK) algorithm.
13. The method of claim 11 , further comprising:
selecting a batch comprising a plurality of pairs of data instances from the dataset; evaluating the first and second feature extractors for the selected batch;
storing a result of the evaluating in at least one queue; and
performing optimization on the queued result of the evaluating.
14. An apparatus for training a machine learning ML model the apparatus comprising:
a memory storing one or more instructions;
a machine learning model; and
at least one processor configured to execute the one or more instructions stored in the memory to:
obtain a dataset comprising a plurality of pairs of data instances, with each pair comprising a plurality of first data instances having a first modality and a plurality of second data instances having a second modality, wherein the first data instance is an image,
evaluate, using a second feature extractor, at least some of the plurality of second data instances to extract a set of features for each of the plurality of second data instances,
assign a first set of class labels to the plurality of first data instances based on the set of features for the plurality of second data instances,
train a first feature extractor using the assigned first set of class labels,
evaluate, using the first feature extractor, at least some of the plurality of first data instances to extract a set of image features for the plurality of first data instances,
assign a second set of class labels to the plurality of second data instances based on the set of image features, and
train the second feature extractor using the assigned second set of class labels,
wherein, to assign the first set of class labels, the at least one processor is further configured to:
determine clusters of first data instances based on the set of features; and assign each pair of data instances in a cluster same class label, and wherein, to assign the second set of class labels, the at least one processor is further configured to:
determine clusters of second data instances based on the set of image features; and assign each pair of data instances in a cluster same class label.
15. The apparatus of claim 14 , wherein the at least one processor is further configured to:
define a first linear classifier for the first modality, a second linear classifier for the second modality and a set of training parameters which are shared between the first modality and the second modality.
16. The apparatus of claim 15 , wherein the first and second linear classifiers are denoted by p(y|x A ) and p(y|x B ) and are defined as,
p
(
y
=
j
|
x
M
)
=
exp
(
p
j
T
ϕ
M
(
x
)
/
τ
)
∑
l
exp
(
p
l
T
ϕ
M
(
x
)
/
τ
)
,
M
∈
{
A
,
B
}
where P={p 1 , . . . , p K } are trainable parameters that are shared between the first modality and the second modality, ϕ M represents the first and second feature extractors, x M represents the first and second feature data instances, y is the class label for class j, each p j may be regarded as a prototype vector for class j that lies in a shared feature space, and τ is temperature in softmax.