SYSTEMS AND METHODS FOR IMAGE RETRIEVAL USING SUPER FEATURES
A training system includes: an iterative attention module configured to, based on first features in input images, determine ordered sets of second features using iterative attention over T iterations, where T is an integer greater than or equal to two; and a training module configured to: selectively input pairs of matching images to the iterative attention module; selectively input non-matching images to the iterative attention module; and based on the ordered sets generated by the iterative attention module based on the input pairs of matching images and the input non-matching images, train the iterative attention module based on minimizing at least one of: a contrastive loss; and a cosine similarity loss.
1 . A training system, comprising:
an iterative attention module configured to, based on first features in input images, determine ordered sets of second features using iterative attention over T iterations,
wherein T is an integer greater than or equal to two; and
a training module configured to:
selectively input pairs of matching images to the iterative attention module;
selectively input non-matching images to the iterative attention module; and
based on the ordered sets generated by the iterative attention module based on the input pairs of matching images and the input non-matching images, train the iterative attention module based on minimizing at least one of:
a contrastive loss; and
a cosine similarity loss.
2 . The training system of claim 1 wherein the training module is configured to train the iterative attention module based on minimizing both of the contrastive loss and the cosine similarity loss.
3 . The training system of claim 1 wherein the training module is configured to determine whether a pair of images is matching or not matching based on an annotation included with the pair.
4 . The training system of claim 1 wherein the training module is configured to determine matching first features in an input pair of matching images.
5 . The training system of claim 4 wherein the training module is configured to determining matching features using nearest neighbor matching.
6 . The training system of claim 4 wherein the training module is configured to train the iterative attention module using only matching features that satisfy predetermined criteria.
7 . The training system of claim 6 wherein the predetermined criteria include the matching features being reciprocal nearest neighbors.
8 . The training system of claim 6 wherein the predetermined criteria include the matching features passing the Lowe first to second neighbor ratio test.
9 . The training system of claim 6 wherein the predetermined criteria include the matching features having a same feature identifier.
10 . The training system of claim 4 wherein the training module is configured to determine the contrastive loss using the equation:
ℒ
super
=
∑
(
s
,
s
+
∈
P
)
[
s
-
s
+
2
+
∑
n
∈
N
(
i
(
s
)
)
[
u
′
-
s
-
n
2
2
]
+
]
,
where pair p=(s, s + )∈P a pair of first features extracted from a pair of matching images x, x + , u′ is a predetermined hyper parameter, and the nonmatches for each s are given by features from all non-matching images with the same first feature identifier as s.
11 . The training system of claim 4 wherein the training module is configured to determine the cosine similarity loss based on attention maps of the first features of each image.
12 . The training system of claim 11 wherein the training module is configured to determine the cosine similarity loss using the equation:
ℒ
attn
(
x
)
=
1
N
(
N
-
1
)
∑
i
≠
j
α
i
T
·
α
j
α
j
2
α
j
2
,
i
,
j
∈
{
1
,
…
,
N
}
,
where α=[α j ], α j ∈ L , j=1, . . . , N are the attention maps after a last iteration of the iterative attention module.
13 . The training system of claim 12 wherein the training module is configured to determine the cosine similarity loss based on off-diagonal elements of an N×N self-correlation matrix α T α of the attention maps.
14 . A training method, comprising:
training an iterative attention module configured to, based on first features in input images, determine ordered sets of second features using iterative attention over T iterations,
wherein T is an integer greater than or equal to two, and
wherein the training includes:
selectively inputting pairs of matching images to the iterative attention module;
selectively inputting non-matching images to the iterative attention module; and
based on the ordered sets generated by the iterative attention module based on the input pairs of matching images and the input non-matching images, training the iterative attention module based on minimizing at least one of:
a contrastive loss; and
a cosine similarity loss.
15 . The training method of claim 14 wherein the training includes training the iterative attention module based on minimizing both of the contrastive loss and the cosine similarity loss.
16 . The training method of claim 14 wherein the training includes determining whether a pair of images is matching or not matching based on an annotation included with the pair.
17 . The training method of claim 14 wherein the training includes determining matching first features in an input pair of matching images.
18 . The training method of claim 17 wherein the training includes determining matching features using nearest neighbor matching.
19 . The training method of claim 17 wherein the training includes training the iterative attention module using only matching features that satisfy predetermined criteria.
20 . The training method of claim 19 wherein the predetermined criteria include the matching features being reciprocal nearest neighbors.