System for three-dimensional geometric guided student-teacher feature matching (3DG-STFM)
A system for three-dimensional geometric guided student-teacher feature matching includes a multi-modal teacher model configured to determine feature matching between a pair of RGB-D images, each RGB-D image being a combination of a RGB image and its corresponding depth image; a mono-modal student model configured to determine feature matching from the pair of RGB images and the teacher model, the teacher model guiding the student model to learn RGB-induced depth information for the feature matching on both coarse and fine levels; a coarse-level knowledge transfer loss function for determining loss of transferring coarse-level matching knowledge from the teacher model to the student model; and a fine-level knowledge transfer loss function for determining loss of transferring fine-level matching knowledge from the teacher model to the student model, wherein the fine-level knowledge transfer loss function guides the student model to learn a fine-level prediction distribution with priority.
1. A system for three-dimensional geometric guided student-teacher feature matching (3DG-STFM), comprising:
a multi-modal teacher model configured to determine feature matching between a pair of RGB-D images; wherein each RGB-D image is a combination of a RGB image and its corresponding depth image, wherein the depth image is an image channel in which each pixel relates to a distance between an image plane and a corresponding object in the RGB image;
a mono-modal student model configured to determine feature matching from the pair of RGB images and the teacher model, wherein the teacher model guides the student model to learn RGB-induced depth information for the feature matching on both coarse and fine levels;
a coarse-level knowledge transfer loss function for determining loss of transferring coarse-level matching knowledge from the teacher model to the student model; and
a fine-level knowledge transfer loss function for determining loss of transferring fine-level matching knowledge from the teacher model to the student model, wherein the fine-level knowledge transfer loss function guides the student model to learn a fine-level prediction distribution with priority,
wherein each of the teacher model and the student model comprises:
a feature pyramid network (FPN) configured to extract coarse-level local features with ⅛ of an original image resolution and fine-level features with ½ of the original image resolution from the pair of images; and
a coarse-level local feature transform module configured to
flatten maps of the extracted coarse-level features to one-dimensional vectors;
perform positional encoding on the flattened one-dimensional vectors to assign each feature entry a unique embedding vector in a sinusoidal format to encode spatial information; and
process the encoded local feature vectors by a coarse-level matching transformer comprising self-attention and cross-attention layers to generate a logits layer comprising a correlation matrix.
2. The system of claim 1 , wherein the FPN of the teacher model is designed for 4-channel inputs of RGB-D images, and the FPN of the student model is designed for 3-channel inputs of RGB images, wherein D is depth information.
3. The system of claim 1 , wherein each of the teacher model and the student model further comprises a coarse-level matching module, and a fine-level matching module.
4. The system of claim 1 , wherein the self-attention and cross-attention layers are interleaved in the coarse-level matching transformer by L c times.
5. The system of claim 3 , wherein the coarse-level matching module is configured to
generate a confidence score matrix P c by applying a dual softmax operation to the correlation matrix S c on both row and column directions:
P c ( i,j )=softmax( S c ( i ,·) j )*softmax( S c (·, j ) i )
wherein P c (i, j) is a matching probability at matching position (i, j); and
select matching pairs with probability scores higher than a threshold θ c to yield coarse-level matching predictions.
6. The system of claim 5 , wherein given the ground-truth matrix derived from correspondence labels, cross-entropy loss that measures the performance of the coarse-level local feature transform module and the coarse-level matching module is calculated as:
ℒ
c
=
-
FL
(
P
c
(
i
,
j
)
)
log
(
P
c
)
FL
(
p
)
=
α
(
1
-
p
^
)
γ
,
p
ˆ
=
{
p
1
-
p
if
y
=
1
,
otherwise
wherein gt is a correspondence set generated by the ground-truth labels, FL with a predicted probability p is a focal loss term for addressing imbalance between matching and unmatching pairs.
7. The system of claim 5 , wherein the fine-level matching module is configured to
project the coarse matching position (i, j) on the fine-level feature map at position (î,ĵ) and extract its features from a neighboring w×w size window as matching candidates;
upsample the selected coarse-level features and concatenate the upsampled coarse-level features to the fine-level features; and
process the concatenated fine-level features by a fine-level matching transformer to predict correspondences on subpixel-level.
8. The system of claim 7 , wherein the fine-level matching transformer contains L f self-attention and cross-attention layers to aggregate fine-level local information and generate a heatmap distribution for the location refinement.
9. The system of claim 7 , wherein the final loss based on the direct supervision is calculated by:
ℒ
f
=
1
σ
2
(
ι
^
)
μ
(
ι
^
)
-
J
^
gt
2
2
where (î,ĵ) is the ground-truth position wrapped from an image solution to a fine-level heatmap scale, μ(î) is the prediction associated to a coarse position î, σ 2 (î) is the total variance of the heatmap distribution, and f is the set of fine matches predicted by the module.
10. The system of claim 9 , wherein the total variance of the similarity distribution is treated as uncertainty to assign a weight to each fine-level match, and wherein the larger total variance indicates it is an uncertain prediction and associates with low weights.
11. The system of claim 5 , wherein the coarse-level knowledge transfer loss function is adapted to decompose the logits layer to multiple independent query distributions and guides the student model to learn the distributions.
12. The system of claim 11 , wherein the coarse-level knowledge transfer loss function determines a mutual query divergence (MQD) loss that employs all the mutual query distributions for knowledge transfer:
ℒ
MQD
=
-
1
n
[
-
∑
i
=
1
n
FL
(
p
S
(
i
)
)
p
ˆ
S
(
i
)
log
(
p
ˆ
T
(
i
)
)
]
p
ˆ
S
(
ι
)
=
exp
(
o
S
k
T
)
∑
k
=
1
L
exp
(
o
S
k
T
)
p
ˆ
T
(
ι
)
=
exp
(
o
T
k
T
)
∑
k
=
1
L
exp
(
o
T
k
T
)
p
S
(
ι
)
=
exp
(
o
S
k
)
∑
k
=
1
L
exp
(
o
S
k
)
where {circumflex over (p)} S (i) and {circumflex over (p)} T (i) are respectively student's and teacher's query distributions distilled at temperature T, additional focal loss weight FL is added to balance the matching/unmatching ground-truth pairs, p S (i) is the standard confidence score predicted by the student model, and the total MQD loss MQD on the coarse level is the mean of knowledge distillation (KD) loss of all n distributions, wherein n is equal to 2×hw.
13. The system of claim 7 , wherein the fine-level knowledge transfer loss function is adapted to assign weights to the prediction of the teacher model, with the prediction with higher certainty is assigned with larger weights.
14. The system of claim 13 , wherein the heatmaps of the teacher model and the student model are of Gaussian distributions N t (μ t , σ t 2 ) and N s (μ s ,σ s 2 ), respectively.
15. The system of claim 14 , wherein the fine-level knowledge transfer loss function determines an attentive loss ( att ) that is applied to help the student model learn the Gaussian distributions:
ℒ
att
=
1
2
σ
t
(
ι
^
)
2
(
μ
S
(
ι
^
)
-
μ
t
(
ι
^
)
)
2
where the μ s (î) and μ t (î) are the expectations of student's and teacher's output distributions which correspond to match (î,ĵ) in the fine-level correspondence set f .
16. The system of claim 15 , wherein the attentive loss is treated as a l 2 distance loss that pays more attention to the prediction associated with large attention weight
1
2
σ
t
(
ι
^
)
2
.
17. The system of claim 15 , wherein the teacher prediction with a small total variance indicates the teacher model is quite certain about the location of the correspondence, wherein the loss is assigned with a large weight to guide the student model to learn the certain predictions from the teacher model in priority.
18. The system of claim 15 , wherein the final losses for the teacher model and the student model are respectively:
teacher =λ 0 c +λ 1 f ; and
student =λ 0 c +λ 1 f +λ 2 MQD +λ 3 att .