Systems and methods for virtual and augmented reality
The description relates the feature matching. Our approach establishes pointwise correspondences between challenging image pairs. It takes off-the-shelf local features as input and uses an attentional graph neural network to solve an assignment optimization problem. The deep middle-end matcher acts as a middle-end and handles partial point visibility and occlusion elegantly, producing a partial assignment matrix.
1 . A computer system comprising:
a computer-readable medium;
a processor connected to the computer-readable medium; and
a set of instructions on the computer-readable medium, including:
a deep middle-end matcher architecture that is executable by the processor and includes:
an attentional graph neural network that has:
a keypoint encoder to map keypoint positions p and their visual descriptors d into a single vector; and
at least one layer that, based on the vector, creates representations f, wherein the at least one layer includes alternating self-attention and cross-attention layers that, based on the vector, repeated L times to create representations f; and
an optimal matching layer that creates an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix, where:
p are keypoint positions,
f are representations,
d are descriptors,
L are a plurality of times,
M by N is a score matrix with a length M and a width N.
2 . The computer system of claim 1 , wherein, in the keypoint encoder, an initial representation (0) x i for each keypoint combines visual appearance and location, with the respective keypoint position embedded into a high-dimensional vector with a Multilayer Perceptron as follows:
(
0
)
x
i
=
d
i
+
MLP
(
p
i
)
.
where:
i are keypoints,
(0) x i is an initial representation for the keypoints,
d i are descriptors for the keypoints,
p i are keypoint positions for the keypoints, and
MLP is a Multilayer Perceptron for the keypoint positions.
3 . The computer system of claim 2 , wherein the keypoint encoder allows the attentional graph neural network to reason about appearance and position jointly.
4 . The computer system of claim 1 , wherein, in the keypoint encoder includes a multiplex graph neural network having a single complete graph with nodes that are the keypoints of two images.
5 . The computer system of claim 4 , wherein the graph is a multiplex graph that has two types of undirected edges, namely intra-image edges that connect keypoints i to all other keypoints within the same image and inter-image edges that connect keypoints i to all keypoints in the other image and uses a message passing formulation to propagate information along both types of edges, such that the resulting multiplex graph neural network starts with a high-dimensional state for each node and computes at each layer an updated representation by simultaneously aggregating messages across all given edges for all nodes
where:
E self are self edges, and
E cross are cross edges.
6 . The computer system of claim 5 , wherein if (l) x A i is the intermediate representation for keypoint i in image A at layer 1, the message m E→i is the result of the aggregation from all keypoints {j:(i, j)∈E}, where E∈{E self , E cross }, and a residual message passing update for all i in A is:
(
ℓ
+
1
)
x
i
A
=
(
ℓ
)
x
i
A
+
MLP
(
[
(
ℓ
)
x
i
A
❘
"\[LeftBracketingBar]"
❘
"\[RightBracketingBar]"
m
ε
→
i
]
)
,
where:
[·∥·] denotes concatenation,
i are keypoints,
E self are self edges,
E cross are cross edges,
m E is a message, and
MLP is a Multilayer Perceptron for the keypoint positions.
7 . The computer system of claim 6 , wherein a fixed number of layers with different parameters are chained and alternatively aggregate along the self and cross edges such that, starting from l=1, E=E self if l is odd and E=E cross if l is even.
8 . The computer system of claim 6 , wherein the alternating self- and cross-attention layers are computed with an attention mechanism computes the message m E→i and performs the aggregation, wherein the self edges are based on self-attention and the cross edges are based on cross-attention, wherein, for a representation of i, a query q i , retrieves values v j of some elements based on their attributes, the keys k j , and the message is computed as weighted average of the values:
m
ε
→
i
=
∑
j
:
(
i
,
j
)
∈
ε
α
ij
v
j
,
where:
i are keypoints,
m E is a message,
q i are queries for the keypoints,
v i are retrieves values for the keypoints,
k i are keys for the keypoints,
j is a step variable in a summation formula,
α ij is a Softmax function, and
MLP is a Multilayer Perceptron for the keypoint positions.
9 . The computer system of claim 8 , wherein an attention mask α ij is the Softmax over the key-query similarities:
α
ij
=
Softmax
j
(
q
i
⊤
k
j
)
.
10 . The computer system of claim 1 , wherein final matching descriptors of the alternating self-and cross-attention layers are linear projections:
f
i
A
=
W
(
L
)
x
i
A
+
b
∀
i
∈
𝒜
,
where:
W represents a slope of the linear projection,
B represents a constant in the linear projection, and
A is an image.
11 . The computer system of claim 1 , wherein the optimal matching layer expresses a pairwise score for a set as the similarity of matching descriptors:
S
i
,
j
=
<
f
i
A
,
f
j
B
>
∀
(
i
,
j
)
∈
𝒜
×
ℬ
,
where:
S i,j is a pairwise score,
<·, ·> is the inner product,
A is an image, and
B is an image.
12 . The computer system of claim 11 , wherein the optimal matching layer, for occlusion and visibility suppresses occluded keypoints and augments each set of keypoints with a dustbin score so that unmatched keypoints are explicitly assigned to dustbin scores.
13 . The computer system of claim 12 , wherein the score S is augmented to S by appending a new row and column, the point-to-bin and bin-to-bin scores, filled with a single learnable parameter:
S
_
i
,
N
+
1
=
S
_
M
+
1
,
j
=
S
_
M
+
1
,
N
+
1
=
α
∈
ℝ
.
where:
N+1 and M+1 represent augmentations.
14 . The computer system of claim 12 , wherein the optimal matching layer finds the optimal partial assignment based on the M by N score matrix using the Sinkhorn algorithm for T iterations.
15 . The computer system of claim 14 , wherein after T iterations, the optimal matching layer drops the dustbin scores and recovers P= P 1:M,1:N , where
P
1
N
≤
1
M
and
P
⊤
1
M
≤
1
N
.
is the original assignment and
P
_
1
N
+
1
=
a
and
P
_
⊤
1
M
+
1
=
b
.
is the assignment with the dustbin scores augmented.
16 . A computer-implemented method that includes:
creating, with an attentional graph neural network, a representation by mapping, with a keypoint encoder of the attentional graph neural network keypoint positions p and their visual descriptors d into a single vector;
creating, with an optimal matching layer, a score from the representation by executing, with at least one layer of the attentional graph neural network, based on the vector, representations f, including executing with alternating self- and cross-attention layers of the attentional graph neural network, based on the vector, for L repeated times, to create representations f; and
finding, with the optimal matching layer, an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix,
where:
p are keypoint positions,
f are representations,
L are a plurality of times,
M by N is a score matrix with a length M and a width N.