Systems and methods for virtual and augmented reality
The description relates the feature matching. Our approach establishes pointwise correspondences between challenging image pairs. It takes off-the-shelf local features as input and uses an attentional graph neural network to solve an assignment optimization problem. The deep middle-end matcher acts as a middle-end and handles partial point visibility and occlusion elegantly, producing a partial assignment matrix.
1. A computer system comprising:
a computer-readable medium;
a processor connected to the computer-readable medium; and
a set of instructions on the computer-readable medium, including:
a deep middle-end matcher architecture that includes:
an attentional graph neural network having:
a keypoint encoder to map keypoint positions p and their visual descriptors d into a single vector; and
alternating self-attention and cross-attention layers that, based on the vector, repeated L times to create representations f; and
an optimal matching layer that creates an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix,
where:
p are keypoint positions,
d are descriptors,
L are a plurality of times,
f are representations,
M by N is a score matrix with a length M and a width N.
2. The computer system of claim 1 , wherein, in the keypoint encoder, an initial representation (0) x i for each keypoint combines visual appearance and location, with the respective keypoint position embedded into a high-dimensional vector with a Multilayer Perceptron as follows:
(0) x i =d i +MLP( p i ).
where:
i are keypoints,
(0) x i is an initial representation for the keypoints,
d i are descriptors for the keypoints,
p i are keypoint positions for the keypoints, and
MLP is a Multilayer Perceptron for the keypoint positions.
3. The computer system of claim 2 , wherein the keypoint encoder allows the attentional graph neural network to reason about appearance and position jointly.
4. The computer system of claim 1 , wherein, in the keypoint encoder includes a multiplex graph neural network having a single complete graph with nodes that are the keypoints of two images.
5. The computer system of claim 4 , wherein the graph is a multiplex graph that has two types of undirected edges, namely intra-image edges E self that connect keypoints i to all other keypoints within the same image and inter-image edges E cross that connect keypoints i to all keypoints in the other image and uses a message passing formulation to propagate information along both types of edges, such that the resulting multiplex graph neural network starts with a high-dimensional state for each node and computes at each layer an updated representation by simultaneously aggregating messages across all given edges for all nodes
where:
E self are self edges, and
E cross are cross edges.
6. The computer system of claim 5 , wherein if (l) x A i is the intermediate representation for keypoint i in image A at layer l, the message m E→i is the result of the aggregation from all keypoints {j:(i, j)∈E}, where E∈{E self , E cross }, and a residual message passing update for all i in A is:
(l+1) x i A = (l) x i A +MLP([ (l) x i A ∥m ε→i ]),
where:
[.∥.] denotes concatenation,
i are keypoints,
E self are self edges,
E cross are cross edges,
m E is a message, and
MLP is a Multilayer Perceptron for the keypoint positions.
7. The computer system of claim 6 , wherein a fixed number of layers with different parameters are chained and alternatively aggregate along the self and cross edges such that, starting from l=1, E=E self if l is odd and E=E cross if l is even.
8. The computer system of claim 6 , wherein the alternating self- and cross-attention layers are computed with an attention mechanism computes the message m E→i and performs the aggregation, wherein the self edges are based on self-attention and the cross edges are based on cross-attention, wherein, for a representation of i, a query q i , retrieves values v j of some elements based on their attributes, the keys k j , and the message is computed as weighted average of the values:
m
ℰ
→
i
=
Σ
j
:
(
i
,
j
)
∈
ℰ
α
ij
v
j
,
where:
i are keypoints,
m E is a message,
q i are queries for the keypoints,
v i are retrieves values for the keypoints,
k i are keys for the keypoints,
j is a step variable in a summation formula,
α ij is a Softmax function, and
MLP is a Multilayer Perceptron for the keypoint positions.
9. The computer system of claim 8 , wherein an attention mask α ij is the Softmax over the key-query similarities:
α ij =Softmax j ( q i T k j ).
10. The computer system of claim 1 , wherein final matching descriptors of the alternating self- and cross-attention layers are linear projections:
f
i
A
=
W
(
L
)
x
i
A
+
b
∀
i
∈
A
,
where:
W represents a slope of the linear projection,
B represents a constant in the linear projection, and
A is an image.
11. The computer system of claim 1 , wherein the optimal matching layer expresses a pairwise score for a set as the similarity of matching descriptors:
S
i
,
j
=
<
f
i
A
,
f
j
B
>
∀
(
i
,
j
)
∈
A
×
B
,
where:
S i,j is a pairwise score,
<., .> is the inner product,
A is an image, and
B is an image.
12. The computer system of claim 11 , wherein the optimal matching layer, for occlusion and visibility suppresses occluded keypoints and augments each set of keypoints with a dustbin score so that unmatched keypoints are explicitly assigned to dustbin scores.
13. The computer system of claim 12 , wherein the score S is augmented to S by appending a new row and column, the point-to-bin and bin-to-bin scores, filled with a single learnable parameter:
S i,N+1 = S M+1,j = S M+1,N+1 =α∈R.
where:
N+1 and M+1 represent augmentations.
14. The computer system of claim 12 , wherein the optimal matching layer finds the optimal partial assignment based on the M by N score matrix using the Sinkhorn algorithm for T iterations.
15. The computer system of claim 14 , wherein after T iterations, the optimal matching layer drops the dustbin scores and recovers P=P −1:M,1:N , where
P 1 N+1 =a and P T 1 M ≤1 N .
is the original assignment and
P 1 N+1 =a and P T 1 M+1 =b.
is the assignment with the dustbin scores augmented.
16. A computer-implemented method system that includes:
mapping, with a keypoint encoder of an attentional graph neural network of a deep middle-end matcher architecture, keypoint positions p and their visual descriptors d into a single vector; and
executing, with alternating self- and cross-attention layers of an attentional graph neural network of the deep middle-end matcher architecture, based on the vector, for L repeated times, to create representations f; and
executing an optimal matching layer, of the attentional graph neural network of the deep middle-end matcher architecture, to create an M by N score matrix from the representations f and finding an optimal partial assignment based on the M by N score matrix,
where:
p are keypoint positions,
d are descriptors,
L are a plurality of times,
f are representations,
M by N is a score matrix with a length M and a width N.