Granted Patent B2

US 12,639,579 · App. 18/674,016 · Granted May 26, 2026

Inventors: Paul-Edouard Sarlin (Zurich, CH); Daniel DeTone (San Francisco, CA); Tomasz Jan Malisiewicz (Somerville, MA); Andrew Rabinovich (San Francisco, CA)

Assignee: Magic Leap, Inc.

G06N3/084G06F18/22G06N3/04G06N3/08G06V10/757G06V10/82G06V20/36G06V30/1988

Abstract

Claims (283)

1 . A computer system comprising:

a computer-readable medium;

a processor connected to the computer-readable medium; and

a set of instructions on the computer-readable medium, including:

a deep middle-end matcher architecture that is executable by the processor and includes:

an attentional graph neural network that has:

a keypoint encoder to map keypoint positions p and their visual descriptors d into a single vector; and

at least one layer that, based on the vector, creates representations f, wherein the at least one layer includes alternating self-attention and cross-attention layers that, based on the vector, repeated L times to create representations f; and

an optimal matching layer that creates an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix, where:

p are keypoint positions,

f are representations,

d are descriptors,

L are a plurality of times,

M by N is a score matrix with a length M and a width N.

2 . The computer system of claim 1 , wherein, in the keypoint encoder, an initial representation (0) x i for each keypoint combines visual appearance and location, with the respective keypoint position embedded into a high-dimensional vector with a Multilayer Perceptron as follows:

(

0

)

x

i

=

d

i

+

MLP

⁡

(

p

i

)

.

where:

i are keypoints,

(0) x i is an initial representation for the keypoints,

d i are descriptors for the keypoints,

p i are keypoint positions for the keypoints, and

MLP is a Multilayer Perceptron for the keypoint positions.

3 . The computer system of claim 2 , wherein the keypoint encoder allows the attentional graph neural network to reason about appearance and position jointly.

4 . The computer system of claim 1 , wherein, in the keypoint encoder includes a multiplex graph neural network having a single complete graph with nodes that are the keypoints of two images.

5 . The computer system of claim 4 , wherein the graph is a multiplex graph that has two types of undirected edges, namely intra-image edges that connect keypoints i to all other keypoints within the same image and inter-image edges that connect keypoints i to all keypoints in the other image and uses a message passing formulation to propagate information along both types of edges, such that the resulting multiplex graph neural network starts with a high-dimensional state for each node and computes at each layer an updated representation by simultaneously aggregating messages across all given edges for all nodes

where:

E self are self edges, and

E cross are cross edges.

6 . The computer system of claim 5 , wherein if (l) x A i is the intermediate representation for keypoint i in image A at layer 1, the message m E→i is the result of the aggregation from all keypoints {j:(i, j)∈E}, where E∈{E self , E cross }, and a residual message passing update for all i in A is:

(

ℓ

+

1

)

x

i

A

=

(

ℓ

)

x

i

A

+

MLP

⁡

(

[

(

ℓ

)

x

i

A

⁢

❘

"\[LeftBracketingBar]"

❘

"\[RightBracketingBar]"

⁢

m

ε

→

i

]

)

,

where:

[·∥·] denotes concatenation,

i are keypoints,

E self are self edges,

E cross are cross edges,

m E is a message, and

MLP is a Multilayer Perceptron for the keypoint positions.

7 . The computer system of claim 6 , wherein a fixed number of layers with different parameters are chained and alternatively aggregate along the self and cross edges such that, starting from l=1, E=E self if l is odd and E=E cross if l is even.

8 . The computer system of claim 6 , wherein the alternating self- and cross-attention layers are computed with an attention mechanism computes the message m E→i and performs the aggregation, wherein the self edges are based on self-attention and the cross edges are based on cross-attention, wherein, for a representation of i, a query q i , retrieves values v j of some elements based on their attributes, the keys k j , and the message is computed as weighted average of the values:

m

ε

→

i

=

∑

j

:

(

i

,

j

)

∈

ε

α

ij

⁢

v

j

,

where:

i are keypoints,

m E is a message,

q i are queries for the keypoints,

v i are retrieves values for the keypoints,

k i are keys for the keypoints,

j is a step variable in a summation formula,

α ij is a Softmax function, and

MLP is a Multilayer Perceptron for the keypoint positions.

9 . The computer system of claim 8 , wherein an attention mask α ij is the Softmax over the key-query similarities:

α

ij

=

Softmax

j

⁢

(

q

i

⊤

⁢

k

j

)

.

10 . The computer system of claim 1 , wherein final matching descriptors of the alternating self-and cross-attention layers are linear projections:

f

i

A

=

W

⁢

(

L

)

x

i

A

+

b

⁢

∀

i

∈

𝒜

,

where:

W represents a slope of the linear projection,

B represents a constant in the linear projection, and

A is an image.

11 . The computer system of claim 1 , wherein the optimal matching layer expresses a pairwise score for a set as the similarity of matching descriptors:

S

i

,

j

=

<

f

i

A

,

f

j

B

>

∀

(

i

,

j

)

∈

𝒜

×

ℬ

,

where:

S i,j is a pairwise score,

<·, ·> is the inner product,

A is an image, and

B is an image.

12 . The computer system of claim 11 , wherein the optimal matching layer, for occlusion and visibility suppresses occluded keypoints and augments each set of keypoints with a dustbin score so that unmatched keypoints are explicitly assigned to dustbin scores.

13 . The computer system of claim 12 , wherein the score S is augmented to S by appending a new row and column, the point-to-bin and bin-to-bin scores, filled with a single learnable parameter:

S

_

i

,

N

+

1

=

S

_

M

+

1

,

j

=

S

_

M

+

1

,

N

+

1

=

α

∈

ℝ

.

where:

N+1 and M+1 represent augmentations.

14 . The computer system of claim 12 , wherein the optimal matching layer finds the optimal partial assignment based on the M by N score matrix using the Sinkhorn algorithm for T iterations.

15 . The computer system of claim 14 , wherein after T iterations, the optimal matching layer drops the dustbin scores and recovers P= P 1:M,1:N , where

P

⁢

1

N

≤

1

M

⁢

and

⁢

P

⊤

⁢

1

M

≤

1

N

.

is the original assignment and

P

_

⁢

1

N

+

1

=

a

⁢

and

⁢

P

_

⊤

⁢

1

M

+

1

=

b

.

is the assignment with the dustbin scores augmented.

16 . A computer-implemented method that includes:

creating, with an attentional graph neural network, a representation by mapping, with a keypoint encoder of the attentional graph neural network keypoint positions p and their visual descriptors d into a single vector;

creating, with an optimal matching layer, a score from the representation by executing, with at least one layer of the attentional graph neural network, based on the vector, representations f, including executing with alternating self- and cross-attention layers of the attentional graph neural network, based on the vector, for L repeated times, to create representations f; and

finding, with the optimal matching layer, an M by N score matrix from the representations f and finds an optimal partial assignment based on the M by N score matrix,

where:

p are keypoint positions,

f are representations,

L are a plurality of times,

M by N is a score matrix with a length M and a width N.

Assignments (2)

Continuity (3)

References Cited (400)