IP Library › Granted Patent US 12,639,556
Granted Patent B2
US 12,639,556 · App. 16/927,018 · Granted May 26, 2026

Object-centric learning with slot attention

Inventors: Dirk Weissenborn (Berlin, DE); Jakob Uszkoreit (Berlin, DE); Thomas Unterthiner (Berlin, DE); Aravindh Mahendran (Berlin, DE); Francesco Locatello (Hoofddorp, NL); Thomas Kipf (Amsterdam, NL); Georg Heigold (Aachen, DE); Alexey Dosovitskiy (Berlin, DE)
Assignee: Google LLC
G06N3/063G06F17/16G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,639,556
App. No.
16/927,018
Granted
May 26, 2026
Kind
B2
Abstract

A method involves receiving a perceptual representation including a plurality of feature vectors, and initializing a plurality of slot vectors represented by a neural network memory unit. Each respective slot vector is configured to represent a corresponding entity in the perceptual representation. The method also involves determining an attention matrix based on a product of the plurality of feature vectors transformed by a key function and the plurality of slot vectors transformed by a query function. Each respective value of a plurality of values along each respective dimension of the attention matrix is normalized with respect to the plurality of values. The method additionally involves determining an update matrix based on the plurality of feature vectors transformed by a value function and the attention matrix, and updating the plurality of slot vectors based on the update matrix by way of the neural network memory unit.

Claims (56)

1 . A computer-implemented method for generating entity-centric representations for performing downstream tasks, the method comprising:

providing, to a machine learning (ML) model executed by a processor, a plurality of feature vectors comprising N feature vectors that represent features of an input data;

initializing, by the processor, a plurality of slot vectors represented by a neural network memory unit of the ML model and comprising K slot vectors, wherein initializing the plurality of slot vectors comprises determining initial values for the plurality of slot vectors;

determining updated values of the plurality of slot vectors by executing, using the processor, a plurality of iterations of the ML model in which the ML model processes the plurality of feature vectors and the plurality of slot vectors, wherein the plurality of iterations cause each respective slot vector of the plurality of slot vectors to converge to representing a corresponding entity contained in the input data by replacing the initial values with the updated values during the plurality of iterations, and wherein each respective iteration of the plurality of iterations comprises:

determining, using a slot attention calculator of the ML model, an attention matrix based on a product of (i) the plurality of feature vectors transformed by a key function of the ML model and (ii) the plurality of slot vectors transformed by a query function of the ML model, wherein the attention matrix comprises N vectors corresponding to a first dimension of the attention matrix, wherein each respective vector of the N vectors comprises K values corresponding to a second dimension of the attention matrix, and wherein the slot attention calculator normalizes each respective value of the K values of each respective vector of the N vectors of the attention matrix with respect to the K values of the respective vector;

determining, using a slot update calculator of the ML model, an update matrix based on (i) the plurality of feature vectors transformed by a value function of the ML model and (ii) the attention matrix;

updating values of the plurality of slot vectors based on the update matrix and using the neural network memory unit of the ML model, wherein the neural network memory unit provides the updated values of the plurality of slot vectors as input to the query function for a subsequent iteration of the plurality of iterations;

determining, by the ML model executed by the processor, a task-specific output based on the updated values of the plurality of slot vectors; and

performing, using the processor, a downstream task on one or more entities contained in the input data based on the task-specific output.

2 . The computer-implemented method of claim 1 , wherein a respective iteration of the plurality of iterations refines prior values of the respective slot vector towards convergence to representing the corresponding entity contained in the input data, and wherein the prior values of the respective slot vector are determined by a prior iteration that precedes the respective iteration.

3 . The computer-implemented method of claim 1 , wherein each respective slot vector represents a semantic embedding of the corresponding entity, wherein determining the updated values of the plurality of slot vectors (i) iteratively refines the semantic embedding of the corresponding entity and (ii) causes the respective slot vector to converge to representing the corresponding entity independently of a classification of the corresponding entity.

4 . The computer-implemented method of claim 1 , wherein normalizing by the slot attention calculator each respective value of the K values along each respective vector of the N vectors of the attention matrix with respect to the K values of the respective vector comprises using a softmax function to divide (i) an exponent of the respective value of the K values of the respective vector by (ii) a sum of exponents of the K values of the respective vector.

5 . The computer-implemented method of claim 1 , wherein normalizing by the slot attention calculator each respective value of the K values along each respective vector of the N vectors of the attention matrix with respect to the K values of the respective vector causes the plurality of slot vectors to compete with one another for representing entities contained in the input data by causing two or more slot vector of the plurality of slot vectors to attempt to represent a particular entity of the entities.

6 . The computer-implemented method of claim 1 , wherein determining the update matrix comprises:

determining a product of (i) the plurality of feature vectors transformed by the value function and (ii) a transpose of the attention matrix.

7 . The computer-implemented method of claim 1 , wherein determining the update matrix comprises:

determining an attention weight matrix by dividing (i) each respective value of N values of each respective vector of K vectors corresponding to the second dimension of the attention matrix by (ii) a sum of the N values of the respective vector of the K vectors; and

determining a product of (i) the plurality of feature vectors transformed by the value function and (ii) a transpose of the attention weight matrix.

8 . The computer-implemented method of claim 1 , wherein the plurality of feature vectors are represented by an input matrix comprising: (i) N rows corresponding to a number of the plurality of feature vectors and (ii) I columns corresponding to a dimension of each of the plurality of feature vectors, wherein the plurality of slot vectors are represented by a slot matrix comprising: (i) K rows corresponding to a number of the plurality of slot vectors and (ii) S columns corresponding to a dimension of each of the plurality of slot vectors, wherein the key function comprises a linear transformation represented by a key weight matrix comprising I rows and D columns, wherein the query function comprises a linear transformation represented by a query weight matrix comprising S rows and D columns, wherein the value function comprises a linear transformation represented by a value weight matrix comprising I rows and D columns, and wherein one or more of the key weight matrix, the query weight matrix, or the value weight matrix are learned during training.

9 . The computer-implemented method of claim 8 , wherein determining the attention matrix based on the product comprises:

determining a dot product of (i) the plurality of feature vectors transformed by the key function and (ii) a transpose of the plurality of slot vectors transformed by the query function; and

dividing the dot product by a square root of D.

10 . The computer-implemented method of claim 1 , wherein the plurality of slot vectors are permutation equivariant with respect to one another such that, for multiple different initializations of the plurality of slot vectors with respect to a given input data, a set of values of the plurality of slot vectors is approximately constant and an order of the plurality of slot vectors is variable, and wherein the plurality of slot vectors are permutation invariant with respect to the plurality of feature vectors such that, for multiple different permutations of the plurality of feature vector, the set of values of the plurality of slot vectors is approximately constant.

11 . The computer-implemented method of claim 1 , wherein the input data comprises one or more of: image data, depth image data, point cloud data, time series data, audio data, or text data, wherein the input data is processed by way of one or more machine learning models to generate the plurality of feature vectors, and wherein the corresponding entity represented by the respective slot vector comprises one or more of: an object, a surface, a background, a waveform pattern, or one or more words.

12 . The computer-implemented method of claim 1 , wherein each respective feature vector of the plurality of feature vectors comprises a position embedding that indicates a portion of the input data represented by the respective feature vector.

13 . The computer-implemented method of claim 1 , wherein the neural network memory unit comprises at least one of: (i) a gated recurrent unit (GRU) or (ii) a long-short term memory neural network (LSTM), and wherein one or more weights of the neural network memory unit are learned during training.

14 . The computer-implemented method of claim 1 , wherein updating the plurality of slot vectors based on the update matrix by way of the neural network memory unit comprises:

processing the update matrix by way of the neural network memory unit; and

updating the plurality of slot vectors by way of a feed-forward artificial neural network connected to an output of the neural network memory unit.

15 . The computer-implemented method of claim 1 , wherein performing the downstream task comprises:

performing, by one or more machine learning models, at least one of a supervised learning task or an unsupervised learning task based on the task-specific output, wherein the one or more machine learning models are jointly trained with one or more components of the ML model to perform the at least one of the supervised learning task or the unsupervised learning task.

16 . The computer-implemented method of claim 1 , wherein initializing the plurality of slot vectors comprises:

determining the initial values of the plurality of slot vectors based on one or more of: (i) values selected from a normal distribution or (ii) values of one or more preceding slot vectors determined for a preceding input data processed before the input data, wherein initializing the plurality of slot vectors based on the values of the one or more preceding slot vectors causes the plurality of slot vectors to track entities across a sequence of input data.

17 . The computer-implemented method of claim 1 , wherein, when a number of slot vectors in the plurality of slot vectors exceeds a number of entities contained in the input data, values of one or more slot vector of the plurality of slot vectors are configured to indicate that the one or more slot vectors are unused, and wherein, when the number of entities contained in the input data exceeds the number of slot vectors in the plurality of slot vectors, at least one slot vector of the plurality of slot vectors is configured to represent multiple corresponding entities contained in the input data.

18 . A system for generating entity-centric representations for performing downstream tasks, the system comprising:

a processor; and

a non-transitory computer-readable storage medium having stored thereon instruction that, when executed by the processor, cause the processor to perform operations comprising:

providing, to a machine learning (ML) model executed by the processor, a plurality of feature vectors comprising N feature vectors that represent features of an input data;

initializing a plurality of slot vectors represented by a neural network memory unit of the ML model and comprising K slot vectors, wherein initializing the plurality of slot vectors comprises determining initial values for the plurality of slot vectors;

determining updated values of the plurality of slot vectors by executing a plurality of iterations of the ML model in which the ML model processes the plurality of feature vectors and the plurality of slot vectors, wherein the plurality of iterations cause each respective slot vector of the plurality of slot vectors to converge to representing a corresponding entity contained in the input data by replacing the initial values with the updated values during the plurality of iterations, and wherein each respective iteration of the plurality of iterations comprises:

determining, using a slot attention calculator of the ML model, an attention matrix based on a product of (i) the plurality of feature vectors transformed by a key function of the ML model and (ii) the plurality of slot vectors transformed by a query function of the ML model, wherein the attention matrix comprises N vectors corresponding to a first dimension of the attention matrix, wherein each respective vector of the N vectors comprises K values corresponding to a second dimension of the attention matrix, and wherein the slot attention calculator is configured to normalize each respective value of the K values of each respective vector of the N vectors of the attention matrix with respect to the K values of the respective vector;

determining, using a slot update calculator of the ML model, an update matrix based on (i) the plurality of feature vectors transformed by a value function of the ML model and (ii) the attention matrix;

updating values of the plurality of slot vectors based on the update matrix and using the neural network memory unit of the ML model, wherein the neural network memory unit is configured to provide the updated values of the plurality of slot vectors as input to the query function for a subsequent iteration of the plurality of iterations;

determining, by the ML model, a task-specific output based on the updated values of the plurality of slot vectors; and

performing a downstream task on one or more entities contained in the input data based on the task-specific output.

19 . A non-transitory computer-readable storage medium having stored thereon instruction for generating entity-centric representations for performing downstream tasks, the instructions, when executed by a computing system, cause the computing system to perform operations comprising:

providing, to a machine learning (ML) model executed by the computing system, a plurality of feature vectors comprising N feature vectors that represent features of an input data;

initializing a plurality of slot vectors represented by a neural network memory unit of the ML model and comprising K slot vectors, wherein initializing the plurality of slot vectors comprises determining initial values for the plurality of slot vectors;

determining updated values of the plurality of slot vectors by executing a plurality of iterations of the ML model in which the ML model processes the plurality of feature vectors and the plurality of slot vectors, wherein the plurality of iterations cause each respective slot vector of the plurality of slot vectors to converge to representing a corresponding entity contained in the input data by replacing the initial values with the updated values during the plurality of iterations, and wherein each respective iteration of the plurality of iterations comprises:

determining, using a slot attention calculator of the ML model, an attention matrix based on a product of (i) the plurality of feature vectors transformed by a key function of the ML model and (ii) the plurality of slot vectors transformed by a query function of the ML model, wherein the attention matrix comprises N vectors corresponding to a first dimension of the attention matrix, wherein each respective vector of the N vectors comprises K values corresponding to a second dimension of the attention matrix, and wherein the slot attention calculator is configured to normalize each respective value of the K values of each respective vector of the N vectors of the attention matrix is normalized with respect to the K values of the respective vector;

determining, using a slot update calculator of the ML model, an update matrix based on (i) the plurality of feature vectors transformed by a value function of the ML model and (ii) the attention matrix;

updating values of the plurality of slot vectors based on the update matrix and using the neural network memory unit of the ML model, wherein the neural network memory unit is configured to provide the updated values of the plurality of slot vectors as input to the query function for a subsequent iteration of the plurality of iterations;

determining, by the ML model, a task-specific output based on the updated values of the plurality of slot vectors; and

performing a downstream task on one or more entities contained in the input data based on the task-specific output.

20 . The computer-implemented method of claim 1 , wherein the task-specific output represents one or more task-specific attributes of the one or more entities contained in the input data, and wherein performing the downstream task comprises:

causing a device to interact with an environment represented by the input data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 14, 2020
From: WEISSENBORN, DIRK; USZKOREIT, JAKOB; UNTERTHINER, THOMAS; MAHENDRAN, ARAVINDH; LOCATELLO, FRANCESCO; KIPF, THOMAS; HEIGOLD, GEORG; DOSOVITSKIY, ALEXEY
To: GOOGLE LLC
Reel/Frame 053201/0495 →
Continuity (2)
Provisional Application 63034003 · Jun 3, 2020
Related Publication 20210383199A1 · Dec 9, 2021
References Cited (92)
US 20180137857A1 · Zhou et al. · 2018 [cited by applicant]
US 20190221312A1 · Al Hasan · 2019 [cited by examiner]
US 20200410337A1 · Huang · 2020 [cited by examiner]
US 20210020296A1 · Sjölund · 2021 [cited by examiner]
US 20210034335A1 · Svyatkovskiy · 2021 [cited by examiner]
US 20210089867A1 · Byeon · 2021 [cited by examiner]
US 20210201044A1 · Herdade · 2021 [cited by examiner]
US 20210303939A1 · Hu · 2021 [cited by examiner]
US 20210326742A1 · Rosset · 2021 [cited by examiner]
US 20220043972A1 · Nishida · 2022 [cited by examiner]
CN 110059323B · 2023 [cited by applicant]
KR 20200060191A · 2020 [cited by applicant]
Achlioptas et al., “Learning Representations and Generative Models for 3D Clouds,” International Conference on Machine Learning, 2018, pp. 40-49. [cited by applicant]
Ba et al., “Layer Normalization, Layer Normalization,” arXiv preprint arXiv:1607.06450, Jul. 21, 2016, 14 pages. [cited by applicant]
Bahdanau et al., “Neural Machine Translation by Jointly Learning to Align and Translate,” Published as a conference paper at ICLR 2015, arXiv preprint arXiv:1409.0473, 2014, pp. 1-15. [cited by applicant]
Battaglia et al., “Interaction Networks for Learning about Objects, Relations and Physics,” Advances in Neural Information Processing Systems, 2016, pp. 4502-4510. [cited by applicant]
Battaglia et al., “Relational inductive biases, deep learning, and graph networks,” arXiv preprint arXiv:1806.01261, Oct. 17, 2018, pp. 1-40. [cited by applicant]
Bauckhage, Christian, “Lecture Notes on Data Science: Soft k-Means Clustering,” B-IT, University of Bonn, 2016, 6 pages. [cited by applicant]
Berner et al., “Dota 2 with Large Scale Deep Reinforcement Learning,” arXiv preprint arXiv:1912.06680, Dec. 13, 2019, pp. 1-66. [cited by applicant]
Burgess et al., “MONet: Unsupervised Scene Decomposition and Representation,” arXiv preprint arXiv:1901.11390, Jan. 22, 2019, pp. 1-22. [cited by applicant]
Carion et al., “End-to-End Object Detection with Transformers,” arXiv preprint arXiv:2005.12872, May 28, 2020, pp. 1-26. [cited by applicant]
Chen et al., “Unsupervised Object Segmentation by Redrawing,” Advances in Neural Information Processing Systems, 2019, pp. 12705-12716. [cited by applicant]
Cho et al., “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation,” arXiv preprint arXiv:1406.1078, Sep. 3, 2014, 15 pages. [cited by applicant]
Cuturi, Marco, “Sinkhorn Distances: Lightspeed Computation of Optimal Transport,” Advances in Neural Information Processing Systems, 2013, pp. 2292-2300. [cited by applicant]
De Cao et al., “MolGAN: An implicit generative model for small molecular graphs,” arXiv preprint arXiv:1805.11973, May 30, 2018, 11 pages. [cited by applicant]
Devin et al., “Deep Object-Centric Representations for Generalizable Robot Learning,” arXiv preprint arXiv:1708.04225, Sep. 26, 2017, 8 pages. [cited by applicant]
Engelcke et al., “Genesis: Generative Scene Inference and Sampling with Object-Centric Latent Representations,” arXiv preprint arXiv:1907.13052, Published as a conference paper at ICLR 2020, Feb. 3, 2020, pp. 1-17. [cited by applicant]
Eslami et al., “Attend, Infer, Repeat: Fast Scene Understanding with Generative Models,” Advances in Neural Information Processing Systems, 2016, pp. 3225-3233. [cited by applicant]
Everingham et al., “The Pascal Visual Object Classes Challenge: A Retrospective,” International Journal of Computer Vision, 2015, pp. 98-136, vol. 111, No. 1. [cited by applicant]
Fan et al., “A Point Set Generation Network for 3D Object Reconstruction from a Single Image,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 605-613. [cited by applicant]
Goodfellow et al., “Generative Adversarial Nets,” Advances in Neural Information Processing Systems, 2014, pp. 2672-2680. [cited by applicant]
Goyal et al., “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour,” arXiv preprint arXiv:1706.02677, Apr. 30, 2018, pp. 1-12. [cited by applicant]
Goyal et al., “Recurrent Independent Mechanisms,” arXiv preprint arXiv:1909.10893, Jul. 2, 2020, pp. 1-33. [cited by applicant]
Greff et al., “Multi-Object Representation Learning with Iterative Variational Inference,” arXiv preprint arXiv:1903.00450, May 15, 2019, 27 pages. [cited by applicant]
Greff et al., “Neural Expectation Maximization,” Advances in Neural Information Processing Systems, 2017, pp. 6692-6701. [cited by applicant]
Greff et al., “Tagger: Deep Unsupervised Perceptual Grouping,” Advances in Neural Information Processing Systems, 2016, pp. 4484-4492. [cited by applicant]
Gregor et al., “DRAW: A Recurrent Neural Network for Large Generation,” arXiv preprint arXiv:1502.04623, May 20, 2015, 10 pages. [cited by applicant]
He et al., “Deep Residual Learning for Image Recognition,” Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778. [cited by applicant]
Hinton et al. “Matrix Capsules with EM Routing,” International Conference on Learning Representations, 2018, pp. 1-15. [cited by applicant]
Huang et al., “Better Set Representations for Relational Reasoning,” arXiv preprint arXiv:2003.04448, Jun. 17, 2020, 21 pages. [cited by applicant]
Hubert et al., “Comparing Partitions,” Journal of Classification, 1985, pp. 193-218, vol. 2. [cited by applicant]
Jacobs et al., “Adaptive Mixtures of Local Experts,” Neural Computation, 1991, pp. 79-87, vol. 3. [cited by applicant]
Johnson et al., “CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning,” Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, pp. 2901-2910. [cited by applicant]
Kabra et al., “Multi-Object Datasets,” https://github.com/deepmind/multi-object-datasets, 2019, 7 pages. [cited by applicant]
Kahneman et al., “The Reviewing of Object Files: Object-Specific Integration of Information,” Cognitive Psychology, 1992, pp. 175-219, vol. 24. [cited by applicant]
Kingma et al., “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980, Jan. 30, 2017, pp. 1-15. [cited by applicant]
Kipf et al., “Constrastive Learning of Structured World Models,” arXiv preprint arXiv:1911.12247, Jan. 5, 2020, pp. 1-21. [cited by applicant]
Kipf et al., “Semi-Supervised Classification with Graph Convolutional Networks,” arXiv preprint arXiv:1609.02907, Feb. 22, 2017, pp. 1-14. [cited by applicant]
Knyazev et al., “Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation,” arXiv preprint arXiv:2005.08230, May 17, 2020, pp. 1-17. [cited by applicant]
Kosiorek et al., “Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects,” Advances in Neural Information Processing Systems, 2018, pp. 8606-8616. [cited by applicant]
Kuhn, Harold W., “Chapter 2—The Hungarian Method for the Assignment Problem,” Naval Research Logistics Quarterly, 1955, pp. 83-97, No. 2, vol. 1-2. [cited by applicant]
Kulkarni et al., “Unsupervised Learning of Object Keypoints for Perception and Control,” Advances in Neural Information Processing Systems, 2019, pp. 10723-10733. [cited by applicant]
Lecun et al., Convolutional Networks for Images, Speech, and Time-Series, The Handbook of Brain Theory and Neural Networks, 1995, vol. 3361, No. 10. [cited by applicant]
Lee et al., “Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks,” arXiv preprint arXiv:1810.00825, May 26, 2019, 17 pages. [cited by applicant]
Li et al., “Gated Graph Sequence Neural Networks,” arXiv preprint arXiv:1511.05493, Published as a conference paper at ICLR 2016, Sep. 22, 2017, pp. 1-20. [cited by applicant]
Lin et al., “A Structured Self-Attentive Sentence Embedding,” arXiv preprint arXiv:1703.0313, Published as a conference paper at ICLR 2017, Mar. 9, 2017, pp. 1-15. [cited by applicant]
Lin et al., “SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition,” arXiv preprint arXiv: 2001.02497, Mar. 15, 2020, pp. 1-22. [cited by applicant]
Locatello et al., “Competitive Training of Mixtures of Independent Deep Generative Models,” arXiv preprint arXiv:1804.11130, Mar. 3, 2019, pp. 1-20. [cited by applicant]
Luong et al., “Effective Approaches to Attention-based Neural Machine Translation,” arXiv preprint arXiv:1508.04025, Sep. 20, 2015, 11 pages. [cited by applicant]
Marino et al., “Iterative Amortized Inference,” arXiv preprint arXiv:1807.09356, Jul. 24, 2018, 20 pages. [cited by applicant]
Mnih et al., “Recurrent Models of Visual Attention,” Advances in Neural Information Processing Systems, 2014, pp. 2204-2212. [cited by applicant]
Mrowca et al, “Flexible Neural Representation for Physics Prediction,” Advances in Neural Information Processing Systems, 2018, pp. 8799-8810. [cited by applicant]
Nash et al., “The Multi-Entity Variational Autoencoder,” 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 2017, 8 pages. [cited by applicant]
Parascandolo et al., “Learning Independent Causal Mechanisms,” Proceedings of the 35th International Conference on Machine Learning, 2018, pp. 4036-4044. [cited by applicant]
Rand, William M., “Objective Criteria for the Evaluation of Clustering Methods,” Journal of the American Statistical Association, Dec. 1971, pp. 846-850, vol. 66, No. 336. [cited by applicant]
Ren et al., “End-to-End Instance Segmentation with Recurrent Attention,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6656-6664. [cited by applicant]
Rezatofighi et al., “Learn to Predict Sets Using Feed-Forward Neural Networks,” arXiv preprint arXiv:2001.11845, Jan. 30, 2020, pp. 1-15. [cited by applicant]
Romera-Paredes et al., “Recurrent Instance Segmentation,” arXiv preprint arXiv:1511.08250, Oct. 24, 2016, pp. 1-24. [cited by applicant]
Sabour et al., “Dynamic Routing Between Capsules,” Advances in Neural Information Processing Systems, 2017, pp. 2856-3866. [cited by applicant]
Sanchez-Gonzalez et al., “Learning to Simulate Complex Physics with Graph Networks,” arXiv preprint arXiv:2020.09405, Feb. 21, 2020, 19 pages. [cited by applicant]
Scarselli et al., “The graph neural network model,” IEEE Transactions on Neural Networks, 2009, pp. 61-80, vol. 20, No. 1. [cited by applicant]
Shaw et al, “Self-Attention with Relative Position Representations,” arXiv preprint arXiv:1803.02155, Apr. 12, 2018, 5 pages. [cited by applicant]
Simonovsky et al., “GraphVAE: Towards Generations of Small Graphs Using Variational Autoencoders,” arXiv preprint arXiv:1802.03480, Feb. 9, 2018, 10 pages. [cited by applicant]
Stelzner et al., “Faster Attend-Infer-Repeat with Tractable Probabilistic Models,” Proceedings of the 36th International Conference on Machine Learning, Long Beach, California, 2019, pp. 5966-5975. [cited by applicant]
Stewart et al., “End-to-End people detection in crowded scenes,” Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2325-2333. [cited by applicant]
Sun et al., “Stochastic Prediction of Multi-Agent Interactions from Partial Observations,” arXiv preprint arXiv:1902.09641, Feb. 25, 2019, pp. 1-15. [cited by applicant]
Tsai et al., “Capsules with Inverted Dot-Product Attention Routing,” arXiv preprint arXiv:2002.04764, Feb. 26, 2020, pp. 1-15. [cited by applicant]
Van Steenkiste et al., “Investigating Object Compositionality in Generative Adversarial Networks,” arXiv preprint arXiv:1810.10340, Dec. 3, 2019, pp. 1-38. [cited by applicant]
Van Steenkiste et al., “Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions,” arXiv preprint arXiv:1802.10353, Feb. 28, 2018, pp. 1-15. [cited by applicant]
Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems, 2017, pp. 5998-6008. [cited by applicant]
Vinyals et al., “Grandmaster level in StarCraft II using multi-agent reinforcement learning,” Nature, Nov. 14, 2019, pp. 350-354, vol. 575, No. 7782. [cited by applicant]
Von Kügelgen et al., “Towards Causal Generative Scene Models via Competition of Experts,” arXiv preprint arXiv:2004.12906, 2020, pp. 1-20. [cited by applicant]
Watters et al., “Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs,” arXiv preprint arXiv:1901.07017, Aug. 14, 2019, pp. 1-35. [cited by applicant]
Watters et al., “Visual Interaction Networks: Learning a Physics Simulator from Video,” Advances in Neural Information Processing Systems, 2017, pp. 4539-4547. [cited by applicant]
Yang et al., “Learning to Manipulate Individual Objects in an Image,” arXiv preprint arXiv:2004.05495, 2020, pp. 6558-6567. [cited by applicant]
Yi et al., “CLEVRER: Collision Events for Video Representation and Reasoning,” arXiv preprint arXiv: 1910.01442, Mar. 8, 2020, pp. 1-19. [cited by applicant]
Ying et al., “Hierarchical Graph Representation Learning with Differentiable Pooling,” Advances in Neural Information Processing Systems, 2018, pp. 4800-4810. [cited by applicant]
Zaheer et al., “Deep Sets,” Advances in Neural Information Processing Systems, 2017, pp. 3391-3401. [cited by applicant]
Zeng et al., “DMM-Net: Differential Mask-Matching Network for Video Object Segmentation,” Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 3929-3938. [cited by applicant]
Zhang et al., “Deep Set Prediction Networks,” Advances in Neural Information Processing Systems, 2019, pp. 3207-3217. [cited by applicant]
Zhang et al., “FSPool: Learning Set Representations with Featurewise Sort Pooling,” arXiv preprint arXiv: 1906.02795, May 1, 2020, pp. 1-20. [cited by applicant]
Chinese Patent Office, Office Action issued on Mar. 8, 2024, in connection with Chinese Patent Application No. 202110619117.8, 13 pages. [cited by applicant]