IP Library › Granted Patent US 12,694,672
Granted Patent B2
US 12,694,672 · App. 18/029,980 · Granted Jul 28, 2026

Neural networks implementing attention over object embeddings for object-centric visual reasoning

Inventors: Fengning Ding (London, GB); Adam Anthony Santoro (London, GB); Felix George Hill (London, GB); Matthew Botvinick (Philadelphia, PA); Luis Piloto (London, GB)
Assignee: GDM Holding LLC
G06V20/41G06V10/26G06V10/776G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,694,672
App. No.
18/029,980
Filed
Apr 3, 2023
Granted
Jul 28, 2026
Kind
B2
Examiner
HAIDER, SYED
Art Unit
2633
USPC
382/156
Abstract

A video processing system configured to analyze a sequence of video frames to detect objects in the video frames and provide information relating to the detected objects in response to a query. The query may comprise, for example, a request for a prediction of a future event, or of the location of an object, or a request for a prediction of what would happen if an object were modified. The system uses a transformer neural network subsystem to process representations of objects in the video.

Claims (31)

1 . A video processing system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:

an object segmentation subsystem that is configured to receive an input comprising a sequence of video frames and to process each video frame of the sequence of video frames to generate an object segmentation output wherein, for each video frame, the object segmentation output comprises a representation of each object of a plurality of objects in the video frame;

a transformer neural network subsystem that is configured to receive an input sequence comprising i) an object input sequence including the object segmentation output for each video frame combined with an encoding of a position of the video frame in the sequence of video frames and ii) a query input sequence comprising a sequence of vectors representing a query, and to process the input sequence in parallel for all video frames in the sequence of video frames by applying a succession of attention neural network layers to generate an output sequence comprising a transformed input element for each element of the input sequence, wherein one or more of the attention neural network layers has an attention layer input for each element of the input sequence, the elements of the input sequence comprising a respective element for each representation of each of the plurality of objects of each of the video frames and for each of the sequence of vectors representing the query, and is configured to attend over representations of each of the plurality of objects in each of the video frames and each of the sequence of vectors representing the query by applying an attention mechanism over the attention layer inputs for the elements of the input sequence that include a respective element for each of the representations for each of the video frames and for each of the sequence of vectors representing the query to generate an attention layer output for each element of the input sequence; and

an output subsystem that is configured to receive one or more elements of the transformed input sequence and to process the one or more elements of the transformed input sequence by applying one or more output neural network layers to generate a video processing system output, wherein the video processing system output represents a response to the query on the objects in the sequence of video frames.

2 . The video processing system of claim 1 wherein the input sequence further comprises a summary token element, wherein the output sequence comprises a transformed summary token element representing a summary of the input sequence, and wherein the output subsystem is configured to receive the transformed summary token element and to process the transformed summary token element to generate the video processing system output.

3 . The video processing system of claim 1 wherein the object input sequence comprises a sequence of elements for each video frame, each comprising one of the representations combined with the encoding of the position of the video frame.

4 . The video processing system of claim 1 wherein the object segmentation subsystem is configured to determine, for each of the objects, a set of parameters parameterizing a distribution of a latent variable vector representing the object, and the object segmentation output comprises a mean value vector of the distribution for each object, where the mean value vector is the representation of the object.

5 . The video processing system of claim 1 further comprising a query embedding neural network subsystem that is configured to receive a sequence of tokens representing the query and to process the sequence of tokens to generate the sequence of vectors representing the query, wherein the vectors representing the query comprise embeddings of the tokens.

6 . The video processing system of claim 5 wherein the tokens comprise words or wordpieces in a natural language.

7 . The video processing system of claim 1 wherein elements of the input sequence each include a label to identify whether the element belongs to the object input sequence or to the query input sequence.

8 . The video processing system of claim 1 , wherein the input sequence further comprises a summary token element, wherein the output sequence comprises a transformed summary token element representing a summary of the input sequence, and wherein the output subsystem is configured to receive the transformed summary token element and to process the transformed summary token element to generate the video processing system output, and wherein the summary token element belongs to the query input sequence.

9 . The video processing system of claim 1 wherein one or more of the attention neural network layers is configured to apply a self-attention mechanism over the attention layer input, followed by a feed-forward neural network layer to generate the attention layer output.

10 . The video processing system of claim 1 wherein the attention mechanism is configured to apply each of a query transformation, a key transformation, and a value transformation to the attention layer input for each element of the input sequence to derive a respective query vector, key vector, and value vector; to apply each query vector to each key vector to determine respective weights for each value vector, and combining the value vectors using the respective weights to determine the attention layer output for each element of the input sequence.

11 . The video processing system of claim 1 , wherein the video processing system output defines a probability distribution over a set of possible responses to the query.

12 . The video processing system of claim 1 , wherein the video processing system further comprises a prediction neural network that is configured to receive and process, for each video frame, the transformed input elements for the representations of each object of the plurality of objects in the video frame, and to generate from the transformed input elements a predicted representation for the representation of each object in each video frame, and wherein the video processing system has been trained by performing operations comprising: suppressing, in the object segmentation output, the representation of one or more the objects for one or more of the video frames; determining an infill loss dependent upon a measure of difference between the suppressed representation and the predicted representation, and backpropagating gradients of the infill loss to adjust parameters of the transformer neural network subsystem.

13 . The video processing system of claim 1 , wherein the video processing system has been trained by performing operations comprising: determining an auxiliary loss dependent upon a measure of a distance between the response to the query and a correct response to the query, and backpropagating gradients of the auxiliary loss to adjust parameters of the transformer neural network subsystem.

14 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:

an object segmentation subsystem that is configured to receive an input comprising a sequence of video frames and to process each video frame of the sequence of video frames to generate an object segmentation output wherein, for each video frame, the object segmentation output comprises a representation of each object of a plurality of objects in the video frame;

a transformer neural network subsystem that is configured to receive an input sequence comprising i) an object input sequence including the object segmentation output for each video frame combined with an encoding of a position of the video frame in the sequence of video frames and ii) a query input sequence comprising a sequence of vectors representing a query, and to process the input sequence in parallel for all video frames in the sequence of video frames by applying a succession of attention neural network layers to generate an output sequence comprising a transformed input element for each element of the input sequence, wherein one or more of the attention neural network layers has an attention layer input for each element of the input sequence, the elements of the input sequence comprising a respective element for each representation of each of the plurality of objects of each of the video frames and for each of the sequence of vectors representing the query, and is configured to attend over representations of each of the plurality of objects in each of the video frames and each of the sequence of vectors representing the query by applying an attention mechanism over the attention layer inputs for the elements of the input sequence that include a respective element for each of the representations for each of the video frames and for each of the sequence of vectors representing the query to generate an attention layer output for each element of the input sequence; and

an output subsystem that is configured to receive one or more elements of the transformed input sequence and to process the one or more elements of the transformed input sequence by applying one or more output neural network layers to generate a video processing system output, wherein the video processing system output represents a response to the query on the objects in the sequence of video frames.

15 . A method performed by one or more computers, the method comprising:

receiving an input comprising a sequence of video frames; and

processing the input comprising the sequence of video frames using a video processing system to generate a video processing system output, wherein the video processing system comprises:

an object segmentation subsystem that is configured to receive the input comprising the sequence of video frames and to process each video frame of the sequence of video frames to generate an object segmentation output wherein, for each video frame, the object segmentation output comprises a representation of each object of a plurality of objects in the video frame;

a transformer neural network subsystem that is configured to receive an input sequence comprising i) an object input sequence including the object segmentation output for each video frame combined with an encoding of a position of the video frame in the sequence of video frames and ii) a query input sequence comprising a sequence of vectors representing a query, and to process the input sequence in parallel for all video frames in the sequence of video frames by applying a succession of attention neural network layers to generate an output sequence comprising a transformed input element for each element of the input sequence, wherein one or more of the attention neural network layers has an attention layer input for each element of the input sequence, the elements of the input sequence comprising a respective element for each representation of each of the plurality of objects of each of the video frames and for each of the sequence of vectors representing the query, and is configured to attend over representations of each of the plurality of objects in each of the video frames and each of the sequence of vectors representing the query by applying an attention mechanism over the attention layer inputs for the elements of the input sequence that include a respective element for each of the representations for each of the video frames and for each of the sequence of vectors representing the query to generate an attention layer output for each element of the input sequence; and

an output subsystem that is configured to receive one or more elements of the transformed input sequence and to process the one or more elements of the transformed input sequence by applying one or more output neural network layers to generate a video processing system output, wherein the video processing system output represents a response to the query on the objects in the sequence of video frames.

16 . The method of claim 15 wherein the input sequence further comprises a summary token element, wherein the output sequence comprises a transformed summary token element representing a summary of the input sequence, and wherein the output subsystem is configured to receive the transformed summary token element and to process the transformed summary token element to generate the video processing system output.

17 . The method of claim 15 wherein the object input sequence comprises a sequence of elements for each video frame, each comprising one of the representations combined with the encoding of the position of the video frame.

18 . The method of claim 15 wherein the object segmentation subsystem is configured to determine, for each of the objects, a set of parameters parameterizing a distribution of a latent variable vector representing the object, and the object segmentation output comprises a mean value vector of the distribution for each object, where the mean value vector is the representation of the object.

19 . The video processing system of claim 1 wherein the query comprises a request for a prediction of a future event or state relating to one or more of the objects in the sequence of video frames.

20 . The method of claim 15 further comprising a query embedding neural network subsystem that is configured to receive a sequence of tokens representing the query and to process the sequence of tokens to generate the sequence of vectors representing the query, wherein the vectors representing the query comprise embeddings of the tokens.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 13, 2023
From: DING, FENGNING; SANTORO, ADAM ANTHONY; HILL, FELIX GEORGE; BOTVINICK, MATTHEW; PILOTO, LUIS
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 063933/0191 →
Continuity (2)
Provisional Application 63087041 · Oct 2, 2020
Related Publication 20240020972A1 · Jan 18, 2024
References Cited (55)
US 20170185846A1 · Hwangbo · 2017 [cited by examiner]
US 20190354567A1 · Dehghani et al. · 2019 [cited by applicant]
US 20200034627A1 · Zhu · 2020 [cited by examiner]
US 20210103733A1 · Jiang · 2021 [cited by examiner]
US 20210248376A1 · Zhao · 2021 [cited by examiner]
US 20210321172A1 · Krishnamurthy · 2021 [cited by examiner]
US 20210390316A1 · Schoneveld · 2021 [cited by examiner]
US 20230178199A1 · Soltani Bidgoli · 2023 [cited by examiner]
CN 108733792A · 2018 [cited by applicant]
WO WO2020104499 · 2020 [cited by applicant]
Battaglia et al., “Relational inductive biases, deep learning, and graph networks,” CoRR, Jun. 4, 2018, arxiv.org/abs/1806.01261, 40 pages. [cited by applicant]
Brown et al., “Language models are few-shot learners,” CoRR, May 28, 2020, arXiv:2005.14165, 75 pages. [cited by applicant]
Burgess et al., “MONet: Unsupervised scene decomposition and representation,” CoRR, Jan. 22, 2019, arXiv:1901.11390, 22 pages. [cited by applicant]
Carion et al., “End-to-End Object Detection with Transformers,” CoRR, May 26, 2020, arxiv.org/abs/2005.12872, 26 pages. [cited by applicant]
Carreira et al., “Quo vadis, action recognition? A new model and the kinetics dataset,” CoRR, May 22, 2017, arxiv.org/abs/1705.07750, 10 pages. [cited by applicant]
Chen et al., “A simple framework for contrastive learning of visual representations,” CoRR, Feb. 13, 2020, arXiv:2002.05709, 20 pages. [cited by applicant]
Chen, “Object-based attention: A tutorial review,” Attention, Perception, & Psychophysics, Jun. 7, 2012, 74:784-802. [cited by applicant]
Dai et al., “Transformer-XL: Attentive language models beyond a fixed-length context,” CoRR, Jan. 9, 2019, arXiv:1901.02860, 20 pages. [cited by applicant]
Desta et al., “Object-based reasoning in VQA,” CoRR, Jan. 29, 2018, arxiv.org/abs/1801.09718, 10 pages. [cited by applicant]
Devlin et al., “BERT: Pre-training of deep bidirectional transformers for language understanding,” CoRR, Oct. 11, 2018, arXiv:1810.04805, 16 pages. [cited by applicant]
Du et al., “Unsupervised discovery of 3D physical objects from video,” CoRR, Jul. 24, 2020, arXiv:2007.12348, 15 pages. [cited by applicant]
Girdhar et al., “CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning,” International Conference on Learning Representations, Dec. 19, 2019, 16 pages. [cited by applicant]
Greff et al., “Multi-object representation learning with iterative variational inference,” CoRR, Mar. 1, 2019, arxiv.org/abs/1903.00450, 28 pages. [cited by applicant]
Gregor et al., “Shaping belief states with generative environment models for RL,” Advances in Neural Information Processing Systems 32, 2019, 13 pages. [cited by applicant]
Hahn et al., “Transformers generalize to the semantics of logics,” CoRR, 2020, arXiv:2003.04218, 15 pages. [cited by applicant]
Han et al., “Video representation learning by dense predictive coding,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, 10 pages. [cited by applicant]
He et al., “Deep residual learning for image recognition,” CoRR, Dec. 10, 2015, arxiv.org/abs/1512.03385, 12 pages. [cited by applicant]
He et al., “Mask R-CNN,” CoRR, Mar. 20, 2017, arxiv.org/abs/1703.06870, 12 pages. [cited by applicant]
Hill et al., “Human instruction-following with deep reinforcement learning via transfer-learning from text,” CoRR, May 19, 2020, arXiv:2005.09382, 19 pages. [cited by applicant]
Hudson et al., “Compositional attention networks for machine reasoning,” CoRR, Mar. 8, 2018, arxiv.org/abs/1803.03067, 20 pages. [cited by applicant]
International Search Report and Written Opinion in International Appln. No. PCT/EP2021/077168, dated Jan. 5, 2022, 16 pages. [cited by applicant]
Johnson et al., “CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning,” CoRR, Dec. 20, 2016, arxiv.org/abs/1612.06890, 17 pages. [cited by applicant]
Kingma et al., “Adam: A method for stochastic optimization,” CoRR, Dec. 22, 2014, arXiv:1412.6980, 15 pages. [cited by applicant]
Lample et al., “Deep learning for symbolic mathematics,” International Conference on Learning Representations, Dec. 19, 2020, 24 pages. [cited by applicant]
Li et al., “Visualbert: A simple and performant baseline for vision and language,” CoRR, Aug. 9, 2019, arXiv:1908.03557, 14 pages. [cited by applicant]
Lin et al., “SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition,” International Conference on Learning Representations, Dec. 19, 2019, 22 pages. [cited by applicant]
Lu, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” CoRR, Aug. 6, 2019, arXiv:1908.02265, 11 pages. [cited by applicant]
Medium.com [online], “Can GPT-3 Make Analogies?,” Aug. 5, 2020, retrieved on May 3, 2024, retrieved from URL<https://medium.com/@melaniemitchell.me/can-gpt-3-make-analogies-16436605c446/>, 26 pages. [cited by applicant]
Raposo et al., “Discovering objects and their relations from entangled scene representations,” CoRR, Feb. 16, 2017, arxiv.org/abs/1702.05068, 16 pages. [cited by applicant]
Ren et al., “Faster R-CNN: towards real-time object detection with region proposal networks,” CoRR, Jun. 4, 2015, arxiv.org/abs/1506.01497, 14 pages. [cited by applicant]
Roelfsema et al., “Object-based attention in the primary visual cortex of the macaque monkey,” Nature, 1998, 395:376-381. [cited by applicant]
Sadhu et al., “Video Object Grounding Using Semantic Roles in Language Description,” 2020 IEEE/CVF Conferences on Computer Vision and Pattern Recognition (CVPR), Jun. 13, 11 pages. [cited by applicant]
Spelke, “Core knowledge,” The American Psychologist, 2000, 55:1233-43. [cited by applicant]
Steenkiste et al., “A perspective on objects and systematic generalization in model-based RL,” CoRR, Jun. 3, 2019, arxiv.org/abs/1906.01035, 6 pages. [cited by applicant]
Su et al., “Vl-bert: Pre-training of generic visual-linguistic representations,” International Conference on Learning Representations, Dec. 19, 2019, 16 pages. [cited by applicant]
Sun et al., “Contrastive bidirectional transformer for temporal representation learning,” CoRR, Jun. 13, 2019, arxiv.org/abs/1906.05743, 12 pages. [cited by applicant]
Sun et al., “VideoBERT: A joint model for video and language representation learning,” CoRR, Apr. 3, 2019, arxiv.org/abs/1904.01766, 13 pages. [cited by applicant]
Vaswani et al., “Attention is all you need,” Advances in neural information processing systems 30, 2017, 11 pages. [cited by applicant]
Wang et al., “Non-local neural networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7794-7803. [cited by applicant]
Wu et al., “Analogical Reasoning for Visually Grounded Language Acquisition,” CoRR, Jul. 22, 2020, arXiV:2007.11668v1, 12 pages. [cited by applicant]
Yi et al., “CLEVRER: Collision events for video representation and reasoning,” International Conference on Learning Representations, Dec. 19, 2019, 19 pages. [cited by applicant]
You et al., “Large batch optimization for deep learning: Training BERT in 76 minutes,” Technical Report UCB/EECS-2019-103, EECS Department, Jun. 21, 2019, 38 pages. [cited by applicant]
Zambaldi et al., “Deep reinforcement learning with relational inductive biases,” International Conference on Learning Representations, Dec. 20, 2018, 18 pages. [cited by applicant]
Zhu et al., “An Empirical Study of Spatial Attention Mechanisms in Deep Networks,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 27, 2019, pp. 6687-6696. [cited by applicant]
Office Action in Chinese Appln. No. 202180067194.5, mailed on Nov. 26, 2025, 22 pages (with English translation). [cited by applicant]