IP Library › Granted Patent US 12,475,384
Granted Patent B2
US 12,475,384 · App. 17/093,185 · Granted Nov 18, 2025

Self-supervised visual-relationship probing

Inventors: Jiuxiang Gu (Greenbelt, MD); Vlad Ion Morariu (Potomac, MD); Tong Sun (San Ramon, CA); Jason wen yong Kuen (Santa Clara, CA); Handong Zhao (San Jose, CA)
Assignee: Adobe Inc.
G06N5/022G06N3/08G06N7/00G06T7/90
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,475,384
App. No.
17/093,185
Granted
Nov 18, 2025
Kind
B2
Abstract

Methods and systems disclosed herein relate generally to systems and methods for generating visual relationship graphs that identify relationships between objects depicted in an image. A vision-language application uses transformer encoders to generate a graph structure, in which the graph structure represents a dependency between a first region and a second region of an image. The dependency indicates that a contextual representation of the first region was derived, at least in part, by processing the second region. The contextual representation identifies a predicted identity of an image object depicted in the first region. The predicted identity is determined at least in part by identifying a relationship between the first region and other data objects associated with various modalities.

Claims (69)

1 . A method comprising:

receiving an image;

receiving an indication of a vision-language (VL) operation relating to the image, wherein the VL operation includes at least one of a VL understanding task or a VL generation task,

generating, by a vision-language modeling application, an input embedding that identifies a visual characteristic of a first region within the image and a position of the first region within the image;

encoding, with a first transformer encoder of the vision-language modeling application, the input embedding into an intra-modality representation of the first region, wherein the intra-modality representation identifies an image object depicted in the first region based on analyzing a second region within the image and the intra-modality representation is a first feature vector;

encoding, with a second transformer encoder of the vision-language modeling application, the intra-modality representation into an inter-modality representation of the first region, wherein the inter-modality representation is a second feature vector based on one or more visual feature vectors representing the image object and one or more textual feature vectors corresponding to a token that describes the image object, wherein the token is included in a plurality of tokens that are derived from a text sequence;

generating, by the vision-language modeling application and from the inter-modality representation, a graph structure that represents a dependency between the first region and the second region, wherein the dependency indicates that the inter-modality representation of the first region was derived, at least in part, by processing the second region and comprising:

computing pairwise distances between the one or more visual feature vectors and the one or more textual feature vectors of the inter-modality representations of the first region, wherein the pairwise distances represent relationships between the visual feature vectors and between the textual feature vectors, respectively; and

constructing the graph structure based using the pairwise distances, wherein the relationship between the first region and the second region are based on the pairwise distances;

executing the VL operation using the image and based on the dependency of the graph structure; and

outputting a result, comprising information about the image based on an output of the execution of the VL operation.

2 . The method of claim 1 , wherein the VL operation further comprises at least one of: using the graph structure to identify another image that depicts a second image object that shares the visual characteristic and the position identified by the input embedding of the first region, or using the dependency of the graph structure to determine whether the text sequence characterizes a plurality of image objects depicted in the image.

3 . The method of claim 1 , wherein the graph structure includes a set of edges connecting the first region and one or more other regions, and wherein a length of an edge of the set of edges identifies a degree of relatedness between the first region and another region to which the edge is connected.

4 . The method of claim 1 , wherein encoding, with the second transformer encoder of the vision-language modeling application, the intra-modality representation into the inter-modality representation of the first region includes:

executing, by the vision-language modeling application, a shared self-attention sub-layer of the second transformer encoder to process a plurality of regions and generate a first output;

executing, by the vision-language modeling application, the shared self-attention sub-layer to process the plurality of tokens and generate a second output; and

generating, by the vision-language modeling application, the inter-modality representation for the first region based on the first output and the second output.

5 . The method of claim 4 , further comprising:

executing, by the vision-language modeling application, a cross-attention sub-layer of the second transformer encoder to process the plurality of regions with the plurality of tokens and generate a third output; and

generating, by the vision-language modeling application, the inter-modality representation for the first region based on the second output and the third output.

6 . The method of claim 1 , further comprising overlaying the graph structure over the image.

7 . The method of claim 1 , further comprising generating a heat map that represents the graph structure, wherein the heat map includes a set of heat-map elements, and wherein a color of a particular heat-map element identifies a degree of relatedness between the first region and a region of one or more other regions.

8 . A system comprising:

a processor;

an input-embedding module configured to generate an input embedding for a token of a set of tokens, wherein the input embedding encodes a position of the token within a text sequence from which the set of tokens were derived;

a first transformer encoding module configured to encode the input embedding that represents the token into an intra-modality representation of the token, wherein the intra-modality representation identifies a definition of the token based on an analysis of one or more other tokens from the set of tokens and the intra-modality representation is a first feature vector; and

a second transformer encoding module configured to encode the intra-modality representation into an inter-modality representation of the token, wherein the inter-modality representation is a second feature vector based on one or more textual feature vectors including the token defining a region of an image depicting an image object and one or more visual feature vectors representing the image object; and

a relationship-probing module configured to generate, from the inter-modality representation, a graph structure that represents one or more dependencies between the token and the one or more other tokens by:

computing pairwise distances between the one or more visual feature vectors and between the one or more textual feature vectors of the inter-modality representations, respectively, wherein the pairwise distances represent relationships between the visual feature vectors and the textual feature vectors; and

constructing the graph structure based using the pairwise distances, wherein the relationship between the region of the image and other regions of the image are based on the pairwise distances; and

a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:

receiving the image;

receiving an indication of a vision-language (VL) operation relating to the image, wherein the VL operation includes at least one of a VL understanding task or a VL generation task;

outputting the image to the input-embedding module;

receiving, from the relationship-probing module, the graph structure;

executing the VL operation using the image and based on the dependency of the graph structure; and

outputting a result, comprising information about the image based on an output of the execution of the VL operation.

9 . The system of claim 8 , wherein the instructions further cause the processor to:

generate another graph structure that represents one or more second dependencies between a plurality of regions of the image, wherein the one or more second dependencies between the plurality of regions are derived by processing the set of tokens.

10 . The system of claim 8 , wherein the graph structure includes a set of edges connecting the token with the one or more other tokens, and wherein a length of an edge of the set of edges identifies a degree of relatedness between the token and another token to which the edge is connected.

11 . The system of claim 8 , wherein the second transformer encoding module is configured to encode the intra-modality representation into the inter-modality representation of the token by:

applying a shared self-attention sub-layer of the second transformer encoding module to process the set of tokens thereby generating a first output;

applying the shared self-attention sub-layer to process a plurality of regions thereby generating a second output; and

generating the inter-modality representation for the region based on the first output and the second output.

12 . The system of claim 11 , wherein the instructions further cause the processor to:

apply a cross-attention sub-layer of the second transformer encoding module to process the set of tokens with the plurality of regions thereby generating a third output; and

generate the inter-modality representation for the token based on the second output and the third output.

13 . The system of claim 8 , wherein the instructions further cause the processor to:

generate a dependency tree that represents the graph structure; and

overlay the dependency tree over the text sequence.

14 . The system of claim 8 , wherein the instructions further cause the processor to generate a heat map that represents the graph structure, wherein the heat map includes a set of heat-map elements, and wherein a color of a particular heat-map element identifies a degree of relatedness between a corresponding token and a token of the one or more other tokens.

15 . A computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform actions including:

receiving an image;

receiving an indication of a vision-language (VL) operation relating to the image, wherein the VL operation includes at least one of a VL understanding task or a VL generation task;

identifying, for each data object of a plurality of multimodal data objects in the image, an intra-modality representation derived from an input embedding that represents the data object, wherein:

the data object of the plurality of multimodal data objects represents:

a region of a plurality of regions depicted in the image; or

a token of a plurality of tokens in a text characterizing the plurality of regions; and

the intra-modality representation is a first feature vector;

identifying, for each intra-modality representation of a particular data object, an inter-modality representation, the inter-modality representation comprising a second feature vector based on one or more visual feature vectors and one or more textual feature vectors, wherein the visual feature vectors are generated by processing intra-modality representations corresponding to image regions and the textual feature vectors are generated by processing intra-modality representations of tokens that describe the particular data object;

a step for generating a graph structure by processing the inter-modality representations of the plurality of multimodal data objects based on pairwise distances between the one or more visual feature vectors and the one or more textual feature vectors of the inter-modality representation, wherein the pairwise distances represent relationships between the visual feature vectors and between the textual feature vectors, respectively;

executing the VL operation using the image and based on a dependency of the graph structure; and

outputting a result, comprising information about the image based on an output of the execution of the VL operation.

16 . The computer program product of claim 15 , wherein the intra-modality representation is generated by applying a Bidirectional Encoder Representations from Transformers (BERT) model to the data object, wherein the intra-modality representation identifies one or more characteristics of the data object and one or more associations between the data object and other data objects of the plurality of multimodal data objects.

17 . The computer program product of claim 15 , further comprising instructions configured to cause the one or more data processors to perform actions including:

generating, for each data object of the plurality of multimodal data objects, the input embedding that represents the data object, wherein the input embedding is a third feature vector generated by applying a convolutional neural network to the data object.

18 . The computer program product of claim 15 , wherein the graph structure is an image-based graph structure that identifies one or more dependencies between the plurality of regions.

19 . The computer program product of claim 15 , wherein the graph structure is a text-based graph structure that identifies one or more dependencies between each pair of the plurality of tokens.

20 . The computer program product of claim 15 , further comprising instructions configured to cause the one or more data processors to perform actions including generating a heat map that represents the graph structure, wherein the heat map includes a set of heat-map elements, and wherein a color of a particular heat-map element identifies a degree of relatedness between the data object and another data object of the plurality of multimodal data objects.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 9, 2020
From: GU, JIUXIANG; MORARIU, VLAD ION; SUN, TONG; KUEN, JASON WEN YONG; ZHAO, HANDONG
To: ADOBE INC.
Reel/Frame 054317/0062 →
Continuity (1)
Related Publication 20220147838A1 · May 12, 2022
References Cited (73)
US 20170132526A1 · Cohen · 2017 [cited by examiner]
US 20180181592A1 · Chen · 2018 [cited by examiner]
US 20200160178A1 · Kar · 2020 [cited by examiner]
US 20210081728A1 · Lai · 2021 [cited by examiner]
US 20210275918A1 · Devaranjan · 2021 [cited by examiner]
US 20210374547A1 · Wang · 2021 [cited by examiner]
US 20220014807A1 · Lin · 2022 [cited by examiner]
Liu et al.(“Graph Structured Network for Image-Text Matching” , Apr. 1, 2020) (Year: 2020). [cited by examiner]
Tan et al.(“LXMERT: Learning Cross-Modality Encoder Representations from Transformers”, Dec. 3, 2019) (Year: 2019). [cited by examiner]
ICCV (“Scene Graph Representation and Learning”, Oct. 28, 2019) (Year: 2019). [cited by examiner]
Koner et al. (“Relation Transformer Network” , Apr. 13, 2020) (Year: 2020). [cited by examiner]
Tripathi et al.(“Triplet-Aware Scene Graph Embeddings” , Sep. 19, 2019) (Year: 2019). [cited by examiner]
Anderson et al., Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 18-23, 2018, pp. 6077-6086. [cited by applicant]
Antol et al., VQA: Visual Question Answering, Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2425-2433. [cited by applicant]
Ba et al., Layer Normalization, arXiv:1607.06450, Available Online at: https://arxiv.org/abs/1607.06450, 2016, 14 pages. [cited by applicant]
Chen et al., A Simple Framework for Contrastive Learning of Visual Representations, Proceedings of the 37th International Conference on Machine Learning, 2020, 20 pages. [cited by applicant]
Chen et al., Microsoft COCO Captions: Data Collection and Evaluation Server, arXiv:1504.00325, 2015, 7 pages. [cited by applicant]
Chen et al., Scene Graph Prediction with Limited Labels, 2019 International Conference on Computer Vision, 2019, 11 pages. [cited by applicant]
Chen et al., UNITER: Learning Universal Image-text Representations, arXiv:1909.11740 [cs.CV], Available Online at: https://arxiv.org/abs/1909.11740, 2019, 26 pages. [cited by applicant]
Clark et al., What Does BERT Look At? An Analysis of BERT's Attention, In BlackBoxNLP@ACL, 2019, 11 pages. [cited by applicant]
Coenen et al., Visualizing and Measuring the Geometry of Bert, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), 2019, 16 pages. [cited by applicant]
Dai, Transformer-XL: Attentive Language Models Beyond a Fixed-length Context, Annual Meeting of the Association for Computational Linguistics, 2019, 20 pages. [cited by applicant]
Devlin et al., BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Huma… [cited by applicant]
Dornadula et al., Visual Relationships as Functions: Enabling Few-shot Scene Graph Prediction, International Conference on Computer Vision Workshop (ICCVW), 2019, 10 pages. [cited by applicant]
Garcia et al., Few-Shot Learning with Graph Neural Networks, International Conference on Learning Representations, 2018, 13 pages. [cited by applicant]
Girshick, Fast R-CNN, In Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1440-1448. [cited by applicant]
Gu et al., Scene Graph Generation with External Knowledge and Image Reconstruction, Conference on Computer Vision and Pattern Recognition, 2019, 10 pages. [cited by applicant]
Gu et al., Unpaired Image Captioning via Scene Graph Alignments, International Conference on Computer Vision, Available Online at: https://arxiv.org/abs/1903.10658, 2019, 10 pages. [cited by applicant]
He et al., Deep Residual Learning for Image Recognition, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778. [cited by applicant]
Hewitt et al., A Structural Probe for Finding Syntax in Word Representations, North American Chapter of the Association for Computational Linguistics, 2019, 10 pages. [cited by applicant]
Hu et al., Relation Networks for Object Detection, Computer Vision and Pattern Recognition, 2018, 10 pages. [cited by applicant]
Hudson et al., GQA: A New Dataset for Real-world Visual Reasoning and Compositional Question Answering, Conference on Computer Vision and Pattern Recognition, Available Online at: https://arxiv.org/abs/1902.09506, 2019,… [cited by applicant]
Johnson et al., Image Retrieval Using Scene Graphs, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, 11 pages. [cited by applicant]
Johnson et al., Inferring and Executing Programs for Visual Reasoning, International Conference on Computer Vision, 2017, 10 pages. [cited by applicant]
Karpathy et al., Deep Visual-Semantic Alignments for Generating Image Descriptions, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 10 pages. [cited by applicant]
Kingma et al., ADAM: A Method for Stochastic Optimization, In International Conference on Learning Representations, 2015, pp. 1-15. [cited by applicant]
Krishna et al., Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations, International Journal of Computer Vision, vol. 123, 2017, pp. 32-73. [cited by applicant]
Levine et al., SenseBERT: Driving Some Sense into BERT, 2020 Association for Computational Linguistics, 2020, 12 pages. [cited by applicant]
Li et al., Oscar: Object-semantics Aligned Pre-training for Vision-language Tasks, arXiv preprint arXiv:2004.06165, 2020, 21 pages. [cited by applicant]
Li et al., Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training, Association for the Advancement of Artificial Intelligence 2020, 2020, 8 pages. [cited by applicant]
Li et al., VisualBERT: A Simple and Performant Baseline for Vision and Language, arXiv:1908.03557, Available Online at: https://arxiv.org/abs/1908.03557, 2019, 14 pages. [cited by applicant]
Lin et al., Microsoft COCO: Common Objects in Context, European Conference on Computer Vision, Feb. 21, 2015, pp. 740-755. [cited by applicant]
Lu et al., ViLBERT: Pretraining Task-agnostic Visiolinguistic Representations for Vision-and-language Tasks, 2019 Conference on Neural Information Processing Systems, 2019, 11 pages. [cited by applicant]
Lu et al., Visual Relationship Detection with Language Priors, European Conference on Computer Vision 2016, 2016, 19 pages. [cited by applicant]
Macleod et al., Understanding Blind People's Experiences with Computer-Generated Captions of Social Media Images, CHI 2017, Available Online at: https://www.microsoft.com/enus/research/uploads/prod/2016/10/captions_chi2… [cited by applicant]
Ng et al., Facebook FAIR's WMT19 News Translation Task Submission, arXiv preprint arXiv:1907.06616, Available Online at: https://arxiv.org/abs/1907.06616, 2019, 7 pages. [cited by applicant]
Norcliffe-Brown et al., Learning Conditioned Graph Structures for Interpretable Visual Question Answering, 32nd Conference on Neural Information Processing Systems (NIPS 2018), 2018, 13 pages. [cited by applicant]
Oord et al., Representation Learning with Contrastive Predictive Coding, arXiv preprint arXiv:1807.03748, 2018, 13 pages. [cited by applicant]
Ordonez et al., Im2Text: Describing Images Using 1 Million Captioned Photographs, Neural Information Processing Systems, 2011, 9 pages. [cited by applicant]
Ott et al., fairseq: A Fast, Extensible Toolkit for Sequence Modeling, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), 2019, 6 pages. [cited by applicant]
Pruthi et al., Learning to Deceive with Attention-Based Explanations, arXiv preprint arXiv:1909.07913, 2019, 9 pages. [cited by applicant]
Qi et al., Stanza: A Python Natural Language Processing Toolkit for Many Human Languages, Association for Computational Linguistics (ACL), 2020, 8 pages. [cited by applicant]
Radford et al., Improving Language Understanding by Generative Pre-training, Available Online at: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf, … [cited by applicant]
Ren et al., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, Institute of Electrical and Electronics Engineers Transactions on Pattern Analysis and Machine Intelligence, vol. 39, No. 6, Ju… [cited by applicant]
Rennie et al., Self-critical Sequence Training for Image Captioning, Conference on Computer Vision and Pattern Recognition, Available Online at: https://openaccess.thecvf.com/content_cvpr_2017/papers/Rennie_Self-Critica… [cited by applicant]
Santoro et al., A Simple Neural Network Module for Relational Reasoning, Conference on Neural Information Processing Systems, 2017, 16 pages. [cited by applicant]
Sharma et al., Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset for Automatic Image Captioning, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, vol. 1, 2018, pp… [cited by applicant]
Su et al., VL-BERT: Pre-training of Generic Visual-linguistic Representations, The International Conference on Learning Representations (ICLR) 2020, 2020, 16 pages. [cited by applicant]
Suhr et al., A Corpus for Reasoning About Natural Language Grounded in Photographs, Association for Computational Linguistics (ACL), 2019, 22 pages. [cited by applicant]
Sun et al., Videobert: A Joint Model for Video and Language Representation Learning, International Conference on Computer Vision, 2019, 10 pages. [cited by applicant]
Tan et al., LXMERT: Learning Cross-modality, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Nov. 3-7, 20… [cited by applicant]
Teney et al., Graph-structured Representations for Visual Question Answering, Conference on Computer Vision and Pattern Recognition, 2017, 17 pages. [cited by applicant]
Tenney et al., What Do You Learn from Context? Probing for Sentence Structure in Contextualized Word Representations, International Conference on Learning Representations (ICLR), 2019, 17 pages. [cited by applicant]
Vaswani et al., Attention is All You Need, Conference on Neural Information Processing Systems, 2017, 15 pages. [cited by applicant]
Voita et al., Analyzing Multi-Head Self-attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Jul. 2019, pp. … [cited by applicant]
Wang et al., Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks, Conference on Computer Vision and Pattern Recognition, 2019, 9 pages. [cited by applicant]
Wu et al., Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, arXiv preprint arXiv:1609.08144, Available online at: https://arxiv.org/pdf/1609.08144.pdf, Oct. 8, 2016, pp… [cited by applicant]
Wu et al., Unsupervised Feature Learning via Non-Parametric Instance Discrimination, Conference on Computer Vision and Pattern Recognition, 2018, 10 pages. [cited by applicant]
Yang et al., Auto-encoding Scene Graphs for Image Captioning, Conference on Computer Vision and Pattern Recognition, 2019, 15 pages. [cited by applicant]
Yang et al., XLNet: Generalized Autoregressive Pretraining for Language Understanding, Conference on Neural Information Processing Systems, 2019, 18 pages. [cited by applicant]
Yu et al., Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation, International Conference on Computer Vision, 2017, 9 pages. [cited by applicant]
Zellers et al., Neural Motifs: Scene Graph Parsing with Global Context, Conference on Computer Vision and Pattern Recognition, 2018, 10 pages. [cited by applicant]
Zhu et al., Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books, 2015 Institute of Electrical and Electronics Engineers International Conference on Computer Vision, 201… [cited by applicant]