IP Library › Granted Patent US 12,579,184
Granted Patent B2
US 12,579,184 · App. 18/470,526 · Granted Mar 17, 2026

Method and system for image categorization using a visual language model

Inventors: Ivona Najdenkoska (Amsterdam, NL); Mohammad Derakhshani (Amsterdam, NL); Yuki Asano (Amsterdam, NL); Cees Snoek (Volendam, NL); Marcel Worring (Amsterdam, NL)
Assignee: INCEPTION AI IP LTD
G06F16/55G06V10/762G06V20/70
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,579,184
App. No.
18/470,526
Granted
Mar 17, 2026
Kind
B2
Abstract

Methods, systems, and techniques for image categorization using a visual language model. A set of images is clustered into clusters respectively corresponding to differently categorized objects. Names are respectively assigned to the clusters, and image captions are respectively generated for the clusters using the names. The image captions and respective images represent image-text pairs. Those image-text pairs are input to the visual language model as context for a query. The query is then input to the visual language model. The query includes a request to categorize a query image of a class represented in the context. In response to the query, the visual language model performs an open-ended generative categorization of the query image.

Claims (45)

1 . A method for image categorization using a visual language model, the method comprising:

(a) clustering a set of unlabeled images into clusters respectively corresponding to differently categorized objects;

(b) respectively assigning names to the clusters, wherein the names are semantically unrelated to each other;

(c) respectively generating image captions for the clusters using the names, wherein the image captions and respective images comprise image-text pairs;

(d) inputting the image-text pairs to the visual language model as context for a query; and

(e) inputting the query to the visual language model,

wherein the query comprises a request to categorize a query image of a class represented in the context and wherein the visual language model performs an open-ended generative categorization of the query image in response to the query.

2 . The method of claim 1 , wherein the visual language model is a small-scale visual language model.

3 . The method of claim 2 , wherein the visual language model has fewer than 2 billion parameters.

4 . The method of claim 1 , wherein the visual language model has between approximately 124 million parameters and approximately 1.3 billion parameters.

5 . The method of claim 4 , wherein the visual language model has approximately 355 million parameters.

6 . The method of claim 1 , wherein the names comprise nonsense words, random numbers, or random nouns.

7 . The method of claim 6 , wherein the names are random nouns.

8 . The method of claim 1 , wherein generating the image captions is performed in a self-supervised manner.

9 . The method of claim 1 , wherein the clustering comprises:

(a) generating visual embeddings of the set of unlabeled images; and

(b) applying k-means clustering to the visual embeddings.

10 . The method of claim 1 , wherein generating the image captions comprises:

(a) respectively generating visual embeddings of centroids of the clusters using a vision encoder;

(b) generating word embeddings of the names of the clusters, wherein the word embeddings represent the names in a language model token-space; and

(c) using a cost function to match the centroid from each of the clusters to the names based on the visual embeddings of the centroids and the word embeddings, wherein the images of any one of the clusters share one of the names.

11 . The method of claim 1 , wherein the image captions for the different clusters comprise a text string shared by all the clusters.

12 . The method of claim 11 , wherein the image captions respectively comprise the text string and the names appended together, and wherein the query comprises the text string.

13 . The method of claim 1 , wherein the visual language model comprises a vision encoder and a language mode, wherein inputting the image-text pairs to the visual language model comprises part of training the visual language model, and wherein parameters of the vision encoder are frozen during the training and parameters of the language model are adjusted during the training.

14 . The method of claim 1 , wherein the context comprises a sequence of interleaved pairs of the images and corresponding ones of the captions.

15 . The method of claim 14 , wherein the context further comprises tokens denoting positions of the images and the captions in the sequence.

16 . The method of claim 1 , wherein the context comprises a plurality of i-way j-shot tasks based on the clusters, and wherein similarity between the clusters of each of the tasks varies over the tasks.

17 . The method of claim 1 , wherein the context comprises a plurality of i-way j-shot tasks based on the clusters, and wherein j varies over the tasks.

18 . A system for image categorization using a visual language model, the system comprising:

(a) at least one database having stored thereon a set of unlabeled images;

(b) at least one processor communicatively coupled to the at least one database; and

(c) at least one non-transitory computer readable medium communicatively coupled to the at least one processor, wherein the computer readable medium has stored thereon computer program code that is executable by at least one processor and that, when executed by the at least one processor, causes the at least one processor to perform a method comprising:

(i) clustering the set of unlabeled images into clusters respectively corresponding to differently categorized objects;

(ii) respectively assigning names to the clusters, wherein the names are semantically unrelated to each other;

(iii) respectively generating image captions for the clusters using the names, wherein the image captions and respective images comprise image-text pairs;

(iv) inputting the image-text pairs to the visual language model as context for a query; and

(v) inputting the query to the visual language model,

wherein the query comprises a request to categorize a query image of a class represented in the context and wherein the visual language model performs an open-ended generative categorization of the query image in response to the query.

19 . A non-transitory computer readable medium having stored thereon computer program code that is executable by at least one processor and that, when executed by the at least one processor, causes the at least one processor to perform a method for image categorization using a visual language model, the method comprising:

(a) clustering a set of unlabeled images into clusters respectively corresponding to differently categorized objects;

(b) respectively assigning names to the clusters, wherein the names are semantically unrelated to each other;

(c) respectively generating image captions for the clusters using the names, wherein the image captions and respective images comprise image-text pairs;

(d) inputting the image-text pairs to the visual language model as context for a query; and

(e) inputting the query to the visual language model,

wherein the query comprises a request to categorize a query image of a class represented in the context and wherein the visual language model performs an open-ended generative categorization of the query image in response to the query.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 28, 2025
From: INCEPTION INSTITUTE OF ARTIFICIAL INTELLIGENCE LTD
To: INCEPTION AI IP LTD
Reel/Frame 070659/0018 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 19, 2023
From: NAJDENKOSKA, IVONA; DERAKHSHANI, MOHAMMAD; ASANO, YUKI; SNOEK, CEES; WORRING, MARCEL
To: INCEPTION INSTITUTE OF ARTIFICIAL INTELLIGENCE, LTD.
Reel/Frame 065276/0190 →
Continuity (1)
Related Publication 20250094482A1 · Mar 20, 2025
References Cited (64)
Tsimpoukelli, Maria, et al. “Multimodal Few-Shot Learning with Frozen Language Models.” arXiv preprint arXiv:2106.13884 (2021). https://arxiv.org/abs/2106.13884 (Year: 2021). [cited by examiner]
Ji, Xu, João F. Henriques, and Andrea Vedaldi. “Invariant Information Clustering for Unsupervised Image Classification and Segmentation.” arXiv preprint arXiv: 1807.06653 (2019).https://arxiv.org/abs/1807.06653 (Year: 2… [cited by examiner]
Cao, Jize, et al. “Behind the scene: Revealing the secrets of pre-trained vision-and-language models.” European Conference on Computer Vision. Cham: Springer International Publishing, 2020.https://link.springer.com/chap… [cited by examiner]
Ucar, Talip, et al. “One-shot learning for language modelling.” arXiv preprint arXiv:2007.09679 (2020).https://arxiv.org/abs/2007.09679 (Year: 2020). [cited by examiner]
Xie, Junyuan, Ross Girshick, and Ali Farhadi. “Unsupervised Deep Embedding for Clustering Analysis.” arXiv preprint arXiv:1511.06335 (2015).https://arxiv.org/abs/1511.06335 (Year: 2015). [cited by examiner]
Noroozi, Mehdi, et al. “Boosting Self-Supervised Learning via Knowledge Transfer.” arXiv preprint arXiv:1805.00385 (2018).https://arxiv.org/abs/1805.00385 (Year: 2018). [cited by examiner]
Katz, Andrew, Umair Shakir, and Ben Chambers. “The utility of large language models and generative AI for education research.” arXiv preprint arXiv:2305.18125 (2023).https://arxiv.org/abs/2305.18125 (Year: 2023). [cited by examiner]
Han, Kai, et al. “What's in a Name? Beyond Class Indices for Image Recognition.” arXiv preprint arXiv:2304.02364 (2023).https://arxiv.org/abs/2304.02364 (Year: 2023). [cited by examiner]
Bossard, Lukas et al., “Food-101—Mining Discriminative Components with Random Forests,” 16 pages. [cited by applicant]
Bojanowski, Piotr et al., “Unsupervised Learning by Predicting Noise,” 10 pages, arXiv:1704.05310v1 [stat.ML], Apr. 18, 2017. [cited by applicant]
Asano, Yuki M. et al., “Self-Labelling Via Simultaneous Clustering and Representation Learning,” University of Oxford, 22 pages, ICLR 2020, arXiv: 1911.05371v3 [cs.CV], Feb. 19, 2020. [cited by applicant]
Asano, Yuki M. et al., “Labelling unlabelled videos from scratch with multi-modal self-supervision,” NeurIPS 2020, 15 pages, arXiv:2006.13662v3 [cs. CV], Feb. 28, 2021. [cited by applicant]
Alayrac, Jean-Baptiste et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” NEURLPS 2022, 54 pages, arXiv:2204.14198v2 [cs.CV], Nov. 15, 2022. [cited by applicant]
Abadi, Martin et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” 19 pages, arXiv:1603.04467v2 [cs. DC], Mar. 16, 2016. [cited by applicant]
“SeCAt: Self-Context Adaptation of Small Visual Language Models,” NeurIPS 2023, 13 pages. [cited by applicant]
Yang, Kevin et al., “Re: Generating Longer Stories With Recursive Reprompting and Revision,” UC Berkeley, 86 pages, arXiv:2210.06774v3 [cs.CL], Oct. 22, 2022. [cited by applicant]
Yang, Jianwei et al., “Joint Unsupervised Learning of Deep Representations and Image Clusters,” Virginia Tech, 19 pages, arXiv:1604.03628v3 [cs.CV], Jun. 20, 2016. [cited by applicant]
Yang, Antoine et al., “Zero-Shot Video Question Answering via Frozen Bidirectional Language Models,” PSL Research University, 25 pages, arXiv:2206.08155v2 [cs.CV], Oct. 10, 2022. [cited by applicant]
Xie, Junyuan et al., “Unsupervised Deep Embedding for Clustering Analysis,” 10 pages, arXiv:1511.06335v2 [cs.LG], May 24, 2016. [cited by applicant]
Xiao, Jianxiong et al., “SUN Database: Large-scale Scene Recognition from Abbey to Zoo,” Massachusetts Institute of Technology, Brown University, 8 pages. [cited by applicant]
Wolf, Thomas et al., “Transformers: State-of-the-Art Natural Language Processing,” Hugging Face, Brooklyn, USA, pp. 38-45, Nov. 16-20, 2020. [cited by applicant]
Wei, Jerry et al., “Larger Language Models Do In-Context Learning Differently,” 51 pages, arXiv:2303.03846v2 [cs.CL], Mar. 8, 2023. [cited by applicant]
Wei, Jason et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” 43 pages, arXiv:2201.11903v6 [cs.CL], Jan. 10, 2023. [cited by applicant]
Wei, Jason et al., “Emergent Abilities of Large Language Models,” Transactions on Machine Learning Research, Aug. 2022, 30 pages, arXiv:2206.07682v2 [cs.CL], Oct. 26, 2022. [cited by applicant]
Wei, Jason et al., “Finetuned Language Models Are Zero-Shot Learners,” ICLR 2022, 46 pages, arXiv:2109.01652v5 [cs.CL], Feb. 8, 2022. [cited by applicant]
Wang, Wenhui et al., “Image as a Foreign Language: BEIT Pretraining for All Vision and Vision-Language Tasks,” 18 pages, arXiv:2208.10442v2 [cs.CV], Aug. 31, 2022. [cited by applicant]
Welinder, Peter et al., “The MultidimensionalWisdom of Crowds,” 9 pages. [cited by applicant]
Gansbeke, Wouter Van et al., “SCAN: Learning to Classify Images without Labels,” 26 pages, arXiv:2005.12320v2 [cs.CV], Jul. 3, 2020. [cited by applicant]
Tsimpoukelli, Maria et al., “Multimodal Few-Shot Learning with Frozen Language Models,” 19 pages, arXiv:2106.13884v2 [cs.CV], Jul. 3, 2021. [cited by applicant]
Tay, Yi et al., “Transcending Scaling Laws with 0.1% Extra Compute,” 21 pages, arXiv:2210.11399v2 [cs.CL], Nov. 16, 2022. [cited by applicant]
Tan, Bowen et al., “Progressive Generation of Long Text with Pretrained Language Models,” 12 pages, arXiv:2006.15720v2 [cs.CL], Apr. 14, 2021. [cited by applicant]
Smith, Shaden et al., “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model,” 44 pages, arXiv:2201.11990v3 [cs.CL], Feb. 4, 2022. [cited by applicant]
Sharma, Piyush et al., “Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning,” pp. 2556-2565, Venice, California. [cited by applicant]
Saharia, Chitwan et al., “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding,” 46 pages, arXiv:2205.11487v1 [cs.CV], May 23, 2022. [cited by applicant]
Ramesh, Aditya et al., “Zero-Shot Text-to-Image Generation,” 20 pages, arXiv:2102.12092v2 [cs.CV], Feb. 26, 2021. [cited by applicant]
Rae, Jack W., et al., “Scaling Language Models: Methods, Analysis & Insights from Training Gopher,” DeepMind 2021, 120 pages, arXiv:2112.11446v2 [cs.CL], Jan. 21, 2022. [cited by applicant]
Radford, Alec et al., “Learning Transferable Visual Models From Natural Language Supervision,” 48 pages, arXiv:2103.00020v1 [cs.CV], Feb. 26, 2021. [cited by applicant]
Parkhi, Omkar M. et al., “Cats and Dogs,” IEEE Xplore, pp. 3498-3505, 2012. [cited by applicant]
Noroozi, Mehdi et al., “Boosting Self-Supervised Learning via Knowledge Transfer,” 9 pages, arXiv:1805.00385v1 [cs.CV], May 1, 2018. [cited by applicant]
Nilsback, Maria-Elena et al., “Automated flower classification over a large No. of classes,” Visual Geometry Group, Department of Engineering Science, University of Oxford, UK, 8 pages. [cited by applicant]
Najdenkoska, Ivona et al., “Meta Learning To Bridge Vision and Language Models for Multimodal Few-Shot Learning,” ICLR 2023, University of Amsterdam, NL, 13 pages, arXiv:2302.14794v1 [cs. CV], Feb. 28, 2023. [cited by applicant]
Mokady, Ron et al., “ClipCap: CLIP Prefix for Image Captioning,” The Blavatnik School of Computer Science, Tel Aviv University, 10 pages, arXiv:2111.09734v1 [cs.CV], Nov. 18, 2021. [cited by applicant]
Merullo, Jack et al., “Linearly Mapping From Image to Text Space,” ICRL 2023, 35 pages, arXiv:2209.15162v3 [cs.CL], Mar. 9, 2023. [cited by applicant]
Melas-Kyriazi, Luke et al., “Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization,” University of Oxford, UK, 25 pages. [cited by applicant]
Lu, Yao et al., “Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity,” 13 pages, arXiv:2104.08786v2 [cs.CL], Mar. 3, 2022. [cited by applicant]
Liu, Jiachang et al., “What Makes Good In-Context Examples for GPT-3?,” 12 pages, arXiv:2101.06804v1 [cs.CL], Jan. 17, 2021. [cited by applicant]
Li, Junnan et al., “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” 13 pages, arXiv:2301.12597v3 [cs.CV], Jun. 15, 2023. [cited by applicant]
Kuhn, H.W., “The Hungarian Method for the Assignment Problem,” Bryn Mawr College, 15 pages. [cited by applicant]
Koh, Jing Yu et al., “Grounding Language Models to Images for Multimodal Inputs and Outputs,” 18 pages, arXiv:2301.13823v4 [cs.CL], Jun. 13, 2023. [cited by applicant]
Kingma, Diederik P. et al., “Adam: a Method for Stochastic Optimization,” ICLR 2015, 15 pages, arXiv:1412.6980v9 [cs.LG], Jan. 30, 2017. [cited by applicant]
Johnson, Jeff et al., “Billion-scale similarity search with GPUs,” 12 pages, arXiv: 1702.08734v1 [cs.CV], Feb. 28, 2017. [cited by applicant]
Jia, Chao et al., “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,” 14 pages, arXiv:2102.05918v2 [cs.CV], Jun. 11, 2021. [cited by applicant]
Ji, Xu et al., “Invariant Information Clustering for Unsupervised Image Classification and Segmentation,” 10 pages, arXiv:1807.06653v4 [cs.CV], Aug. 22, 2019. [cited by applicant]
Jaegle, Andrew et al., “Perceiver IO: A General Architecture for Structured Inputs & Outputs,” ICLR 2022, 29 pages, arXiv:2107.14795v3 [cs.LG], Mar. 15, 2022. [cited by applicant]
Huang, Shaohan et al., “Language Is Not All You Need: Aligning Perception with Language Models,” 26 pages, arXiv:2302.14045v2 [cs.CL], Mar. 1, 2023. [cited by applicant]
Hoffmann, Jordan et al., “Training Compute-Optimal Large Language Models,” DeepMind, 2023, 36 pages, arXiv:2203.15556v1 [cs.CL], Mar. 29, 2022. [cited by applicant]
Hao, Yaru et al., “Language Models are General-Purpose Interfaces,” 32 pages, arXiv:2206.06336v1 [cs.CL], Jun. 13, 2022. [cited by applicant]
Gavrilyuk, Kirill et al., “Motion-Augmented Self-Training for Video Recognition at Smaller Scale,” 14 pages, arXiv:2105.01646v1 [cs.CV], May 4, 2021. [cited by applicant]
Gao, Leo et al., “The Pile: An 800GB Dataset of Diverse Text for Language Modeling,” EleutherAI, 39 pages, arXiv:2101.00027v1 [cs.CL], Dec. 31, 2020. [cited by applicant]
Dai, Zihang et al., “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context,” 20 pages, arXiv:1901.02860v3 [cs.LG], Jun. 2, 2019. [cited by applicant]
Chowdhery, Aakanksha et al., “PaLM: Scaling Language Modeling with Pathways,” Google Research, 87 pages, arXiv:2204.02311v5 [cs.CL], Oct. 5, 2022. [cited by applicant]
Chan, Stephanie C.Y. et al., “Data Distributional Properties Drive Emergent In-Context Learning in Transformers,” 18 pages, arXiv:2205.05055v6 [cs.LG], Nov. 17, 2022. [cited by applicant]
Caron, Mathilde et al., “Deep Clustering for Unsupervised Learning of Visual Features,” 30 pages, arXiv: 1807.05520v2 [cs.CV], Mar. 18, 2019. [cited by applicant]
Brown, Tom B. et al., “Language Models are Few-Shot Learners,” OpenAI, 75 pages, arXiv:2005.14165v4 [cs.CL], Jul. 22, 2020. [cited by applicant]