IP Library › Granted Patent US 12,462,552
Granted Patent B2
US 12,462,552 · App. 18/217,248 · Granted Nov 4, 2025

System and method for prompt searching

Inventors: Devin T. Willmott (Pittsburgh, PA); Victor Abayomi Akinwande (Pittsburgh, PA); Yiding Jiang (Pittsburgh, PA); Dylan Jiang Sam (Pittsburgh, PA); Jeremy Kolter (Pittsburgh, PA)
Assignees: Robert Bosch GmbH; Carnegie Mellon University
G06V10/86G06F40/284G06V10/761G06V10/774G06V10/82G06V20/70
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,462,552
App. No.
18/217,248
Granted
Nov 4, 2025
Kind
B2
Abstract

A computer-implemented method that includes receiving a plurality of input images, generating a visual matrix utilizing the plurality of images and an image encoder, wherein the visual matrix includes a list of encoded images, receiving a plurality of text prompts, selecting a text prompt from the plurality of text prompts, send the first one of the text prompts to a language model to generate a candidate list of tokens, selecting tokens, converting the text prompts into updated text prompts via appending the tokens, generating a text matrix utilizing the text prompt and text encoder, and utilizing numerical values assigned at an image-text similarity matrix, determining a score associated with the image-text similarity matrix; and evaluating a criteria and outputting a final token to the updated text prompt in response to identifying a highest score associated with the final token after evaluating each of the plurality of text prompts.

Claims (56)

1 . A computer-implemented method for a pre-trained machine-learning network, the computer-implemented method comprising the following steps:

(i) receiving a plurality of input images including a plurality of pixels;

(ii) generating a visual matrix utilizing the plurality of input images and an image encoder of the machine learning network, wherein the visual matrix includes a list of encoded images;

(iii) receiving a plurality of text prompts;

(iv) selecting a first one of the text prompts from the plurality of text prompts;

(v) send the first one of the text prompts to a large language model (LLM) to generate a candidate list of tokens, wherein the candidate list of tokens is generated by selecting a subset from every token associated with the first one of the text prompts, wherein the subset includes highest-probable tokens associated with the first one of the text prompts, wherein the highest-probable tokens are calculated in response to output of the LLM;

(vi) selecting one or more tokens from the candidate list;

(vii) converting the one of the text prompts into updated text prompts via appending the one or more selected tokens associated to the plurality of text prompts;

(viii) generating a text matrix utilizing both (1) the updated text prompt that include one or more tokens and (2) a text encoder of the machine learning network, wherein the text matrix includes a list of encoded visual descriptors that includes the updated text prompt with one or more tokens;

(ix) multiplying the text matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns a numerical value indicating similarities between each of encoded visual descriptors and each of the encoded images, wherein similarities are indicated by entries of the image-text similarity matrix having numerical values;

(x) utilizing the numerical values assigned at the image-text similarity matrix, determining a score associated with the image-text similarity matrix; and

(xi) when the score falls below a threshold, repeating steps (vi-xi) for a second token for the first one of the text prompts, and when the score exceeds the threshold, adding the one or more tokens to the updated text prompts and repeating steps (iv-xi) for a remainder of each of the plurality of text prompts; and

(xii) outputting a final token to the updated text prompt in response to identifying a highest score associated with the final token after evaluating each of the plurality of text prompts.

2 . The method of claim 1 , wherein determining a score associated with the image-text similarity matrix includes utilizing KL-divergence.

3 . The method of claim 1 , wherein the image-text similarity matrix is a matrix of one-hot encodings of each image.

4 . The method of claim 1 , wherein the text encoder is a contrastive language-image pre-training (CLIP) text encoder and the image encoder is a CLIP image encoder.

5 . The method of claim 1 , wherein parameters associated with the image encoder and text encoder are not modified.

6 . The method of claim 1 , wherein the threshold is a number of iterations.

7 . The method of claim 1 , wherein the threshold includes a length associated with the plurality of text prompts.

8 . The method of claim 1 , wherein the threshold is a convergence threshold.

9 . The method of claim 1 , wherein one of the text prompts is associated with a class representative of the one of the plurality of input images.

10 . The method of claim 1 , wherein the one or more tokens include a word.

11 . A system, comprising:

a processor programmed to:

(i) receive a plurality of input images indicative of radar, sonar, video, picture, sound, or LiDar information;

(ii) generate a visual matrix utilizing the plurality of input images and an image encoder of the machine learning network, wherein the visual matrix includes a list of encoded images;

(iii) receive a plurality of text prompts;

(iv) select a first one of the text prompts from the plurality of text prompts;

(v) send the first one of the text prompts to a large language model (LLM) to generate a candidate list of tokens, wherein the candidate list of tokens is generated by selecting a subset of tokens from every token associated with the first one of the text prompts, wherein the subset includes highest-probable tokens associated with the first one of the text prompts, wherein the highest-probable tokens are calculated in response to output of the LLM;

(vi) select one or more tokens from the candidate list;

(vii) convert the one of the text prompts into updated text prompts via appending the one or more selected tokens associated to the plurality of text prompts;

(viii) generate a text matrix utilizing both (1) the updated text prompt that include one or more tokens and (2) a text encoder of the machine learning network, wherein the text matrix includes a list of encoded visual descriptors that includes the updated text prompt with one or more tokens;

(ix) multiply the text matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns a numerical value indicating similarities between each of encoded visual descriptors and each of the encoded images, wherein similarities are indicated by entries of the image-text similarity matrix having numerical values;

(x) utilizing the numerical values assigned at the image-text similarity matrix, determine a score associated with the image-text similarity matrix;

(xi) when the score falls below a threshold, repeating steps (vi-xi) for a second token for the first one of the text prompts, and when the score exceeds the threshold, adding the one or more tokens to the updated text prompts and repeating steps (iv-xi) for a remainder of each of the plurality of text prompts; and

(xii) output a final token to the updated text prompt in response to identifying a highest score associated with the final token after evaluating each of the plurality of text prompts.

12 . The system of claim 11 , wherein the one or more tokens include a word.

13 . The system of claim 11 , wherein adding the one or more tokens to the updated text prompts include replacing a previous token with a lower score.

14 . The system of claim 11 , wherein parameters associated with the image encoder and text encoder stay fixed.

15 . A computer-implemented method, comprising:

(i) receiving a plurality of input images;

(ii) generating a visual matrix utilizing the plurality of input images and an image encoder of the machine learning network, wherein the visual matrix includes a list of encoded images;

(iii) receiving a plurality of text prompts;

(iv) selecting a first one of the text prompts from the plurality of text prompts;

(v) sending the first one of the text prompts to a language model (LM) to generate a candidate list of tokens, where in the candidate list of tokens is a subset smaller than all of the tokens associated with the first one of the text prompts;

(vi) selecting one or more tokens from the candidate list;

(vii) converting the one of the text prompts into updated text prompts via appending the one or more selected tokens associated to the plurality of text prompts;

(viii) generating a text matrix utilizing both (1) the updated text prompt that include one or more tokens and (2) a text encoder of the machine learning network, wherein the text matrix includes a list of encoded visual descriptors that includes the updated text prompt with one or more tokens;

(ix) multiplying the text matrix and the visual matrix to generate an image-text similarity matrix, wherein the image-text similarity matrix assigns a numerical value indicating similarities between each of encoded visual descriptors and each of the encoded images, wherein similarities are indicated by entries of the image-text similarity matrix having numerical values;

(x) utilizing the numerical values assigned at the image-text similarity matrix, determining a score associated with the image-text similarity matrix; and

(xi) when the score falls below a threshold, repeating steps (vi-xi) for a second token for the first one of the text prompts, and when the score exceeds the threshold, adding the one or more tokens to the updated text prompts and repeating steps (iv-xi) for a remainder of each of the plurality of text prompts.

16 . The method of claim 15 , wherein adding the one or more tokens to the updated text prompts includes replaced a previous token.

17 . The method of claim 15 , wherein the candidate list of tokens is generated by selecting a subset of tokens from every token associated with the first one of the text prompts, wherein the subset includes highest-probable tokens associated with the first one of the text prompts, wherein the highest-probable tokens are calculated in response to output of the LLM.

18 . The method of claim 15 , wherein the method includes outputting a final token to the updated text prompt in response to identifying a highest score associated with the final token after evaluating each of the plurality of text prompts.

19 . The method of claim 15 , wherein the one or more tokens include a word.

20 . The method of claim 15 , wherein adding the one or more tokens to the updated text prompts include replacing a previous token with a lower score.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 23, 2023
From: WILLMOTT, DEVIN T.; AKINWANDE, VICTOR ABAYOMI; SAM, DYLAN JIANG; KOLTER, JEREMY
To: ROBERT BOSCH GMBH
Reel/Frame 064682/0952 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 23, 2023
From: JIANG, YIDING
To: CARNEGIE MELLON UNIVERSITY
Reel/Frame 064682/0979 →
Continuity (1)
Related Publication 20250005918A1 · Jan 2, 2025
References Cited (37)
US 9094615B2 · Aman · 2015 [cited by examiner]
US 20230122874A1 · Gupta · 2023 [cited by examiner]
WO WO2024158853A1 · 2024 [cited by examiner]
Gintare Karolina Dziugaite et al., “Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data”, arXiv:1703.11008v2 [cs.LG] Oct. 19, 2017, 14 Pages. [cited by applicant]
Alec Radford et al., “Learning Transferable Visual Models From Natural Language Supervision.” arXiv:2103.00020v1 [cs.CV] Feb. 26, 2021, 48 Pages. [cited by applicant]
Alexey Dosovitskiy et al., “An Image is Worth 16X16 Words: Transformers for Image Recognition at Scale.” arXiv:2010.11929v2 [cs.CV] Jun. 3, 2021, 22 Pages. [cited by applicant]
Behnam Neyshabur et al., “Norm-Based Capacity Control in Neural Networks.” arXiv:1503.00036v2 [cs.LG] Apr. 14, 2015, 29 Pages. [cited by applicant]
Behnam Neyshabur et al., “In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning.” arXiv:1412.6614v4 [cs.LG] Apr. 16, 2015, 9 Pages. [cited by applicant]
Behnam Neyshabur et al., “A Pac-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks.” arXiv:1707.09564v2 [cs.LG] Feb. 23, 2018, 9 Pages. [cited by applicant]
Behnam Neyshabur et al., “Exploring Generalization in Deep Learning.” 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 10 Pages. [cited by applicant]
Brian Lester et al., “The Power of Scale for Parameter-Efficient Prompt Tuning.” Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, Nov. 7-11, 2021, 15 Pages. [cited by applicant]
Chao Jia et al., “Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision.” Proceedings of the 38 th International Conference on Machine Learning, PMLR 139, 2021, 13 Pages. [cited by applicant]
David A. McAllester, “PAC-Bayesian Model Averaging.” COLT 1999, pp. 164-170. [cited by applicant]
Pengfei Liu et al., “Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.” arXiv:2107.13586v1 [cs.CL] Jul. 28, 2021, 46 Pages. [cited by applicant]
Gintare Karolina Dziugaite et al., “In Search of Robust Measures of Generalization.” 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, 11 Pages. [cited by applicant]
Gintare Karolina Dziugaite et al., “On the role of data in PAC-Bayes bounds.” Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) 2021, San Diego, California, USA. PMLR: … [cited by applicant]
Gordon Christie et al., “Functional Map of the World.” CVPR 2018, pp. 6172-6180. [cited by applicant]
Hemanth Venkateswara et al., “Deep Hashing Network for Unsupervised Domain Adaptation.” CVPR 2017, pp. 5018-5027. [cited by applicant]
Hugo Touvron et al., “LLaMA: Open and Efficient Foundation Language Models.” arXiv:2302.13971v1 [cs.CL] Feb. 27, 2023, 27 Pages. [cited by applicant]
Jared Kaplan et al., “Scaling Laws for Neural Language Models.” arXiv:2001.08361v1 [cs.LG] Jan. 23, 2020, 30 Pages. [cited by applicant]
John Langford et al., “(Not) Bounding the True Error.” Part of Advances in Neural Information Processing Systems 14 (NIPS 2001), 8 Pages. [cited by applicant]
Maria Perez-Ortiz et al., “Tighter Risk Certificates for Neural Networks.” Journal of Machine Learning Research 22 (2021) 1-40 Submitted Aug. 2020; Revised May 2021; Published Aug. 2021, 40 Pages. [cited by applicant]
Paul Viallard et al., “A General Framework for the Disintegration of Pac-Bayesian Bounds.” arXiv:2102.08649v2 [stat.ML] Oct. 11, 2021, 23 Pages. [cited by applicant]
Peter L. Bartlett et al., “Spectrally-normalized margin bounds for neural networks.” arXiv:1706.08498v2 [cs.LG] Dec. 5, 2017, 24 Pages. [cited by applicant]
Sanae Lotfi et al., “PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization.” 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 15 Pages. [cited by applicant]
Shai Shalev-Shwartz et al., Understanding Machine Learning: From Theory to Algorithms, 2014, 449 Pages. [cited by applicant]
Stephen H. Bach et al., “PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts.” Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics System D… [cited by applicant]
Teven Le Scao et al., “How Many Data Points is a Prompt Worth?” Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2627-26… [cited by applicant]
Tianyu Gao et al., “Making Pre-trained Language Models Better Few-shot Learners.” Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Na… [cited by applicant]
Tom B. Brown et al., “Language Models are Few-Shot Learners.” 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, 25 Pages. [cited by applicant]
V. N. Vapnik et al., “On the uniform convergence of relative frequencies of events to their probabilities.” Theory of Probability and its applications 171, vol. XVI, No. 2, pp. 264-280. [cited by applicant]
Vapnik, “Principles of Risk Minimization for Learning Theory.” NIPS 1991, pp. 831-838. [cited by applicant]
Vaishnavh Nagarajan et al., “Generalization in Deep Networks: The Role of Distance from Initialization.” arXiv:1901.01672v2 [cs.LG] Jan. 13, 2019, 15 Pages. [cited by applicant]
Yoad Tewel et al., “ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic.” arXiv:2111.14447v2 [cs.CV] Mar. 31, 2022, 28 Pages. [cited by applicant]
Vaishnavh Nagarajan et al., “Uniform convergence may be unable to explain generalization in deep learning.” 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, 12 Pages. [cited by applicant]
Xiang Lisa Li et al., “Prefix-Tuning: Optimizing Continuous Prompts for Generation.” Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on… [cited by applicant]
Yiding Jiang et al., “Fantastic Generalization Measures and Where to Find Them.” arXiv:1912.02178v1 [cs.LG] Dec. 4, 2019, 33 Pages. [cited by applicant]