IP Library › Granted Patent US 12705872
Granted Patent B2
US 12705872 · App. 18/413,775 · Granted Aug 11, 2026

Systems and methods for contrastive fine-tuning of image-text machine learning models

Inventors: Ivan Batalov (Pittsburgh, PA); Chaithanya Kumar Mummadi (Pittsburgh, PA); Wan-Yi Lin (Wexford, PA)
Assignee: Robert Bosch GmbH
G06V10/82G06V20/41G06V20/56
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12705872
App. No.
18/413,775
Granted
Aug 11, 2026
Kind
B2
Abstract

A method for fine tuning a pre-trained machine learning model includes receiving, from a pre-trained machine learning model, at least one image embedding corresponding to first training data used to train the pre-trained machine learning model. The method also includes receiving, from the pre-trained machine learning model, at least one text embedding corresponding to the at least one image embedding. The method also includes generating at least one perturbation vector that includes the at least one image embedding, the at least one text embedding, a perturbation magnitude value, and a perturbation direction value. The method also includes generating second training data based on the at least one perturbation vector, and fine tuning the pre-trained machine learning model using the second training data.

Claims (41)

1 . A method for fine tuning a pre-trained machine learning model, the method comprising:

receiving, from a pre-trained machine learning model, at least one image embedding corresponding to first training data used to train the pre-trained machine learning model, wherein the at last one image embedding is from an image encoder;

receiving, from the pre-trained machine learning model, at least one text embedding corresponding to the at least one image embedding, wherein the at last one text embedding is from a text encoder, wherein the image encoder and text encoder are part of the pre-trained machine learning model;

generating at least one perturbation vector that includes each of the at least one image embedding, the at least one text embedding, a perturbation magnitude value, and a perturbation direction value;

generating second training data based on the at least one perturbation vector; and

fine tuning the pre-trained machine learning model using the second training data, wherein the fine tuning is accomplished in response to zero-shot classification without updating any weights of the pre-trained machine learning model.

2 . The method of claim 1 , further comprising determining the perturbation magnitude value and the perturbation direction value based on at least one other image embedding corresponding to the first training data having a lowest correlation with the at least one image embedding.

3 . The method of claim 1 , further comprising determining the perturbation magnitude value and the perturbation direction value based on at least one other text embedding corresponding to the first training data having a lowest correlation with the at least one image embedding.

4 . The method of claim 1 , further comprising determining the perturbation magnitude value and the perturbation direction value based on at least one other randomly selected image embedding corresponding to the first training data.

5 . The method of claim 1 , further comprising determining the perturbation magnitude value and the perturbation direction value based on at least one other randomly selected text embedding corresponding to the first training data.

6 . The method of claim 1 , further comprising determining the perturbation magnitude value and the perturbation direction value based on at least one other image embedding corresponding to the first training data having a highest correlation with the at least one image embedding.

7 . The method of claim 1 , further comprising determining the perturbation magnitude value and the perturbation direction value based on at least one other text embedding corresponding to the first training data having a highest correlation with the at least one image embedding.

8 . The method of claim 1 , wherein the pre-trained machine learning model is pre-trained using contrastive language-image pre-training.

9 . The method of claim 1 , wherein the pre-trained machine learning model, having been fine-tuned using the second training data, is configured to classify sensor data.

10 . The method of claim 9 , wherein the sensor data is associated with at least one sensor associated with at least one machine.

11 . The method of claim 10 , wherein the at least one machine includes a vehicle.

12 . A system for fine tuning a pre-trained machine learning model, the system comprising:

a processor; and

a memory including instructions that, when executed by the processor, cause the processor to:

receive, from a pre-trained machine learning model, at least one image embedding corresponding to first training data used to train the pre-trained machine learning model;

receive, from the pre-trained machine learning model, at least one text embedding corresponding to the at least one image embedding;

generate at least one perturbation vector that includes each of the at least one image embedding, the at least one text embedding, a perturbation magnitude value, and a perturbation direction value;

generate second training data based on the at least one perturbation vector; and

fine tune the pre-trained machine learning model using the second training data, wherein the fine tuning is accomplished in response to zero-shot classification without updating any weights of the pre-trained machine learning model.

13 . The system of claim 12 , wherein the instructions further cause the processor to determine the perturbation magnitude value and the perturbation direction value based on at least one other image embedding corresponding to the first training data having a lowest correlation with the at least one image embedding.

14 . The system of claim 12 , wherein the instructions further cause the processor to determine the perturbation magnitude value and the perturbation direction value based on at least one other text embedding corresponding to the first training data having a lowest correlation with the at least one image embedding.

15 . The system of claim 12 , wherein the instructions further cause the processor to determine the perturbation magnitude value and the perturbation direction value based on at least one other randomly selected image embedding corresponding to the first training data.

16 . The system of claim 12 , wherein the instructions further cause the processor to determine the perturbation magnitude value and the perturbation direction value based on at least one other randomly selected text embedding corresponding to the first training data.

17 . The system of claim 12 , wherein the instructions further cause the processor to determine the perturbation magnitude value and the perturbation direction value based on at least one other image embedding corresponding to the first training data having a highest correlation with the at least one image embedding.

18 . The system of claim 12 , wherein the instructions further cause the processor to determine the perturbation magnitude value and the perturbation direction value based on at least one other text embedding corresponding to the first training data having a highest correlation with the at least one image embedding.

19 . The system of claim 12 , wherein the pre-trained machine learning model is pre-trained using contrastive language-image pre-training.

20 . An apparatus for controlling a machine, the apparatus comprising:

a processor; and

a memory including instructions that, when executed by the processor, cause the processor to:

receive, from a pre-trained machine learning model, at least one image embedding corresponding to first training data used to train the pre-trained machine learning model;

receive, from the pre-trained machine learning model, at least one text embedding corresponding to the at least one image embedding;

generate at least one perturbation vector that includes all of the following: the at least one image embedding, the at least one text embedding, a perturbation magnitude value, and a perturbation direction value;

generate second training data based on the at least one perturbation vector;

fine tune the pre-trained machine learning model using the second training data, wherein the fine tuning is accomplished in response to zero-shot classification without updating any weights of the pre-trained machine learning model;

receive, from the fine-tuned pre-trained machine learning model, classified sensor data corresponding to at least one sensor of machine; and

selectively controlling the machine based on the classified sensor data.