System and method for transfer semantic segmentation via learnable image prompting of foundation models

1 . A computer-implemented method, comprising:

receiving one or more fixed text prompts and one or more images, wherein the fixed text prompts are associated with the one or more images;

in response to utilizing the fixed text prompt and the one or more images at a foundation model associated with a machine-learning network, outputting an intermediate representation from generating a series of objects and a task, wherein the foundational model includes an encoder, a decoder, and a prediction head;

decoding, utilizing the decoder of the foundation model, the intermediate representation utilizing the decoder to generate a matrix associated with the task associated with the fixed text prompt and the image; and

in response to identifying a highest probability associated with the matrix utilizing label selection, outputting a final label associated with a visual based prediction task.

2 . The computer-implemented method of claim 1 , wherein the foundational model is a multimodal model.

3 . The computer-implemented method of claim 1 , wherein the method includes utilizing, at an image prompt network, the one or more images to generate a fixed-dimensional continuous latent vector.

4 . The computer-implemented method of claim 1 , wherein the decoder is utilizing a task-specific decoder.

5 . The computer-implemented method of claim 4 , wherein the method includes utilizing a task-specific encoder.

6 . The computer-implemented method of claim 1 , wherein the decoder is a task-specific decoder that includes a learnable functional map that utilizes an input as a vector from an intermediation latent representation space from an encoder.

7 . The computer-implemented method of claim 1 , wherein the visual based prediction task is not including in pretraining of the foundation model.

8 . The computer-implemented method of claim 1 , wherein the visual based prediction task is semantic segmentation and the final label includes a semantic segmentation image.

9 . A method, comprising:

receiving one or more fixed text prompts at a foundational model, wherein the foundational model includes an encoder, a decoder, and a prediction head;

receiving one or more images at a learnable image prompt network, wherein the fixed text prompts are associated with the one or more images;

generating a fixed-dimensional continuous latent vector at the learnable image prompt network utilizing the one or more images;

in response to utilizing the fixed text prompt and the fixed-dimensional continuous latent vector at the foundation model associated with a machine-learning network, outputting an intermediate representation from generating a series of objects and a task;

decoding the intermediate representation utilizing a decoder associated with the foundation model to generate a matrix associated with the task associated with the fixed text prompt and the image; and

in response to identifying a highest probability associated with the matrix utilizing label selection, outputting a final label associated with a visual based prediction task.

10 . The method of claim 9 , wherein the method further includes combining, utilizing a fusion model, representations from the foundation model with a task-specific representations from an encoder.

11 . The method of claim 9 , wherein the visual based prediction task is semantic segmentation and the final label includes a semantic segmentation image.

12 . The method of claim 9 , wherein the visual based prediction task is not including in pretraining of the foundation model.

13 . The method of claim 9 , wherein the task is a single task.

14 . The system of claim 9 , wherein the foundation model is configured to output an intermediate representation from one of a foundation model encoder, a foundation model decoder, a representation from an output of the head prediction head of the found model, or a combination of multiple foundation model prediction heads.

15 . A system, comprising:

a controller configured to:

receive one or more fixed text prompts and one or more images, wherein the fixed text prompts are associated with the one or more images;

in response to utilizing the fixed text prompt and the one or more images at a foundation model associated with a machine-learning network, output an intermediate representation from generating a series of objects and a task, wherein the foundational model includes an encoder a decoder, and a prediction head;

decode the intermediate representation utilizing the decoder of the foundation model to generate a matrix associated with the task associated with the fixed text prompt and the image; and

in response to identifying a highest probability associated with the matrix utilizing label selection, output a final label associated with a visual based prediction task.

16 . The system of claim 15 , wherein the visual based prediction task is semantic segmentation and the final label includes a semantic segmentation image.

17 . The system of claim 15 , wherein the visual based prediction task is not including in pretraining of the foundation model.

18 . The system of claim 15 , wherein the foundation model includes a language interface and visual interface.

19 . The system of claim 15 , wherein the system includes a task encoder configured to send one or more visual representations to the task decoder.

20 . The system of claim 15 , wherein the one or more images include a red green blue (RGB) image, sound image, video image, or radar image.