Multi-modal retrieval using an intermediate noise state

1 . A method for data processing, comprising:

obtaining a text prompt;

generating, using an image generation model, a first intermediate noise state based on the text prompt;

retrieving a second intermediate noise state from a database by identifying a candidate text prompt in the database based on a similarity between the text prompt and the candidate text prompt, identifying a candidate noise state based on an association in the database between the identified candidate text prompt and the candidate noise state, and identifying the candidate noise state as the second intermediate noise state based on a similarity between the first intermediate noise state and the candidate noise state; and

generating, using the image generation model, a synthetic image based on the text prompt and the second intermediate noise state.

2 . The method of claim 1 , wherein generating the first intermediate noise state comprises:

obtaining an initial noise state; and

performing reverse diffusion to remove noise from the initial noise state.

3 . The method of claim 1 , wherein generating the synthetic image comprises:

performing reverse diffusion to remove noise from the second intermediate noise state.

4 . The method of claim 1 , further comprising:

obtaining a plurality of text prompts; and

generating a plurality of candidate intermediate noise states based on the plurality of text prompts, respectively.

5 . The method of claim 1 , wherein retrieving the second intermediate noise state comprises:

encoding the text prompt to obtain a text embedding;

selecting a plurality of candidate objects based on the text embedding, wherein each of the plurality of candidate objects includes a candidate noise state and a candidate text embedding that is compared to the text embedding; and

responsive to determining that the first intermediate noise state and the candidate noise state of a corresponding candidate object of the plurality of candidate objects satisfies a similarity condition, selecting the candidate noise state as the second intermediate noise state.

6 . The method of claim 5 , further comprising:

computing a similarity score between the text embedding and each of the plurality of candidate text embeddings, respectively, wherein the text embedding is compared based on the similarity score.

7 . The method of claim 5 , wherein:

the similarity condition relates to a similarity between the first intermediate noise state and a corresponding noise state included in a candidate object that includes the candidate noise state.

8 . The method of claim 1 , wherein retrieving the second intermediate noise state comprises:

computing a distance between the first intermediate noise state and one or more of a plurality of candidate intermediate noise states, wherein the second intermediate noise state is selected based on the distance.

9 . The method of claim 8 , wherein retrieving the second intermediate noise state further comprises:

selecting a nearest intermediate noise state based on the distance, wherein the second intermediate noise state comprises a subsequent noise state generated from a same text prompt as the nearest intermediate noise state.

10 . The method of claim 9 , wherein retrieving the second intermediate noise state further comprises:

determining the subsequent noise state based on a comparison to a subsequent noise state threshold.

11 . The method of claim 1 , wherein:

the first intermediate noise state corresponds to a first diffusion time step and the second intermediate noise state corresponds to a second diffusion time step that is greater than the first diffusion time step.

12 . A method for data processing, comprising:

obtaining a query comprising a

text embedding and a first intermediate noise state;

selecting a plurality of candidate objects based on the text embedding, wherein each of the plurality of candidate objects includes a candidate noise state and a candidate text embedding that is compared to the text embedding;

responsive to determining that the first intermediate noise state and the candidate noise state of a corresponding candidate object of the plurality of candidate objects satisfies a similarity condition, selecting the candidate noise state as a second intermediate noise state; and

providing a search result in response to the query based on the second intermediate noise state.

13 . The method of claim 12 , further comprising:

obtaining a plurality of text prompts;

encoding the plurality of text prompts to obtain a plurality of candidate text embeddings; and

generating, using an image generation model, a plurality of candidate intermediate noise states based on the plurality of candidate text embeddings, respectively, wherein the plurality of candidate objects includes the plurality of candidate text embeddings and the plurality of candidate intermediate noise states.

14 . The method of claim 12 , further comprising:

generating, using an image generation model, a synthetic image based on the text embedding and the candidate noise state in response to the search result, wherein the candidate noise state is included in the search result.

15 . The method of claim 12 , wherein:

the search result comprises an image generated based on the candidate text embedding.

16 . The method of claim 12 , wherein:

the search result comprises a text prompt corresponding to the candidate text embedding.

17 . The method of claim 12 , further comprising:

generating the text embedding based on an input text prompt; and

generating, using an image generation model, the first intermediate noise state based on the text embedding.

18 . A system for data processing, comprising:

at least one memory component;

at least one processor executing instructions stored in the at least one memory component; and

a query component configured to:

obtain a query comprising a text embedding and a first intermediate noise state;

select a plurality of candidate objects based on the text embedding, wherein each of the plurality of candidate objects includes a candidate noise state and a candidate text embedding that is compared to the text embedding;

responsive to determining that the first intermediate noise state and the candidate noise state of a corresponding candidate object of the plurality of candidate objects satisfies a similarity condition, select the candidate noise state as a second intermediate noise state; and

provide a search result in response to the query based on the second intermediate noise state.

19 . The system of claim 18 , the system further comprising:

a database storing the plurality of candidate objects.

20 . The system of claim 18 , the system further comprising:

an image generation model comprising image generation parameters stored in the at least one memory component, wherein the image generation model is trained to generate the first intermediate noise state and to generate a synthetic image based on search result.