Systems and methods for generating a queryable image from text
In some implementations, the techniques described herein relate to a method including (i) receiving, by a processor, user input describing at least one parameter for a query image, (ii) generating, via a generative machine learning model executed by the processor, the query image based at least in part on the user input describing the at least one parameter for the query image, (iii) providing, by the processor, the query image as input to an image-based search algorithm, and (iv) returning a result received by the processor from the image-based search algorithm.
1 . A method comprising:
receiving, by a processor, user input describing at least one parameter for a query image;
generating, via a prompt generation module, a structured prompt by transforming the user input into a predetermined format that includes the at least one parameter and automatically generated context-dependent keywords not present in the user input, the context-dependent keywords selected from a library of predetermined keywords based on the at least one parameter to enhance image generation quality;
generating, via a generative machine learning model executed by the processor, an initial iteration of the query image based at least in part on the structured prompt;
transmitting the initial iteration of the query image to a client device for display to a user;
displaying the initial iteration of the query image via an interactive interface that enables direct visual modification of the initial iteration;
receiving user input specifying a change to the initial iteration of the query image;
in response to receiving the user input specifying the change, generating, via the generative machine learning model, a new iteration of the query image based on the direct visual modification and the user input specifying the change while maintaining unmodified aspects of the initial iteration;
providing, by the processor, a final iteration of the query image as input to an image-based search algorithm; and
returning a result received by the processor from the image-based search algorithm.
2 . The method of claim 1 , wherein receiving the user input comprises: providing an image creation interface to a user; and receiving, via the image creation interface, an image created by the user as the user input.
3 . The method of claim 1 , wherein the user input comprises a text description of the at least one parameter.
4 . The method of claim 1 , wherein generating, via the generative machine learning model executed by the processor, the query image based at least in part on the user input describing the at least one parameter for the query image, comprises: providing the at least one parameter to a prompt generation module; receiving, from the prompt generation module, a prompt for the generative machine learning model that is based on the at least one parameter and comprises a predetermined structure; and providing the prompt to the generative machine learning model.
5 . A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
receiving, by a processor, user input describing at least one parameter for a query image;
generating, via a prompt generation module, a structured prompt by transforming the user input into a predetermined format that includes the at least one parameter and automatically generated context-dependent keywords not present in the user input, the context-dependent keywords selected from a library of predetermined keywords based on the at least one parameter to enhance image generation quality;
generating, via a generative machine learning model executed by the processor, an initial iteration of the query image based at least in part on the structured prompt;
transmitting the initial iteration of the query image to a client device for display to a user;
displaying the initial iteration of the query image via an interactive interface that enables direct visual modification of the initial iteration;
receiving user input specifying a change to the initial iteration of the query image;
in response to receiving the user input specifying the change, generating, via the generative machine learning model, a new iteration of the query image based on the direct visual modification and the user input specifying the change while maintaining unmodified aspects of the initial iteration;
providing, by the processor, a final iteration of the query image as input to an image-based search algorithm; and
returning a result received by the processor from the image-based search algorithm.
6 . The non-transitory computer-readable storage medium of claim 5 , wherein receiving the user input comprises: providing an image creation interface to a user; and receiving, via the image creation interface, an image created by the user as the user input.
7 . The non-transitory computer-readable storage medium of claim 5 , wherein the user input comprises a text description of the at least one parameter.
8 . The non-transitory computer-readable storage medium of claim 5 , wherein generating, via the generative machine learning model executed by the processor, the query image based at least in part on the user input describing the at least one parameter for the query image, comprises: providing the at least one parameter to a prompt generation module; receiving, from the prompt generation module, a prompt for the generative machine learning model that is based on the at least one parameter and comprises a predetermined structure; and providing the prompt to the generative machine learning model.
9 . A device comprising: a processor; and a storage medium for tangibly storing thereon logic for execution by the processor, the logic comprising instructions for:
receiving, by a processor, user input describing at least one parameter for a query image;
generating, via a prompt generation module, a structured prompt by transforming the user input into a predetermined format that includes the at least one parameter and automatically generated context-dependent keywords not present in the user input, the context-dependent keywords selected from a library of predetermined keywords based on the at least one parameter to enhance image generation quality;
generating, via a generative machine learning model executed by the processor, an initial iteration of the query image based at least in part on the structured prompt;
transmitting the initial iteration of the query image to a client device for display to a user;
displaying the initial iteration of the query image via an interactive interface that enables direct visual modification of the initial iteration;
receiving user input specifying a change to the initial iteration of the query image;
in response to receiving the user input specifying the change, generating, via the generative machine learning model, a new iteration of the query image based on the direct visual modification and the user input specifying the change while maintaining unmodified aspects of the initial iteration;
providing, by the processor, a final iteration of the query image as input to an image-based search algorithm; and
returning a result received by the processor from the image-based search algorithm.
10 . The device of claim 9 , wherein receiving the user input comprises: providing an image creation interface to a user; and receiving, via the image creation interface, an image created by the user as the user input.
11 . The device of claim 9 , wherein the user input comprises a text description of the at least one parameter.
12 . The method of claim 1 , wherein the structured prompt includes one or more negative clauses.
13 . The method of claim 1 , wherein generating the structured prompt comprises repeating key elements from the user input in multiple comma-separated phrases.
14 . The method of claim 1 , wherein the context-dependent keywords are selected from a library of predetermined keywords based on the at least one parameter.
15 . The method of claim 1 , wherein the prompt generation module comprises a large language model trained to transform natural language descriptions into generative model prompts.
16 . The method of claim 1 , wherein the interactive interface includes an inpainting tool for modifying portions of the initial iteration of the query image.
17 . The method of claim 1 , wherein the interactive interface includes tools for cutting, copying, and pasting elements of the initial iteration of the query image.
18 . The method of claim 1 , wherein the interactive interface includes tools for adjusting brightness, contrast, and color settings of the initial iteration.
19 . The method of claim 1 , further comprising embedding constraints in the structured prompt to control output image characteristics of the generative machine learning model.
20 . The method of claim 1 , wherein the user input comprises a combination of text input and an image created using an image creation interface provided to a user.