Input preprocessing for generating images of synthetic products
A computer-implemented technique for preprocessing user input to generate images of synthetic products representing conceptual products includes receiving user input, including text and/or images, indicative of a conceptual product, and selecting a machine learning (ML) model from a set of models based on characteristics such as product category or maker. Image inputs can be converted to text-based descriptions, combined with text inputs, and configured as a prompt instruction for the selected ML model, incorporating constraints (e.g., material, production, cost) and user feedback. The ML model can generate one or more images of a synthetic product, which can be refined iteratively based on further feedback. The system supports recognition of known and unknown objects in images and can adapt prompt instructions accordingly. The generated images include synthetic products that are producible as physical products, enabling efficient conceptual product visualization and refinement.
1 . A computer-implemented method for preprocessing user input for a machine learning (ML) model configured to generate an image of a synthetic product that represents a conceptual product, the method comprising:
receiving user input including at least one of a text-based input or an image input indicative of a conceptual product;
selecting a particular ML model from among a set of ML models based on a characteristic of the conceptual product, wherein selecting the particular ML model from among the set of ML models comprises:
determining that the characteristic of the conceptual product is indicative of a particular category of products; and
identifying a particular maker of products based on the characteristic of the conceptual product,
wherein the particular ML model is configured to generate one or more images of a synthetic product that represents the conceptual product for the particular category of products by the particular maker of products and in accordance with the characteristic;
configuring, based on the user input, a text-based prompt instruction as an input to the selected ML model,
wherein the text-based prompt instruction is configured to direct an output of the selected ML model in accordance with the characteristic of the conceptual product; and
generating, based on the text-based prompt instruction input to the selected ML model, the one or more images including the synthetic product that represents the conceptual product having one or more physical attributes.
2 . The method of claim 1 , wherein the image input includes an object, the method further comprising:
converting the image input into a text-based representation including a description of the object; and
combining the text-based input and the text-based representation of the image input into a description of the conceptual product included in the text-based prompt instruction.
3 . The method of claim 1 , further comprising:
recognizing a known object included in the image input;
converting the image input into a text-based representation of an unknown object that corresponds to a modified version of the known object;
generating a description of the conceptual product based on the text-based representation,
wherein the text-based prompt instruction is generated based on the description of the conceptual product.
4 . The method of claim 1 , wherein generating the text-based prompt instruction further comprises:
incorporating constraints associated with creation of the physical product,
wherein the constraints include a material property, a production capability, and a cost constraint.
5 . The method of claim 1 , further comprising:
receiving user feedback of the one or more images of the conceptual product; and
refining the one or more images of the conceptual product based on the user feedback.
6 . The method of claim 1 , wherein to configure the text-based prompt instruction as input to the selected ML model further comprises:
incorporating user feedback into the text-based prompt instruction,
wherein the one or more images of the conceptual product are refined based on the user feedback incorporated into the text-based prompt instruction.
7 . The method of claim 1 , wherein the selected ML model is selected based on a physical characteristic of the conceptual product including:
a product category,
an intended use of the conceptual product, or
a target user of the conceptual product.
8 . The method of claim 1 , wherein configuring the text-based prompt instruction further comprises:
analyzing constraints associated with creation of a physical version of the conceptual product; and
configuring wording of the text-based prompt based on the constraints.
9 . A non-transitory, computer-readable storage medium comprising instructions recorded thereon, wherein the instructions, when executed by at least one data processor of a system, cause the system to:
receive user input including at least one of a text-based input or an image input indicative of a conceptual product;
select a particular ML model from among a set of ML models based on a characteristic of the conceptual product, wherein selection of the particular ML model from among the set of ML models comprises causing the system to:
determine that the characteristic of the conceptual product is indicative of a particular category of products; and
identify a particular maker of products based on the characteristic of the conceptual product, wherein the particular ML model is configured to generate one or more images of a synthetic product that represents the conceptual product for the particular category of products by the particular maker of products and in accordance with the characteristic;
configure, based on the user input, a text-based prompt instruction as an input to the selected ML model, wherein the text-based prompt instruction is configured to direct an output of the selected ML model in accordance with the characteristic of the conceptual product; and
generate, based on the text-based prompt instruction input to the selected ML model, the one or more images including the synthetic product that represents the conceptual product having one or more physical attributes.
10 . The non-transitory, computer-readable storage medium of claim 9 , wherein the image input includes an object, wherein the system is further caused to:
convert the image input into a text-based representation including a description of the object; and
combine the text-based input and the text-based representation of the image input into a description of the conceptual product included in the text-based prompt instruction.
11 . The non-transitory, computer-readable storage medium of claim 9 , wherein the system is further caused to:
recognize a known object included in the image input;
convert the image input into a text-based representation of an unknown object that corresponds to a modified version of the known object;
generate a description of the conceptual product based on the text-based representation, wherein the text-based prompt instruction is generated based on the description of the conceptual product.
12 . The non-transitory, computer-readable storage medium of claim 9 , wherein to generate the text-based prompt instruction comprises causing the system to:
incorporate constraints associated with creation of the physical product, wherein the constraints include a material property, a production capability, and a cost constraint.
13 . The non-transitory, computer-readable storage medium of claim 9 , wherein the system is further caused to:
receive user feedback of the one or more images of the conceptual product; and
refine the one or more images of the conceptual product based on the user feedback.
14 . The non-transitory, computer-readable storage medium of claim 9 , wherein the system is further caused to:
configure the text-based prompt instruction as input to the selected ML model by incorporating user feedback into the text-based prompt instruction, wherein the one or more images of the conceptual product are refined based on the user feedback incorporated into the text-based prompt instruction.
15 . The non-transitory, computer-readable storage medium of claim 9 , wherein the selected ML model is selected based on a physical characteristic of the conceptual product, the physical characteristic comprising:
a product category, an intended use of the conceptual product, or a target user of the conceptual product.
16 . The non-transitory, computer-readable storage medium of claim 9 , wherein configuring the text-based prompt instruction further comprises causing the system to:
analyze constraints associated with creation of a physical version of the conceptual product; and
configure wording of the text-based prompt based on the constraints.
17 . A system comprising:
at least one hardware processor; and
at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
receive user input including at least one of a text-based input or an image input indicative of a conceptual product;
select a particular ML model from among a set of ML models based on a characteristic of the conceptual product, wherein selection of the particular ML model from among the set of ML models comprises causing the system to:
determine that the characteristic of the conceptual product is indicative of a particular category of products; and
identify a particular maker of products based on the characteristic of the conceptual product, wherein the particular ML model is configured to generate one or more images of a synthetic product that represents the conceptual product for the particular category of products by the particular maker of products and in accordance with the characteristic;
configure, based on the user input, a text-based prompt instruction as an input to the selected ML model, wherein the text-based prompt instruction is configured to direct an output of the selected ML model in accordance with the characteristic of the conceptual product; and
generate, based on the text-based prompt instruction input to the selected ML model, the one or more images including the synthetic product that represents the conceptual product having one or more physical attributes.
18 . The system of claim 17 , wherein the image input includes an object, and wherein the system is further caused to:
convert the image input into a text-based representation including a description of the object; and
combine the text-based input and the text-based representation of the image input into a description of the conceptual product included in the text-based prompt instruction.
19 . The system of claim 17 , wherein the system is further caused to:
recognize a known object included in the image input;
convert the image input into a text-based representation of an unknown object that corresponds to a modified version of the known object;
generate a description of the conceptual product based on the text-based representation; and
generate the text-based prompt instruction based on the description of the conceptual product.
20 . The system of claim 17 , wherein to generate the text-based prompt instruction comprises causing the system to:
incorporate constraints associated with creation of the physical product, wherein the constraints include a material property, a production capability, and a cost constraint.
21 . The system of claim 17 , wherein the system is further caused to:
receive user feedback of the one or more images of the conceptual product; and
refine the one or more images of the conceptual product based on the user feedback.
22 . The system of claim 17 , wherein the system is further caused to:
configure the text-based prompt instruction as input to the selected ML model by incorporating user feedback into the text-based prompt instruction, wherein the one or more images of the conceptual product are refined based on the user feedback incorporated into the text-based prompt instruction.
23 . The system of claim 17 , wherein the selected ML model is selected based on a physical characteristic of the conceptual product, the physical characteristic comprising:
a product category, an intended use of the conceptual product, or a target user of the conceptual product.
24 . The system of claim 17 , wherein to configure the text-based prompt instruction comprises causing the system to:
analyze constraints associated with creation of a physical version of the conceptual product; and
configure wording of the text-based prompt based on the constraints.