Personalized branding with prompt adaptation in large language models and visual language models
A data processing system implements techniques for generating personalized content using a brand kit. The system receives a natural language prompt to generate content in a design application on the client device of a user and analyzes the prompt to determine whether the user intends to apply a brand kit to the generated content. The system automatically generates a brand kit for the user if one does not already exist and applies the brand kit to content generated using one or more generative models to create personalized content. The system includes a prompt generation unit that generates a plurality of model-specific prompts to the one or more generative models to cause the one or more generative models to create the personalized content.
1 . A data processing system comprising:
a processor; and
a memory storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
receiving a natural language prompt input by a user of a design application, the natural language prompt requesting that the design application generate content described in the natural language prompt;
analyzing the natural language prompt using a first language model trained to output a prediction whether the user intended to generate personalized content using a brand kit comprising a set of electronic assets providing example of a visual identity of a brand associated with the user;
responsive to the first language model outputting a prediction that the user intended to generate personalized content using a brand kit, obtaining a brand kit associated with the user;
generating intermediate content based on the natural language prompt by generating a plurality of first model-specific prompts using a prompt construction layer, each model-specific prompt of the first model-specific prompts being provided as an input to a respective generative model of a plurality of generative models associated with the designer application to cause the respective generative model to generate at least one aspect of the intermediate content; and
customizing the intermediate content to create personalized content based on the brand kit by generating a plurality of second model-specific prompts using the prompt construction layer, each model-specific prompt of the second model-specific prompts being provided as an input to a respective generative model of the plurality of generative models to cause the respective generative model to generate at least one aspect of the personalized content based on the brand kit.
2 . The data processing system of claim 1 , wherein the assets included in the brand kit include one or more of a color palette, one or more fonts, and a logo.
3 . The data processing system of claim 1 , wherein receiving a natural language prompt input by the user of the design application further comprises receiving a sample image associated with the natural language prompt, and wherein generating the intermediate content based on the natural language prompt further comprises generating the intermediate content based on the natural language prompt and the samples image by generating a plurality of first model-specific prompts based on the natural language prompt and the sample image.
4 . The data processing system of claim 1 , wherein obtaining the brand kit associated with the user further comprises:
determining that the user is not associated with a brand kit; and
generating the brand kit based at least in part on the natural language prompt and information associated with the user.
5 . The data processing system of claim 4 , wherein generating the brand kit based further comprises generating the brand kit based at least in part on the natural language prompt, the information associated with the user, and one or both of features of a product or service identified in the natural language prompt and a target audience for the product or service.
6 . The data processing system of claim 1 , wherein obtaining the brand kit associated with the user further comprises:
determining that the user is associated with an organization that has an organization-specific brand kit; and
using the organization-specific brand kit as the brand kit.
7 . The data processing system of claim 1 , wherein the memory further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
performing brand-specific fact checks on the personalized content to identify aspects of the intermediate content which are factually incorrect; and
correcting the aspects of the intermediate content which are factually incorrect.
8 . The data processing system of claim 1 , wherein the brand kit includes additional information including one or more of target audience information, sales style information, and product features information.
9 . The data processing system of claim 1 , wherein the second language model is a large language model (LLM).
10 . The data processing system of claim 1 , wherein the memory further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
causing the designer application to present the personalized content on a user interface of the designer application.
11 . A method implemented in a data processing system for automatically generating personalized content, the method comprising:
receiving a natural language prompt input by a user of a design application, the natural language prompt requesting that the design application generate content described in the natural language prompt;
analyzing the natural language prompt using a first language model trained to output a prediction whether the user intended to generate personalized content using a brand kit comprising a set of electronic assets providing example of a visual identity of a brand associated with the user;
responsive to the first language model outputting a prediction that the user intended to generate personalized content using a brand kit, obtaining a brand kit associated with the user;
generating intermediate content based on the natural language prompt by generating a plurality of first model-specific prompts using a prompt construction layer, each model-specific prompt of the first model-specific prompts being provided as an input to a respective generative model of a plurality of generative models associated with the designer application to cause the respective generative model to generate at least one aspect of the intermediate content; and
customizing the intermediate content to create personalized content based on the brand kit by generating a plurality of second model-specific prompts using the prompt construction layer, each model-specific prompt of the second model-specific prompts being provided as an input to a respective generative model of the plurality of generative models to cause the respective generative model to generate at least one aspect of the personalized content based on the brand kit.
12 . The method of claim 11 , wherein the assets included in the brand kit include one or more of a color palette, one or more fonts, and a logo.
13 . The method of claim 11 , wherein receiving a natural language prompt input by the user of the design application further comprises receiving a sample image associated with the natural language prompt, and wherein generating the intermediate content based on the natural language prompt further comprises generating the intermediate content based on the natural language prompt and the samples image by generating a plurality of first model-specific prompts based on the natural language prompt and the sample image.
14 . The method of claim 11 , wherein obtaining the brand kit associated with the user further comprises:
determining that the user is not associated with a brand kit; and
generating the brand kit based at least in part on the natural language prompt and information associated with the user.
15 . The method of claim 14 , wherein generating the brand kit based further comprises generating the brand kit based at least in part on the natural language prompt, the information associated with the user, and one or both of features of a product or service identified in the natural language prompt and a target audience for the product or service.
16 . A data processing system comprising:
a processor; and
a memory storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
receiving a natural language prompt input by a user of a design application, the natural language prompt requesting that the design application generate content described in the natural language prompt;
analyzing the natural language prompt using a first language model trained to output a prediction whether the user intended to generate personalized content using a brand kit comprising a set of electronic assets providing example of a visual identity of a brand associated with the user;
responsive to the first language model outputting a prediction that the user intended to generate personalized content using a brand kit, generating a brand kit associated with the user;
generating intermediate content based on the natural language prompt by generating a plurality of first model-specific prompts using a prompt construction layer, each model-specific prompt of the first model-specific prompts being provided as an input to a respective generative model of a plurality of generative models associated with the designer application to cause the respective generative model to generate at least one aspect of the intermediate content; and
customizing the intermediate content to create personalized content based on the brand kit by generating a plurality of second model-specific prompts using the prompt construction layer, each model-specific prompt of the second model-specific prompts being provided as an input to a respective generative model of the plurality of generative models to cause the respective generative model to generate at least one aspect of the personalized content based on the brand kit.
17 . The data processing system of claim 16 , wherein the assets included in the brand kit include one or more of a color palette, one or more fonts, and a logo.
18 . The data processing system of claim 16 , wherein receiving a natural language prompt input by the user of the design application further comprises receiving a sample image associated with the natural language prompt, and wherein generating the intermediate content based on the natural language prompt further comprises generating the intermediate content based on the natural language prompt and the samples image by generating a plurality of first model-specific prompts based on the natural language prompt and the sample image.
19 . The data processing system of claim 16 , wherein generating the brand kit based further comprises generating the brand kit based at least in part on the natural language prompt, information associated with the user, and one or both of features of a product or service identified in the natural language prompt and a target audience for the product or service.
20 . The data processing system of claim 16 , wherein the memory further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
performing brand-specific fact checks on the personalized content to identify aspects of the intermediate content which are factually incorrect; and
correcting the aspects of the intermediate content which are factually incorrect.