Text editing of digital images
Digital image text editing techniques as implemented by an image processing system are described that support increased user interaction in the creation and editing of digital images through understanding a content creator's intent as expressed using text. In one example, a text user input is received by a text input module. The text user input describes a visual object and a visual attribute, in which the visual object specifies a visual context of the visual attribute. A feature representation generated by a text-to-feature system using a machine-learning module based on the text user input. The feature representation is passed to an image editing system to edit a digital object in a digital image, e.g., by applying a texture to an outline of the digital object within the digital image.
1 . A method for digital image text editing comprising:
generating, by a processing device, a color profile by a model based on a text user input, the model including a generator trained using training digital images and training text as inputs;
generating, by the processing device, a color gradient based on a feature representation that includes the color profile;
segmenting, by the processing device, a digital object from a digital image, the digital object identified in the text user input;
editing, by the processing device, the segmented digital object in the digital image based on the color gradient; and
presenting, by the processing device, the digital image including the edited segmented digital object.
2 . The method as described in claim 1 , further comprising collecting the training digital images based on an image search performed using the training text.
3 . The method as described in claim 1 , wherein the editing the segmented digital object includes applying a texture and one or more colors based on the color gradient to the segmented digital object by a texture machine learning model.
4 . The method as described in claim 3 , wherein the texture machine learning model is a texture generative adversarial network trained in a first stage to perform digital image editing and a second stage to fine tune the texture machine learning model to reproduce and propagate textures.
5 . The method as described in claim 1 , wherein the text user input describes a visual object and a visual attribute, the visual object specifying a visual context of the visual attribute.
6 . The method as described in claim 5 , wherein the visual object references a physical object and the visual attribute describes an appearance of the physical object.
7 . The method as described in claim 1 , wherein:
the generator is a generator of a generative adversarial network that receives as an input the training text as part of training; and
the generator is trained using a discriminator as part of the generative adversarial network, the discriminator is configured to receive as an input the training text, image features extracted from the training digital images using machine learning, and a candidate color profile generated by the generator as part of the training of the generator.
8 . The method as described in claim 1 , wherein the color profile includes a color histogram representation.
9 . The method as described in claim 1 , wherein the feature representation further includes one or more of a texture, contrast, lighting, or luminance based on the text user input.
10 . A system for digital image text editing comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
collecting training digital images based on an image search performed using training text;
generating a feature representation including a color profile by a model based on a text user input, the model including a generator trained using the training text and training feature representations generated from the training digital images;
editing a digital object in a digital image to have colors based on the feature representation; and
presenting the digital image with the edited digital object by the processing device.
11 . The system as described in claim 10 , wherein the text user input specifies a visual object that references a physical object and a visual attribute that describes an appearance of the physical object.
12 . The system as described in claim 10 , wherein collecting the training digital images includes identifying salient portions of the training digital images that are salient to the training text and the training feature representations are generated based on the salient portions of the training digital images.
13 . The system as described in claim 12 , wherein a convolutional neural network based classification model is used to identify the salient portions of the training digital images using visual attention to focus on parts of the training digital images.
14 . The system as described in claim 10 , wherein editing the digital object includes using an additional generator trained as part of a texture generative adversarial network to apply a texture and colors from the feature representation within an outline of the digital object as segmented within the digital image.
15 . The system as described in claim 10 , wherein:
the generator is a generator of a generative adversarial network that-receives as an input the training text as part of training; and
the generator is trained using a discriminator as part of the generative adversarial network, the discriminator is configured to receive as an input the training text, image features extracted from the training digital images using machine learning, and a candidate feature representation generated by the generator as part of the training of the generator.
16 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, the processing device performs operations for digital image text editing comprising:
generating a feature representation that includes a color profile by a model based on a text user input, the model including a generator trained using training digital images and training text;
segmenting a digital object from a digital image, the digital object identified in the text user input;
editing the segmented digital object in the digital image based on the feature representation; and
presenting the digital image including the edited segmented digital object by the processing device.
17 . The non-transitory computer-readable medium as described in claim 16 , wherein the training digital images are collected based on an image search performed using the training text.
18 . The non-transitory computer-readable medium as described in claim 16 , wherein editing the segmented digital object in the digital image includes applying a texture and one or more colors based on the color profile to the segmented digital object by a texture machine learning model.
19 . The non-transitory computer-readable medium as described in claim 16 , wherein the color profile includes a color histogram representation.
20 . The non-transitory computer-readable medium as described in claim 16 , wherein the generator is a generator of a generative adversarial network.