Utilizing a diffusion neural network for mask aware image and typography editing
The present disclosure relates to systems, methods, and non-transitory computer readable media for utilizing a diffusion neural network for mask aware image and typography editing. For example, in one or more embodiments the disclosed systems utilize a text-image encoder to generate a base image embedding from a base digital image. Moreover, the disclosed systems generate a mask-segmented image by combining a shape mask with the base digital image. In one or more implementations, the disclosed systems utilize noising steps of a diffusion noising model to generate a mask-segmented image noise map from the mask-segmented image. Furthermore, the disclosed systems utilize a diffusion neural network to create a stylized image corresponding to the shape mask from the base image embedding and the mask-segmented image noise map.
1 . A computer-implemented method comprising:
generating, from a base digital image utilizing a trained text-image encoder, a base image embedding comprising a vector representation of the base digital image;
generating a mask-segmented image by segmenting the base digital image utilizing a shape mask;
generating, utilizing noising steps of a diffusion noising model, a mask-segmented image noise map from the mask-segmented image; and
generating, utilizing a diffusion neural network, a stylized image corresponding to the shape mask in a style of the base digital image by denoising the mask-segmented image noise map utilizing denoising layers of the diffusion neural network conditioned on the base image embedding.
2 . The computer-implemented method of claim 1 , further comprising generating the shape mask from a typography character and wherein creating the stylized image comprises generating a stylized typography character that reflects the base digital image utilizing the diffusion neural network.
3 . The computer-implemented method of claim 1 , wherein generating, utilizing the noising steps of the diffusion noising model, the mask-segmented image noise map comprises utilizing reverse diffusion steps of a reverse diffusion neural network to generate the mask-segmented image noise map from the mask-segmented image.
4 . The computer-implemented method of claim 1 , wherein creating the stylized image comprises generating an intermediate noise map from the base image embedding utilizing a denoising step of the diffusion neural network conditioned on the base image embedding.
5 . The computer-implemented method of claim 4 , wherein creating the stylized image comprises generating the stylized image from the intermediate noise map utilizing additional denoising steps of the diffusion neural network conditioned on the base image embedding.
6 . The computer-implemented method of claim 1 , further comprising generating a stylized animation by generating a plurality of stylized images utilizing a first structural number of steps for a first frame of the stylized animation and a second structural number of steps for a second frame of the stylized animation.
7 . The computer-implemented method of claim 1 , wherein generating the mask-segmented image noise map comprises:
selecting a structural number of steps based on user interaction with a client device; and
utilizing the structural number of steps of the diffusion noising model to generate the mask-segmented image noise map from the mask-segmented image.
8 . The computer-implemented method of claim 7 , wherein creating the stylized image comprises utilizing the structural number of steps of the diffusion neural network to create the stylized image from the base image embedding and the mask-segmented image noise map.
9 . The computer-implemented method of claim 1 , further comprising generating the base image embedding by:
receiving, from a client device, an edit text; and
generating the base digital image from the edit text.
10 . A system comprising:
one or more memory devices comprising a base digital image, a typography character mask, a trained text-image encoder, and a diffusion neural network; and
one or more processors configured to cause the system to:
generate, from the base digital image, a base image embedding comprising a vector representation of the base digital image;
generate, utilizing noising steps of a diffusion noising model, a mask-segmented image noise map from a mask-segmented image generated by segmenting the base digital image utilizing the typography character mask; and
generate, utilizing the diffusion neural network, a stylized typography character in a style of the base digital image from the mask-segmented image noise map by conditioning denoising steps of the diffusion neural network utilizing the base image embedding.
11 . The system of claim 10 , wherein the one or more processors are further configured to cause the system to generate the base image embedding from the base digital image utilizing the trained text-image encoder.
12 . The system of claim 10 , wherein the one or more processors are further configured to cause the system to generate an additional stylized typography character utilizing an additional structural number of noising steps.
13 . The system of claim 10 , wherein the one or more processors are further configured to cause the system to:
generate a mask-segmented image from the typography character mask and the base digital image; and
generate the mask-segmented image noise map by utilizing reverse diffusion steps of a reverse diffusion neural network to generate the mask-segmented image noise map from the mask-segmented image.
14 . The system of claim 10 , wherein the one or more processors are further configured to cause the system to:
select a structural number of steps; and
create the stylized typography character from the base image embedding by utilizing the structural number of steps of the diffusion neural network.
15 . A non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
generating, from a base digital image, a base image embedding comprising a vector representation of the base digital image;
determining a structural number of steps based on user interaction with a user interface of a client device;
generating, utilizing the structural number of steps of a diffusion noising model, a mask-segmented image noise map from a mask-segmented image generated by segmenting the base digital image utilizing a shape mask; and
generating, utilizing the structural number of steps of a diffusion neural network, a stylized image corresponding to the shape mask in a style of the base digital image by denoising the mask-segmented image noise map utilizing denoising layers of the diffusion neural network conditioned on the base image embedding.
16 . The non-transitory computer readable medium of claim 15 , wherein the operations further comprise:
generating a mask-segmented digital image by applying the shape mask to the base digital image; and
generating the mask-segmented image noise map from the mask-segmented digital image utilizing the structural number of steps of the diffusion noising model.
17 . The non-transitory computer readable medium of claim 15 , wherein the shape mask comprises a typography character mask and the operations further comprise creating the stylized image by generating a stylized typography character that reflects the base digital image utilizing the structural number of steps of the diffusion neural network.
18 . The non-transitory computer readable medium of claim 15 , wherein the operations further comprise generating a stylized animation by generating a plurality of style images utilizing a plurality of structural numbers of steps.
19 . The non-transitory computer readable medium of claim 15 , wherein the operations further comprise:
generating an intermediate noise map from the mask-segmented image noise map utilizing a denoising step of the diffusion neural network conditioned on the base image embedding; and
generating the stylized image from the intermediate noise map utilizing additional denoising steps of the diffusion neural network conditioned on the base image embedding.
20 . The non-transitory computer readable medium of claim 15 , wherein the operations further comprise:
providing, for display via the user interface of the client device, a mask input element and a structural weight element;
determining the shape mask based on user input with the mask input element; and
selecting the structural number of steps based on user interaction with the structural weight element.