Scribble-to-vector image generation

1 . A method for image generation, comprising:

obtaining a sketch input depicting an object;

processing, using a sketch encoder, the sketch input to obtain sketch guidance, wherein the sketch encoder is initialized by copying parameters of an image generation model and trained to generate sketch guidance for the image generation model based on a training sketch input; and

generating, using the image generation model, a synthesized image based on the sketch guidance, wherein the synthesized image depicts the object from the sketch input.

2 . The method of claim 1 , further comprising:

obtaining a text prompt; and

encoding the text prompt to obtain a text encoding, wherein the synthesized image is generated based on the text encoding.

3 . The method of claim 1 , wherein generating the synthesized image comprises:

providing the sketch guidance as an input to a decoder layer of the image generation model.

4 . The method of claim 1 , further comprising:

generating a vector image based on the synthesized image.

5 . The method of claim 1 , wherein generating the synthesized image comprises:

performing a reverse diffusion process.

6 . The method of claim 1 , wherein encoding the sketch input comprises:

obtaining a noisy input image for the image generation model, wherein the synthesized image is based on the noisy input image.

7 . The method of claim 1 , wherein encoding the sketch input comprises:

obtaining a class label indicating a category of the synthesized image, wherein the sketch guidance is generated based on the class label.

8 . The method of claim 7 , wherein:

the category is selected from a set of categories including an outline category and a color image category.

9 . The method of claim 1 , wherein generating the synthesized image comprises:

obtaining an adherence factor indicating a degree of adherence of the synthesized image to the sketch input, wherein the synthesized image is generated based on the adherence factor.

10 . The method of claim 9 , wherein generating the synthesized image comprises:

applying the adherence factor to the sketch guidance prior to providing the sketch guidance to the image generation model.

11 . The method of claim 1 , wherein:

the sketch encoder and the image generation model are diffusion models.

12 . A method for training a machine learning model, comprising:

obtaining a training set including a sketch input;

initializing a sketch encoder based on parameters of an image generation model; and

training the sketch encoder to generate sketch guidance for the image generation model based on the sketch input;

wherein training the sketch encoder comprises:

generating, using the image generation model, a synthetic image based on an output of the sketch encoder; and

computing a loss function based on the synthetic image and a ground-truth image.

13 . The method of claim 12 , wherein creating the training set comprises:

generating the sketch input based on a ground-truth image by applying white-fill, black-stroke, path dropping, stroke width augmentation, control point perturbation, path translation, rotation, or any combination thereof.

14 . The method of claim 12 , wherein initializing the sketch encoder comprises:

copying the parameters of an encoder of the image generation model.

15 . An apparatus comprising:

at least one processor;

at least one memory storing instruction executable by the at least one processor;

a sketch encoder comprising parameters stored in the at least one memory and trained to generate sketch guidance based on a sketch input, wherein the sketch encoder is trained using a sketch input that is generated based on a ground-truth image by applying white-fill, black-stroke, path dropping, stroke width augmentation, control point perturbation, path translation, rotation, or any combination thereof; and

an image generation model comprising instruction stored in the at least one memory and trained to generate a synthesized image based on the sketch guidance.

16 . The apparatus of claim 15 , wherein:

the image generation model comprises a diffusion model.

17 . The apparatus of claim 15 , wherein:

the sketch encoder comprises a zero-convolution layer and a layer based on an encoder layer of the image generation model.