Teaching language models to draw sketches
A processor-implemented method for image generation using an artificial neural network (ANN) includes receiving an input including one or more of an image or a text prompt. The ANN processes the input to determine one or more virtual brush strokes to generate an output image or one or more commands for controlling an image drawing application to generate the output image. A list of the one or more virtual brush strokes to generate the output image or the one or more commands for controlling the image drawing application to generate the output image. The one or more virtual brush strokes or commands may be executed to generate a sketch based on the input.
1 . A processor-implemented method performed by at least one processor, the processor-implemented method comprising:
receiving a first input including one or more of a first image or a text prompt;
processing, by an artificial neural network (ANN), the first input to determine a sequence of virtual brush strokes to draw an output image and one or more commands corresponding to the sequence of virtual brush strokes for controlling an image drawing application to draw the output image, each virtual brush stroke comprising a brush stroke of a sketch provided in a stroke format or painting provided in the stroke format;
generating a list of the sequence of virtual brush strokes to draw the output image and the one or more commands corresponding to the sequence of virtual brush strokes for controlling the image drawing application to draw the output image, the one or more commands having color, coordinate and width parameters; and
executing the sequence of virtual brush strokes to render a second image of each of the sequence of virtual brush strokes on a stroke-by-stroke basis, wherein the executing comprises, for each virtual brush stroke in the sequence of virtual brush strokes:
rendering a virtual brush stroke on a virtual canvas to produce an updated canvas state; and
providing the updated canvas state as a second input to the ANN to determine a next virtual brush stroke in the sequence of virtual brush strokes.
2 . The processor-implemented method of claim 1 , in which the first input includes only the text prompt.
3 . The processor-implemented method of claim 1 , in which the first input includes the first image and the text prompt and the ANN performs a visual reasoning task to determine the output image.
4 . The processor-implemented method of claim 3 , in which the first image comprises a partial drawing of an object and the ANN generates the list of the sequence of virtual brush strokes to produce a complete drawing of the object.
5 . The processor-implemented method of claim 3 , in which the ANN determines a classification based on the output image.
6 . The processor-implemented method of claim 1 , in which the first image comprises multiple objects, and the ANN determines the list of the sequence of virtual brush strokes to draw the output image, the output image including a subset of the multiple objects.
7 . The processor-implemented method of claim 1 , in which the ANN comprises a language model.
8 . An apparatus, comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
receive a first input including one or more of a first image or a text prompt;
process, by an artificial neural network (ANN), the first input to determine a sequence of virtual brush strokes to draw an output image and one or more commands corresponding to the sequence of virtual brush strokes for controlling an image drawing application to draw the output image, each virtual brush stroke comprising a brush stroke of a sketch provided in a stroke format or painting provided in the stroke format;
generate a list of the sequence of virtual brush strokes to draw the output image and the one or more commands corresponding to the sequence of virtual brush strokes for controlling the image drawing application to draw the output image, the one or more commands having color, coordinate and width parameters; and
execute the sequence of virtual brush strokes to render a second image of each of the sequence of virtual brush strokes on a stroke-by-stroke basis, wherein, for each virtual brush stroke in the sequence of virtual brush strokes, the at least one processor is further configured to:
render a virtual brush stroke on a virtual canvas to produce an updated canvas state, and
provide the updated canvas state as a second input to the ANN to determine a next virtual brush stroke in the sequence of virtual brush strokes.
9 . The apparatus of claim 8 , in which the first input includes only the text prompt.
10 . The apparatus of claim 8 , in which the first input includes the first image and the text prompt and the ANN performs a visual reasoning task to determine the output image.
11 . The apparatus of claim 10 , in which the first image comprises a partial drawing of an object and the ANN generates the list of the sequence of virtual brush strokes to produce a complete drawing of the object.
12 . The apparatus of claim 8 , in which the at least one processor is further configured to determine, by the ANN, a classification based on the output image.
13 . The apparatus of claim 8 , in which the first image comprises multiple objects, and the at least one processor is further configured to determine, by the ANN, the list of the sequence of virtual brush strokes to draw the output image, the output image including a subset of the multiple objects.
14 . The apparatus of claim 8 , in which the ANN comprises a language model.
15 . A non-transitory computer-readable medium storing instructions that when executed by a processor cause the processor to:
receive a first input including one or more of a first image or a text prompt;
process, by an artificial neural network (ANN), the first input to determine a sequence of virtual brush strokes to draw an output image and one or more commands corresponding to the sequence of virtual brush strokes for controlling an image drawing application to draw the output image, each virtual brush stroke comprising a brush stroke of a sketch provided in a stroke format or painting provided in the stroke format;
generate a list of the sequence of virtual brush strokes to draw the output image and the one or more commands corresponding to the sequence of virtual brush strokes for controlling the image drawing application to draw the output image, the one or more commands having color, coordinate and width parameters; and
execute the sequence of virtual brush strokes to render a second image of each of the sequence of virtual brush strokes on a stroke-by-stroke basis, wherein, for each virtual brush stroke in the sequence of virtual brush strokes, the processor is further caused to:
render a virtual brush stroke on a virtual canvas to produce an updated canvas state; and
provide the updated canvas state as a second input to the ANN to determine a next virtual brush stroke in the sequence of virtual brush strokes.
16 . The non-transitory computer-readable medium of claim 15 , in which the first input includes only the text prompt.
17 . The non-transitory computer-readable medium of claim 15 , in which the first input includes the first image and the text prompt and the instructions further cause the processor to perform, by the ANN, a visual reasoning task to determine the output image.
18 . The non-transitory computer-readable medium of claim 17 , in which the first image comprises a partial drawing of an object and the ANN generates the list of the sequence of virtual brush strokes to produce a complete drawing of the object.
19 . The non-transitory computer-readable medium of claim 15 , in which the instructions further cause the processor to determine, by the ANN, a classification based on the output image.
20 . The non-transitory computer-readable medium of claim 15 , in which the first image comprises multiple objects, and the instructions further cause the processor to determine, by the ANN, the list of the sequence of virtual brush strokes to draw the output image, the output image including a subset of the multiple objects.
21 . The non-transitory computer-readable medium of claim 15 , in which the ANN comprises a language model.