Neural compositing by embedding generative technologies into non-destructive document editing workflows
One or more aspects of the method, apparatus, and non-transitory computer readable medium include obtaining an original image, a scene graph describing elements of the original image, and a description of a modification to the original image. The one or more aspects further include updating the scene graph based on the description of the modification. The one or more aspects further include generating a modified image using an image generation neural network based on the updated scene graph, wherein the modified image incorporates content based on the original image and the description of the modification.
1 . A method for neural compositing, comprising:
obtaining an original image, a scene graph describing elements of the original image, and a description of a modification to the original image;
updating the scene graph based on the description of the modification; and
generating a modified image using an image generation neural network based on the updated scene graph, wherein the modified image incorporates content, wherein the content is based on the original image and the description of the modification.
2 . The method of claim 1 , further comprising:
updating the scene graph by adding a node to the scene graph based on the description of the modification.
3 . The method of claim 1 , further comprising:
updating the scene graph by modifying a node of the scene graph based on the description of the modification.
4 . The method of claim 1 , further comprising:
generating a partial image corresponding to a node of the updated scene graph; and
combining the partial image with the original image to obtain the modified image.
5 . The method of claim 4 , further comprising:
generating a seed for the partial image, wherein the partial image is based on the seed, and wherein the node includes the seed as an attribute.
6 . The method of claim 5 , further comprising:
generating an additional partial image corresponding to the node based on the seed; and
combining the additional partial image with the modified image to obtain an additional modified image.
7 . The method of claim 4 , further comprising:
identifying one or more conditions for generating the partial image, wherein the node includes the one or more conditions as attributes.
8 . The method of claim 7 , further comprising:
generating an additional partial image corresponding to the node based on the one or more conditions; and
combining the additional partial image with the modified image to obtain an additional modified image.
9 . The method of claim 1 , wherein:
the image generation neural network is trained to generate the modified image using a diffusion training technique.
10 . The method of claim 1 , further comprising:
receiving an additional description of an additional modification to the original image;
updating the scene graph based on the description of the additional modification; and
generating an additional modified image using the image generation neural network based on the updated scene graph, wherein the additional modified image incorporates content based on the original image and the additional description of the additional modification.
11 . The method of claim 10 , further comprising:
generating a first partial image based on a description of the modification, wherein the modified image is based on the first partial image;
generating a second partial image based on a second description of a second modification, wherein the additional modified image is based on the second partial image; and
resampling the first partial image based on the description and the second partial image.
12 . An apparatus for neural compositing, comprising:
a processor; and
a memory storing instructions and in electronic communication with the processor, the processor being configured to execute the instructions to:
obtain an original image, a scene graph describing elements of the original image, and a description of a modification to the original image;
update the scene graph based on the description of the modification; and
generate a modified image using an image generation neural network based on the updated scene graph, wherein the modified image incorporates content, wherein the content is based on the original image and the description of the modification.
13 . The apparatus of claim 12 , the processor being further configured to execute the instructions to:
update the scene graph by adding a node to the scene graph based on the description of the modification.
14 . The apparatus of claim 12 , the processor being further configured to execute the instructions to:
update the scene graph by modifying a node of the scene graph based on the description of the modification.
15 . The apparatus of claim 12 , the processor being further configured to execute the instructions to:
generate a partial image corresponding to a node of the updated scene graph; and
combine the partial image with the original image to obtain the modified image.
16 . The apparatus of claim 15 , the processor being further configured to execute the instructions to:
generate a seed for the partial image, wherein the partial image is based on the seed, and wherein the node includes the seed as an attribute.
17 . A non-transitory computer readable medium storing code for image generation, the code comprising instructions executable by a processor to:
obtaining an original image, a scene graph describing elements of the original image, and a description of a modification to the original image;
update the scene graph based on the description of the modification; and
generate a modified image using an image generation neural network based on the updated scene graph, wherein the modified image incorporates content, wherein the content is based on the original image and the description of the modification.
18 . The non-transitory computer readable medium of claim 17 , the code further comprising instructions executable by the processor to:
update the scene graph by adding a node to the scene graph based on the description of the modification.
19 . The non-transitory computer readable medium of claim 17 , the code further comprising instructions executable by the processor to:
update the scene graph by modifying a node of the scene graph based on the description of the modification.
20 . The non-transitory computer readable medium of claim 17 , the code further comprising instructions executable by the processor to:
generate a partial image corresponding to a node of the updated scene graph; and
combine the partial image with the original image to obtain the modified image.