Multimodal diffusion models
Systems and methods for image processing are described. Embodiments of the present disclosure obtain a noise image and guidance information for generating an image. A diffusion model generates an intermediate noise prediction for the image based on the noise image. A conditioning network generates noise modulation parameters. The intermediate noise prediction and the noise modulation parameters are combined to obtain a modified intermediate noise prediction. The diffusion model generates the image based on the modified intermediate noise prediction, wherein the image depicts a scene based on the guidance information.
1 . A method comprising:
obtaining a noise input and guidance information indicating a target image control;
generating, using an image generation model, a noise prediction based on the noise input, wherein the noise prediction comprises a prediction of noise to remove from the noise input;
generating noise modulation parameters based on the guidance information using a conditioning network, wherein the noise modulation parameters represent the target image control;
combining the noise prediction and the noise modulation parameters to obtain a modified noise prediction, wherein the modified noise prediction comprises a modified prediction of noise to remove from the noise input based on the target image control; and
generating, using the image generation model, a synthetic image based on the modified noise prediction, wherein the synthetic image depicts a scene based on the target image control.
2 . The method of claim 1 , wherein:
the guidance information comprises non-textual guidance.
3 . The method of claim 1 , wherein:
the guidance information comprises an additional modality other than a training modality used for training the image generation model.
4 . The method of claim 1 , wherein:
the guidance information comprises multiple modalities.
5 . The method of claim 1 , wherein the combining comprises:
performing element-wise multiplication of the noise prediction and a first portion of the noise modulation parameters; and
performing element-wise addition of the noise prediction and a second portion of the noise modulation parameters.
6 . The method of claim 1 , further comprising:
iteratively generating a plurality of intermediate noise predictions corresponding to a plurality of diffusion steps, respectively; and
iteratively generating a plurality of noise modulation parameters corresponding to the plurality of intermediate noise predictions, respectively, wherein the synthetic image is generated based on the plurality of noise modulation parameters.
7 . The method of claim 1 , further comprising:
generating an intermediate image prediction, wherein the noise modulation parameters are generated based on the intermediate image prediction.
8 . The method of claim 7 , further comprising:
generating a subsequent image prediction based on the intermediate image prediction and the modified noise prediction, wherein the synthetic image is generated based on the subsequent image prediction.
9 . An apparatus comprising:
a memory component;
a processing device coupled to the memory component, the processing device configured to perform operations comprising:
obtaining a noise input and guidance information indicating a target image control;
generating, using an image generation model, a noise prediction based on the noise input, wherein the noise prediction comprises a prediction of noise to remove from the noise input;
generating noise modulation parameters based on the guidance information using a conditioning network, wherein the noise modulation parameters represent the target image control;
combining the noise prediction and the noise modulation parameters to obtain a modified noise prediction, wherein the modified noise prediction comprises a modified prediction of noise to remove from the noise input based on the target image control; and
generating, using the image generation model, a synthetic image based on the modified noise prediction, wherein the synthetic image depicts a scene based on the target image control.
10 . The apparatus of claim 9 , wherein:
the image generation model comprises a U-Net architecture, wherein the noise modulation parameters are applied to an output of the U-Net architecture of the image generation model.
11 . The apparatus of claim 9 , wherein:
the image generation model comprises a text-guided diffusion model, and the image generation model takes a text prompt as an input.
12 . The apparatus of claim 9 , wherein:
the conditioning network comprises a U-Net architecture, wherein the noise modulation parameters comprise an output of the U-Net architecture of the conditioning network.
13 . A non-transitory computer readable medium storing code for video processing, the code comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
obtaining a noise input and guidance information indicating a target image control;
generating, using an image generation model, a noise prediction based on the noise input, wherein the noise prediction comprises a prediction of noise to remove from the noise input;
generating noise modulation parameters based on the guidance information using a conditioning network, wherein the noise modulation parameters represent the target image control;
combining the noise prediction and the noise modulation parameters to obtain a modified noise prediction, wherein the modified noise prediction comprises a modified prediction of noise to remove from the noise input based on the target image control; and
generating, using the image generation model, a synthetic image based on the modified noise prediction, wherein the synthetic image depicts a scene based on the target image control.
14 . The non-transitory computer readable medium of claim 13 , wherein:
the guidance information comprises non-textual guidance.
15 . The non-transitory computer readable medium of claim 13 , wherein:
the guidance information comprises an additional modality other than a training modality used for training the image generation model.
16 . The non-transitory computer readable medium of claim 13 , wherein:
the guidance information comprises multiple modalities.
17 . The non-transitory computer readable medium of claim 13 , the code further comprising instructions executable by the at least one processor to perform operations comprising:
performing element-wise multiplication of the noise prediction and a first portion of the noise modulation parameters; and
performing element-wise addition of the noise prediction and a second portion of the noise modulation parameters.
18 . The non-transitory computer readable medium of claim 13 , the code further comprising instructions executable by the at least one processor to perform operations comprising:
iteratively generating a plurality of intermediate noise predictions corresponding to a plurality of diffusion steps, respectively; and
iteratively generating a plurality of noise modulation parameters corresponding to the plurality of intermediate noise predictions, respectively, wherein the synthetic image is generated based on the plurality of noise modulation parameters.
19 . The non-transitory computer readable medium of claim 13 , the code further comprising instructions executable by the at least one processor to perform operations comprising:
generating an intermediate image prediction, wherein the noise modulation parameters are generated based on the intermediate image prediction.
20 . The non-transitory computer readable medium of claim 19 , the code further comprising instructions executable by the at least one processor to perform operations comprising:
generating a subsequent image prediction based on the intermediate image prediction and the modified noise prediction, wherein the synthetic image is generated based on the subsequent image prediction.