Text-guided video generation
A method, apparatus, non-transitory computer readable medium, and system for video generation include obtaining an input image having an element depicted in a first view angle, generating a synthetic image depicting the element of the input image from a second view angle different from the first view angle, generating an intermediate image by interpolating based on the synthetic image, and generating a video based on the synthetic image and the intermediate image, where the video depicts the element of the input image from a changing view angle.
1 . A method comprising:
obtaining an input image having an element depicted from a first camera view angle;
generating a novel view image depicting the element from a second camera view angle different from the first camera view angle, wherein the novel view image is generated using a 3D model of the element, and wherein the novel view image includes an occluded area;
generating, using an image generation model, an inpainted image depicting the element of the input image from the second camera view angle by performing a diffusion process to inpaint the occluded area of the novel view image;
generating an intermediate image by performing image interpolation between the input image and the inpainted image, wherein the intermediate image has a camera view angle between the first camera view angle and the second camera view angle; and
generating a video based on the input image, the inpainted image, and the intermediate image, wherein the video depicts the element of the input image from a changing camera view angle.
2 . The method of claim 1 , further comprising:
generating, using the image generation model, an additional synthetic image depicting the element of the input image from a third camera view angle, wherein the intermediate image is interpolated based on the inpainted image and the additional synthetic image.
3 . The method of claim 1 , further comprising:
identifying a view trajectory, wherein the second camera view angle is based on the view trajectory.
4 . The method of claim 1 , wherein generating the inpainted image comprises:
generating a preliminary image from the second camera view angle, wherein the preliminary image includes an occluded area and the inpainted image is generated by inpainting the occluded area of the preliminary image.
5 . The method of claim 1 , further comprising:
obtaining a depth map of the input image, wherein the inpainted image is generated based on the depth map.
6 . The method of claim 5 , further comprising:
projecting the input image to a 3D point cloud based on the depth map.
7 . The method of claim 5 , further comprising:
generating an occlusion mask based on the depth map, wherein the inpainted image is generated by performing inpainting based on the occlusion mask.
8 . The method of claim 1 , further comprising:
obtaining a text prompt, wherein the inpainted image is generated based on the text prompt.
9 . The method of claim 1 , further comprising:
obtaining a text prompt, wherein the input image is generated based on the text prompt.
10 . The method of claim 1 , wherein:
the image generation model is trained to generate images using training data that includes a ground truth image and an occlusion mask.
11 . An apparatus comprising:
at least one processor;
at least one memory storing instructions executable by the at least one processor;
a view component comprising parameters stored in the at least one memory and configured to generate a novel view image depicting an element from a second camera view angle different from a first camera view angle of an input image, wherein the novel view image is generated using a 3D model of the element, and wherein the novel view image includes an occluded area;
an image generation model comprising parameters stored in the at least one memory and trained to generate an inpainted image depicting the element from the second camera view angle by performing a diffusion process to inpaint the occluded area of the novel view image;
an interpolation component comprising parameters stored in the at least one memory and configured to generate an intermediate image by performing image interpolation between the input image and the inpainted image, wherein the intermediate image has a camera view angle between the first camera view angle and the second camera view angle; and
a video generation component comprising parameters stored in the at least one memory and configured to generate a video based on the input image, the inpainted image, and the intermediate image, wherein the video depicts the element of the input image from a changing camera view angle.
12 . The apparatus of claim 11 , further comprising:
a depth component comprising parameters stored in the at least one memory and configured to generate a depth map based on the input image.
13 . The apparatus of claim 11 , further comprising:
a projection component comprising parameters stored in the at least one memory and configured to project the input image to a 3D point cloud.
14 . The apparatus of claim 11 , further comprising:
a view component comprising parameters stored in the at least one memory and configured to generate a preliminary image from the second camera view angle.
15 . The apparatus of claim 11 , wherein,
the image generation model comprises a diffusion UNet architecture.
16 . A non-transitory computer readable medium storing code for video generation, the code comprising instructions executable by at least one processor to:
obtain an input image having an element depicted from a first camera view angle;
generate a novel view image depicting the element from a second camera view angle different from the first camera view angle, wherein the novel view image is generated using a 3D model of the element, and wherein the novel view image includes an occluded area;
generate, using an image generation model, an inpainted image depicting the element of the input image from the second camera view angle by performing a diffusion process to inpaint the occluded area of the novel view image;
generate an intermediate image by performing image interpolation between the input image and the inpainted image, wherein the intermediate image has a camera view angle between the first camera view angle and the second camera view angle; and
generate a video based on the input image, the inpainted image, and the intermediate image, wherein the video depicts the element of the input image from a changing camera view angle.
17 . The non-transitory computer readable medium of claim 16 , the code further comprising instructions executable by the at least one processor to:
generate, using the image generation model, an additional synthetic image depicting the element of the input image from a third camera view angle, wherein the intermediate image is interpolated based on the inpainted image and the additional synthetic image.
18 . The non-transitory computer readable medium of claim 16 , the code further comprising instructions executable by the at least one processor to:
generate a preliminary image from the second camera view angle, wherein the preliminary image includes an occluded area and the inpainted image is generated by inpainting the occluded area of the preliminary image.
19 . The non-transitory computer readable medium of claim 16 , the code further comprising instructions executable by the at least one processor to:
obtain a depth map of the input image, wherein the inpainted image is generated based on the depth map.
20 . The non-transitory computer readable medium of claim 19 , the code further comprising instructions executable by the at least one processor to:
generate an occlusion mask based on the depth map, wherein the inpainted image is generated by performing inpainting based on the occlusion mask.