Generation of story videos corresponding to user input using generative models
The present disclosure provides systems and methods for video generation corresponding to a user input. Given a user input, a story video with content relevant to the user input can be generated. One aspect includes a computing system comprising a processor and memory. The processor can be configured to execute a program using portions of the memory to receive the user input, generate a story text based on the user input, generate a plurality of story images based on the story text, and output a story including the story text and a story video having content corresponding to the story text, wherein the story video includes the plurality of story images. Additionally or alternatively, the story video can include audio data and a plurality of generated animated videos, each animated video corresponding to a story image in the plurality of story images.
1. A computing system for video generation corresponding to a user input, the computing system comprising:
a processor and memory, the processor configured to execute a program using portions of the memory to:
receive the user input;
generate a story text based on the user input, wherein the story text comprises a plurality of sentences generated by:
generating a sentence based on the user input; and
iteratively generating successive sentences using one or more of previously generated sentences and the user input;
generate a plurality of story images based on the story text, each of the plurality of story images corresponding to one of the plurality of sentences; and
output a story, the story including the story text and a story video having content corresponding to the story text, wherein the story video comprises a plurality of animated videos, each animated video corresponding to a respective story image of the plurality of story images.
2. The computing system of claim 1 , wherein the story video is generated by:
for each of the plurality of animated videos:
performing monocular depth estimation on the respective story image corresponding to the animated video;
generating the animated video by animating the corresponding respective story image using information from the monocular depth estimation.
3. The computing system of claim 1 , wherein the processor is further configured to provide audio data, and wherein the story includes the audio data.
4. The computing system of claim 3 , wherein the audio data is provided based on a selection by a user.
5. The computing system of claim 3 , wherein the audio data is provided based on the story text.
6. The computing system of claim 1 , wherein the user input comprises an input sentence.
7. The computing system of claim 1 , wherein the user input includes an artistic style descriptor, and the plurality of story images is generated based on the artistic style descriptor.
8. The computing system of claim 1 , wherein the plurality of sentences is generated using a sequence-to-sequence transformer model.
9. The computing system of claim 1 , wherein the plurality of story images is generated using a generative diffusion model.
10. A method for video generation corresponding to a user input, the method comprising:
receiving the user input;
generating a story text based on the user input, wherein the story text comprises a plurality of sentences generated by:
generating a sentence based on the user input; and
iteratively generating successive sentences using one or more of previously generated sentences and the user input;
generating a plurality of story images based on the story text, each of the plurality of story images corresponding to one of the plurality of sentences; and
outputting a story including the story text and a story video, wherein the story video comprises a plurality of animated videos, each animated video corresponding to a respective story image of the plurality of story images.
11. The method of claim 10 , wherein the story video is generated by:
for each of the plurality of animated videos:
performing monocular depth estimation on the respective story image corresponding to the animated video;
generating the animated video by animating the corresponding respective story image using information from the monocular depth estimation.
12. The method of claim 10 , further comprising providing audio data based on the story text using a sentiment analysis model, wherein the story includes the audio data.
13. The method of claim 10 , wherein the plurality of sentences is generated using a sequence-to-sequence transformer model.
14. The method of claim 10 , wherein the plurality of story images is generated using a generative diffusion model and a language-text matching model.
15. A computing system for video generation corresponding to a user input, the computing system comprising:
a processor and memory, the processor configured to execute a program using portions of the memory to:
receive the user input, wherein the user input includes one or more words;
generate a story text based on the user input, wherein the story text includes a plurality of sentences having contextual coherence, and wherein the plurality of sentences is generated by:
generating a sentence based on the user input; and
iteratively generating successive sentences using one or more of previously generated sentences and the user input;
generate a plurality of story images, wherein each of the plurality of story images corresponds to a sentence in the plurality of sentences;
select audio data based on the story text; and
output a story including the story text, the selected audio data, and a story video having content corresponding to the story text, wherein the story video comprises a plurality of animated videos, each animated video corresponding to a respective story image of the plurality of story images.
16. The computing system of claim 15 , wherein the story video is generated by:
for each of the plurality of animated videos:
performing monocular depth estimation on the respective story image corresponding to the animated video;
generating the animated video by animating the corresponding respective story image using information from the monocular depth estimation.
17. The computing system of claim 15 , wherein the audio data is selected based on the story text using a sentiment analysis model.
18. The computing system of claim 15 , wherein the plurality of story images is generated using a generative diffusion model and a language-text matching model.