Artificial intelligence (AI)-based customized storytelling video generation system

1 . A system for performing artificial intelligence (AI)-based customized storytelling video generation, comprising:

a story designer AI agent;

a storyboard generator AI agent;

a video creator AI agent;

an agent manager AI agent; and

an observer AI agent, wherein

based on a textual prompt and a reference video provided by a user of the system, under coordination of the agent manager AI agent, the story designer AI agent, the storyboard generator AI agent, the video creator AI agent, and the observer AI agent cooperate to generate a multi-shot video, the multi-shot video depicting a story of a customized subject presented in the reference video,

each of the story designer AI agent, the agent manager AI agent, and the observer AI agent is based on a Large Language Model (LLM),

the storyboard generator AI agent uses a three-step storyboard generation pipeline to preserve consistency of character details of the customized subject across shots of the multi-shot video, the pipeline including a generation step, a removal step, and a redrawing step, and

the video creator AI agent is based on a Latent Diffusion Model (LDM)-based Image-to-Video (I2V) generation model configured to preserve the consistency of the character details of the customized subject within each shot of the multi-shot video.

2 . The system of claim 1 , wherein the story designer AI agent is configured to apply an LLM to generate a story script based on the textual prompt, the generated story script includes a plurality of story descriptions, and each story description corresponds to one shot of the multi-shot video.

3 . The system of claim 2 , wherein the story designer AI agent is further configured to generate the story script by:

identifying a story requirement from the textual prompt,

generating a narrative storyline based on the story requirement,

decomposing the narrative storyline into multiple shots,

generating the plurality of story descriptions, each story description corresponding to one shot of the multiple shots, and

updating the narrative storyline and the plurality of story descriptions, based on a review result made by the observer AI agent.

4 . The system of claim 2 , wherein each story description of the plurality of story descriptions describes, for a corresponding shot of the multi-shot video:

a character shown in the shot,

an action of the character,

a character region in the shot,

a background scene,

a shot type, and

a shot motion.

5 . The system of claim 3 , wherein the story designer AI agent is further configured to update the narrative storyline and the plurality of story descriptions by iteratively performing:

via the agent manager AI agent, sending the narrative storyline and the plurality of story descriptions generated during a previous iteration to the observer AI agent,

via the agent manager AI agent, receiving the review result from the observer AI agent, and

refining the narrative storyline and the plurality of story descriptions, based on the review result,

until a predefined criterion is met.

6 . The system of claim 5 , wherein the predefined criterion is:

the review result from the observer AI agent indicating that the narrative storyline and the plurality of story descriptions are approved, or

a maximum iteration number being reached.

7 . The system of claim 2 , wherein the storyboard generator AI agent is configured to generate a plurality of storyboard images, based on the plurality of story descriptions and the reference video, and each storyboard image presents contents of one shot of the multi-shot video.

8 . The system of claim 7 , wherein the storyboard generator AI agent is further configured to generate the plurality of storyboard images by:

obtaining, based on the reference video, a subject image with background removed, the subject image depicting the customized subject of the multi-shot video,

generating a plurality of initial storyboard images based on the plurality of story descriptions,

applying a removal algorithm to perform subject segmentation on the plurality of initial storyboard images, so as to generate a plurality of subject masks within the plurality of initial storyboard images,

redrawing, based on the subject image, mask positions corresponding to the plurality of subject masks, so as to generate the plurality of storyboard images, and

updating the plurality of storyboard images, based on a review result made by the observer AI agent.

9 . The system of claim 8 , wherein the storyboard generator AI agent is further configured to update the plurality of storyboard images by iteratively performing:

via the agent manager AI agent, sending the plurality of storyboard images generated during a previous iteration to the observer AI agent,

via the agent manager AI agent, receiving the review result from the observer AI agent, and

refining the plurality of storyboard images based on the review result,

until a predefined criterion is met.

10 . The system of claim 7 , wherein the video creator AI agent is configured to apply the LDM-based I2V generation model to:

generate the multi-shot video based on the plurality of story descriptions, the plurality of storyboard images, and the reference video, and

update the multi-shot video based on a review result made by the observer AI agent.

11 . The system of claim 10 , wherein the LDM-based I2V generation model includes a variational autoencoder (VAE) encoder and decoder, a text encoder, an image condition encoder, and a 3D U-Net, and

the LDM-based I2V generation model is further configured to generate the multi-shot video by:

sampling a noisy video from a Gaussian distribution,

encoding the plurality of storyboard images via the VAE encoder to obtain latent representations of the plurality of storyboard images,

concatenating the latent representations with the noisy video as an input into the 3D U-Net,

extracting a text embedding via the text encoder from the plurality of story descriptions,

extracting an image embedding via the image condition encoder from the plurality of storyboard images,

injecting the text embedding and the image embedding into the 3D U-Net,

feeding an output of the 3D U-Net into the VAE decoder, and

obtaining the multi-shot video at an output of the VAE decoder.

12 . The system of claim 11 , wherein the 3D U-Net includes a self-attention block, a temporal attention block, and a cross-attention block, and the text embedding and the image embedding are injected into the cross-attention block.

13 . The system of claim 12 , wherein a parameter is trained for each linear projection in the self-attention block, the temporal attention block, and the cross-attention block.

14 . The system of claim 12 , wherein the cross-attention block includes 16 cross-attention modules, 16 token embeddings are trained for the 16 cross-attention modules, and each token embedding is injected into one cross-attention module.

15 . The system of claim 10 , wherein the LDM-based I2V generation model is trained using a training sample obtained from the reference video.

16 . The system of claim 15 , wherein a localization loss is applied on a cross-attention map during the training of the LDM-based I2V generation model, to maximize a similarity inside a subject location.

17 . The system of claim 10 , wherein the story designer AI agent is further configured to update the multi-shot video by iteratively performing:

via the agent manager AI agent, sending the multi-shot video generated during a previous iteration to the observer AI agent,

via the agent manager AI agent, receiving the review result from the observer AI agent, and

refining the multi-shot video based on the review result,

until a predefined criterion is met.

18 . The system of claim 1 , wherein the agent manager AI agent is configured to manage data flow between the story designer AI agent, the storyboard generator AI agent, the video creator AI agent, and the observer AI agent.

19 . The system of claim 1 , wherein the reviewer AI agent is further configured to apply a multimodal LLM to assess data generated by the story designer AI agent, the storyboard generator AI agent, and the video creator AI agent, and inform the agent manager AI agent of a result of the assessment.

20 . The system of claim 1 , wherein the textual prompt provided by the user of the system includes:

a background description of the multi-shot video,

an action of the customized subject, and

a number of shots included in the multi-shot video.