Granted Patent B2

US 12,639,947 · App. 18/924,055 · Granted May 26, 2026

Inventors: Xiangyuan Zhao (Irvine, CA); Brian Chen (Irvine, CA); Yingnan Zhu (Irvine, CA)

Assignee: Samsung Electronics Co., Ltd.

G06V20/47G06V10/806G06V20/41H04N21/25891H04N21/8549G06V10/776G06V10/82

Abstract

Claims (90)

1 . A method comprising:

generating a multimedia content result using a streaming multimodal video generation model, including:

acquiring video data and audio data associated with a multimedia content item;

performing a first scene detection process on the video data to produce first scene detection results, wherein the first scene detection process divides the video data into separate video scenes;

performing a second scene detection process on the audio data to produce second scene detection results, wherein the second scene detection process divides the audio data into separate audio scenes;

creating prompt text based on one or more user preferences;

generating image frames based on the prompt text; and

compiling the image frames into the multimedia content result; and

displaying the multimedia content result on a display.

2 . The method of claim 1 , further comprising:

determining combined scene detection results based on the first scene detection results and the second scene detection results;

inputting at least a portion of the combined scene detection results into a genre-specific multimodal transformer (GMT) model; and

generating the multimedia content result as a summarization for the multimedia content item using one or more outputs from the GMT model.

3 . The method of claim 2 , wherein generating the multimedia content result as the summarization for the multimedia content item includes:

determining one or more scene importance metrics based on the one or more user preferences and the one or more outputs from the GMT model, wherein the one or more outputs of the GMT model include a scene importance score and a predicted genre; and

generating the summarization based on the one or more scene importance metrics.

4 . The method of claim 3 , further comprising:

performing an alignment of the first scene detection results, the second scene detection results, and genre data, including mapping the first scene detection results, the second scene detection results, and the genre data into a same latent space; and

performing a self-attention process to merge the first scene detection results, the second scene detection results, and the genre data.

5 . The method of claim 4 , wherein the streaming multimodal video generation model includes:

a transformer-based model for creating the prompt text using inputs based on the one or more outputs from the GMT model;

a diffusion model for generating an image frame based on the prompt text; and

a consistency control model for evaluating a contrastive loss using the prompt text and a previously generated image frame from the diffusion model.

6 . The method of claim 1 , wherein creating the prompt text includes:

tracking the one or more user preferences over a period of time;

generating, using a behavior description model, a user behavior description summarizing the tracked one or more user preferences over the period of time; and

generating a script using the user behavior description.

7 . The method of claim 6 , wherein the streaming multimodal video generation model includes:

a diffusion model for receiving the script as the prompt text and generating an image frame based on the prompt text;

a consistency control model for evaluating a contrastive loss using the prompt text and a previously generated image frame from the diffusion model; and

a frame post-processing model for combining the image frames into the multimedia content result.

8 . An electronic device comprising:

at least one processing device configured to:

acquire video data and audio data associated with a multimedia content item;

perform a first scene detection process on the video data to produce first scene detection results, wherein the first scene detection process divides the video data into separate video scenes;

perform a second scene detection process on the audio data to produce second scene detection results, wherein the second scene detection process divides the audio data into separate audio scenes;

generate a multimedia content result using a streaming multimodal video generation model, wherein the at least one processing device is further configured to:

create prompt text based on one or more user preferences;

generate image frames based on the prompt text;

compile the image frames into the multimedia content result; and

display the multimedia content result on a display.

9 . The electronic device of claim 8 , wherein the at least one processing device is further configured to:

determine combined scene detection results based on the first scene detection results and the second scene detection results;

input at least a portion of the combined scene detection results into a genre-specific multimodal transformer (GMT) model; and

generate the multimedia content result as a summarization for the multimedia content item using one or more outputs from the GMT model.

10 . The electronic device of claim 9 , wherein, to generate the multimedia content result as the summarization for the multimedia content item, the at least one processing device is further configured to:

determine one or more scene importance metrics based on the one or more user preferences and the one or more outputs from the GMT model, wherein the one or more outputs of the GMT model include a scene importance score and a predicted genre; and

generate the summarization based on the one or more scene importance metrics.

11 . The electronic device of claim 10 , wherein the at least one processing device is further configured to:

perform an alignment of the first scene detection results, the second scene detection results, and genre data, including mapping the first scene detection results, the second scene detection results, and the genre data into a same latent space; and

perform a self-attention process to merge the first scene detection results, the second scene detection results, and the genre data.

12 . The electronic device of claim 11 , wherein the streaming multimodal video generation model includes:

a transformer-based model configured to create the prompt text using inputs based on the one or more outputs from the GMT model;

a diffusion model configured to generate an image frame based on the prompt text; and

a consistency control model configured to evaluate a contrastive loss using the prompt text and a previously generated image frame from the diffusion model.

13 . The electronic device of claim 8 , wherein, to create the prompt text, the at least one processing device is further configured to:

track the one or more user preferences over a period of time;

generate, using a behavior description model, a user behavior description summarizing the tracked one or more user preferences over the period of time; and

generate a script using the user behavior description.

14 . The electronic device of claim 13 , wherein the streaming multimodal video generation model includes:

a diffusion model configured to receive the script as the prompt text and generate an image frame based on the prompt text;

a consistency control model configured to evaluate a contrastive loss using the prompt text and a previously generated image frame from the diffusion model; and

a frame post-processing model for combining the image frames into the multimedia content result.

15 . A non-transitory machine readable medium comprising instructions that when executed cause at least one processor of an electronic device to:

acquire video data and audio data associated with a multimedia content item;

perform a first scene detection process on the video data to produce first scene detection results, wherein the first scene detection process divides the video data into separate video scenes;

perform a second scene detection process on the audio data to produce second scene detection results, wherein the second scene detection process divides the audio data into separate audio scenes;

generate a multimedia content result using a streaming multimodal video generation model, wherein the instructions when executed further cause the at least one processing device to:

create prompt text based on one or more user preferences;

generate image frames based on the prompt text;

compile the image frames into the multimedia content result; and

display the multimedia content result on a display.

16 . The non-transitory machine readable medium of claim 15 , wherein the instructions when executed further cause the at least one processor of the electronic device to:

determine combined scene detection results based on the first scene detection results and the second scene detection results;

input at least a portion of the combined scene detection results into a genre-specific multimodal transformer (GMT) model; and

generate the multimedia content result as a summarization for the multimedia content item using one or more outputs from the GMT model.

17 . The non-transitory machine readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor of the electronic device to generate the multimedia content result as the summarization for the multimedia content item further include instructions that when executed cause the at least one processor of the electronic device to:

determine one or more scene importance metrics based on the one or more user preferences and the one or more outputs from the GMT model, wherein the one or more outputs of the GMT model include a scene importance score and a predicted genre; and

generate the summarization based on the one or more scene importance metrics.

18 . The non-transitory machine readable medium of claim 17 , wherein the instructions when executed further cause the at least one processor of the electronic device to:

perform an alignment of the first scene detection results, the second scene detection results, and genre data, including mapping the first scene detection results, the second scene detection results, and the genre data into a same latent space; and

perform a self-attention process to merge the first scene detection results, the second scene detection results, and the genre data.

19 . The non-transitory machine readable medium of claim 18 , wherein the streaming multimodal video generation model includes:

a transformer-based model configured to create the prompt text using inputs based on the one or more outputs from the GMT model;

a diffusion model configured to generate an image frame based on the prompt text; and

a consistency control model configured to evaluate a contrastive loss using the prompt text and a previously generated image frame from the diffusion model.

20 . The non-transitory machine readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor of the electronic device to create the prompt text further include instructions that when executed cause the at least one processor of the electronic device to:

track the one or more user preferences over a period of time;

generate, using a behavior description model, a user behavior description summarizing the tracked one or more user preferences over the period of time; and

generate a script using the user behavior description.

Assignments (1)

Continuity (3)

References Cited (22)