Video generating device and method
A video generating device and method are provided. The device analyzes a plurality of real-time images corresponding to a plurality of users to segment a target image from each of the real-time images. The device generates a three-dimensional portrait model corresponding to each of the users based on the target image of each of the real-time images. The device determines a first three-dimensional scenario template from the three-dimensional scenario templates based on a user quantity of the users and the position quantity corresponding to each of the three-dimensional scenario templates. The device composites the three-dimensional portrait models to the spatial label position of the first three-dimensional scenario template to generate a video corresponding to the users.
1 . A video generating device, comprising:
a transceiver interface;
a storage, being configured to store a plurality of three-dimensional scenario templates, wherein each of the three-dimensional scenario templates corresponds to a position quantity and a spatial label position; and
a processor, being electrically connected to the transceiver interface and the storage, and being configured to perform operations comprising:
analyzing a plurality of real-time images corresponding to a plurality of users to segment a target image from each of the real-time images;
generating a three-dimensional portrait model corresponding to each of the users based on the target image of each of the real-time images;
determining a first three-dimensional scenario template from the three-dimensional scenario templates based on a user quantity of the users and the position quantity corresponding to each of the three-dimensional scenario templates; and
compositing the three-dimensional portrait models to the spatial label position of the first three-dimensional scenario template to generate a video corresponding to the users;
wherein the operation of generating the video corresponding to the users further comprises the following operations:
inputting an environment parameter of the first three-dimensional scenario template, the three-dimensional portrait models, and the spatial label position of the first three-dimensional scenario template into a pre-trained diffusion model to generate the video corresponding to the users;
wherein, the environment parameter comprises a lighting indication vector and a pose attribute vector.
2 . The video generating device of claim 1 , wherein the first three-dimensional scenario template further comprises an environment parameter, and the processor further performs the following operations:
rendering the three-dimensional portrait models based on the environment parameter to generate the video corresponding to the users.
3 . The video generating device of claim 1 , wherein the spatial label position of the first three-dimensional scenario template further corresponds to a portrait pose setting, and the processor further performs the following operations:
compositing the three-dimensional portrait model corresponding to the portrait pose setting to the spatial label position of the first three-dimensional scenario template to generate the video corresponding to the user.
4 . The video generating device of claim 1 , wherein the first three-dimensional scenario template further comprises a plurality of spatial perspectives, and the processor further performs the following operations:
generating a perspective video corresponding to each of the spatial perspectives based on the spatial perspectives; and
transmitting the perspective videos to a playing device based on a perspective switching mechanism to make the playing device performs a playing operation.
5 . The video generating device of claim 4 , wherein the perspective switching mechanism is a speaking position priority, a random playing, or a round-robin playing.
6 . The video generating device of claim 1 , wherein the three-dimensional scenario templates are generated based on the following operations:
inputting a plurality of two-dimensional images and a description text corresponding to each of the two-dimensional images into a depth model to generate the three-dimensional scenario templates, wherein the depth model is trained by a plurality of scene depth maps.
7 . The video generating device of claim 1 , wherein the processor is further configured to perform the following operations:
rendering the three-dimensional portrait models in the video in real-time based on the target image of each of the real-time images to update the video.
8 . The video generating device of claim 1 , wherein the operation of segmenting the target image further comprises the following operations:
generating an edge block information corresponding to each of the target images based on an edge state of each of the target images; and
compositing the edge block information and the three-dimensional portrait models to the spatial label position of the first three-dimensional scenario template to generate the video corresponding to the users.
9 . The video generating device of claim 1 , wherein the processor is further configured to perform the following operations:
receiving a perspective switching signal corresponding to a first user, wherein the perspective switching signal is configured to indicate switching to a first spatial perspective; and
generating a first perspective video corresponding to the first spatial perspective based on the perspective switching signal.
10 . A video generating method, being adapted for use in an electronic apparatus, wherein the electronic apparatus stores a plurality of three-dimensional scenario templates, each of the three-dimensional scenario templates corresponds to a position quantity and a spatial label position, and the video generating method comprises:
analyzing a plurality of real-time images corresponding to a plurality of users to segment a target image from each of the real-time images;
generating a three-dimensional portrait model corresponding to each of the users based on the target image of each of the real-time images;
determining a first three-dimensional scenario template from the three-dimensional scenario templates based on a user quantity of the users and the position quantity corresponding to each of the three-dimensional scenario templates; and
compositing the three-dimensional portrait models to the spatial label position of the first three-dimensional scenario template to generate a video corresponding to the users;
wherein the step of generating the video corresponding to the users further comprises the following steps:
inputting an environment parameter of the first three-dimensional scenario template, the three-dimensional portrait models, and the spatial label position of the first three-dimensional scenario template into a pre-trained diffusion model to generate the video corresponding to the users;
wherein, the environment parameter comprises a lighting indication vector and a pose attribute vector.
11 . The video generating method of claim 10 , wherein the first three-dimensional scenario template further comprises an environment parameter, and the video generating method comprises:
rendering the three-dimensional portrait models based on the environment parameter to generate the video corresponding to the users.
12 . The video generating method of claim 10 , wherein the spatial label position of the first three-dimensional scenario template further corresponds to a portrait pose setting, and the video generating method further comprises the following steps:
compositing the three-dimensional portrait model corresponding to the portrait pose setting to the spatial label position of the first three-dimensional scenario template to generate the video corresponding to the user.
13 . The video generating method of claim 10 , wherein the first three-dimensional scenario template further comprises a plurality of spatial perspectives, and the video generating method further comprises the following steps:
generating a perspective video corresponding to each of the spatial perspectives based on the spatial perspectives; and
transmitting the perspective videos to a playing device based on a perspective switching mechanism to make the playing device performs a playing operation.
14 . The video generating method of claim 13 , wherein the perspective switching mechanism is a speaking position priority, a random playing, or a round-robin playing.
15 . The video generating method of claim 10 , wherein the three-dimensional scenario templates are generated based on the following steps:
inputting a plurality of two-dimensional images and a description text corresponding to each of the two-dimensional images into a depth model to generate the three-dimensional scenario templates, wherein the depth model is trained by a plurality of scene depth maps.
16 . The video generating method of claim 10 , wherein the video generating method further comprises the following steps:
rendering the three-dimensional portrait models in the video in real-time based on the target image of each of the real-time images to update the video.
17 . The video generating method of claim 10 , wherein the step of segmenting the target image further comprises the following steps:
generating an edge block information corresponding to each of the target images based on an edge state of each of the target images; and
compositing the edge block information and the three-dimensional portrait models to the spatial label position of the first three-dimensional scenario template to generate the video corresponding to the users.
18 . The video generating method of claim 10 , wherein the video generating method further comprises the following steps:
receiving a perspective switching signal corresponding to a first user, wherein the perspective switching signal is configured to indicate switching to a first spatial perspective; and
generating a first perspective video corresponding to the first spatial perspective based on the perspective switching signal.