Granted Patent B2

US 12,664,742 · App. 18/224,550 · Granted Jun 23, 2026

Inventors: Mária Virčiková (Košice, SK); Matúš Kirchmayer (Košice, SK); Rudolf Jakša (Košice, SK)

Assignee: MATSUKO S.R.O.

G06T19/20G06T7/50G06V10/54G06V40/174G06V40/20H04N7/157G06T2219/2021

Abstract

Claims (54)

1 . A method comprising:

obtaining visual modeling data and verbal communication data of a conference participant in a current time frame as a current sample;

encoding the current sample into a first vector comprising a plurality of states for the current sample as an encoded result, wherein the plurality of states comprises one or more of texture, shape, eye behavior, mouth and facial expression, head movement, landmark position, speech, or script, and wherein each of the plurality of states is characterized by a plurality of elements;

updating at least one of the plurality of states by inputting the encoded result to a generative artificial intelligence model as a prompt;

selecting the visual modeling data of the conference participant in a plurality of past time frames as past samples;

encoding the past samples into a second plurality of vectors representing the plurality of states for the past samples;

updating at least one of the plurality of states by inputting the second plurality of vectors to the generative artificial intelligence model as additional prompts;

obtaining the updated plurality of states as a predicted result; and

updating the visual modeling data based on the predicted result for rendering a three-dimensional (3D) representation of the conference participant in the current time frame or in one or more future time frames.

2 . The method of claim 1 , wherein updating the visual modeling data comprises inputting the predicted result to a reconstruction or completion artificial neural network model to obtain the updated visual modeling data for rendering the 3D representation of the conference participant in the current time frame or in the one or more future time frames.

3 . The method of claim 1 , wherein updating the visual modeling data comprises decoding the predicted result into the updated visual modeling data for rendering the 3D representation of the conference participant in the current time frame or in the one or more future time frames.

4 . The method of claim 1 , wherein the predicted result has a same format as the encoded result.

5 . The method of claim 1 , wherein the visual modeling data comprises one or more of texture data, shape data, facial expression data, head movement data, or eyes behavior data, and the verbal communication data comprises one or more of audio data or text data.

6 . The method of claim 1 , wherein updating at least one of the plurality of states comprises updating one or more of texture, shape, eye behavior, or mouth and facial expression of the conference participant where a portion of the conference participant's head is obstructed.

7 . The method of claim 1 , wherein the plurality of past time frames are sampled at a constant frequency.

8 . The method of claim 1 , wherein the plurality of past time frames are irregularly sampled.

9 . The method of claim 1 , further comprising, prior to encoding the past samples into the second plurality of vectors:

obtaining at least one of the visual modeling data or the verbal communication data of additional one or more conference participants in the plurality of past time frames; and

adding the at least one of the visual modeling data or the verbal communication data of the additional one or more conference participants in the plurality of past time frames into the past samples.

10 . The method of claim 1 , further comprising, prior to encoding the current sample into a first vector:

obtaining at least one of the visual modeling data or the verbal communication data of additional one or more conference participants in the current time frame; and

adding the at least one of the visual modeling data or the verbal communication data of additional one or more conference participants in the current time frame into the current sample.

11 . The method of claim 1 , further comprising:

encoding the first vector into a sequence of token values; and

updating the encoded result with the sequence of token values, such that the encoded result is compatible with the generative artificial intelligence model being a transformer model.

12 . The method of claim 1 , further comprising:

obtaining a sequence of video frames captured by a camera for the conference participant, the sequence of video frames comprising two-dimensional (2D) image data, depth data, and head alignment data; and

reconstructing, for the current time frame of the sequence of video frames, the visual modeling data by projecting the 2D image data from a world space into an object space using the head alignment data and depth data.

13 . The method of claim 12 , further comprising completing at least one missing part of the reconstructed visual modeling data by a pre-trained artificial neural network.

14 . The method of claim 1 , wherein the 3D representation of the conference participant comprises one or more of a hologram, a parametric human representation, a stereoscopic 3D representation, a volumetric representation, a mesh-based representation, a point-cloud based representation, a radiance field representation, or a hybrid representation.

15 . The method of claim 1 , wherein the generative artificial intelligence model is selected from one of transformer models, Stable Diffusion models, Generative Adversarial Networks, or autoencoders.

16 . The method of claim 1 , wherein the plurality of states comprises eye behavior, and the eye behavior comprises eye-blink behavior, and wherein the generative artificial intelligence model is trained based on the current time frame or a plurality of past training time frames for the conference participant that are capable of being encoded into a plurality of training vectors comprising a plurality of training states, the plurality of training states comprising the eye-blink behavior and at least one of facial expression or speech pattern, such that the trained generative artificial intelligence model is capable of generating predicted future states of eye-blink behavior based on the facial expression or the speech pattern.

17 . The method of claim 16 , wherein the generative artificial intelligence is trained to allow a mapping between the eye-blink behavior and at least one of the facial expression or the speech pattern.

18 . A system comprising:

a network interface;

a processor communicatively coupled to the network interface; and

a non-transitory computer readable medium communicatively coupled to the processor and having stored thereon computer program code that is executable by the processor and that, when executed by the processor, causes the processor to perform a method comprising:

obtaining visual modeling data and verbal communication data of a conference participant in a current time frame as a current sample;

encoding the current sample into a first vector comprising a plurality of states for the current sample as an encoded result, wherein the plurality of states comprises one or more of texture, shape, facial expression, head movement, landmark position, speech, or script, and wherein each of the plurality of states is characterized by a plurality of elements;

updating at least one of the plurality of states by inputting the encoded result to a generative artificial intelligence model as a prompt;

selecting the visual modeling data of the conference participant in a plurality of past time frames as past samples;

encoding the past samples into a second plurality of vectors representing the plurality of states for the past samples;

updating at least one of the plurality of states by inputting the second plurality of vectors to the generative artificial intelligence model as additional prompts;

obtaining the updated plurality of states as a predicted result; and

updating the visual modeling data based on the predicted result for rendering a three-dimensional (3D) representation of the conference participant in the current time frame or in one or more future time frames.

19 . A non-transitory computer readable medium having encoded thereon computer program code that is executable by a processor and that, when executed by the processor, causes the processor to perform a method comprising:

obtaining visual modeling data and verbal communication data of a conference participant in a current time frame as a current sample;

encoding the current sample into a first vector comprising a plurality of states for the current sample as an encoded result, wherein the plurality of states comprises one or more of texture, shape, facial expression, head movement, landmark position, speech, or script, and wherein each of the plurality of states is characterized by a plurality of elements;

updating at least one of the plurality of states by inputting the encoded result to a generative artificial intelligence model as a prompt;

selecting the visual modeling data of the conference participant in a plurality of past time frames as past samples;

encoding the past samples into a second plurality of vectors representing the plurality of states for the past samples;

updating at least one of the plurality of states by inputting the second plurality of vectors to the generative artificial intelligence model as additional prompts;

obtaining the updated plurality of states as a predicted result; and

updating the visual modeling data based on the predicted result for rendering a three-dimensional (3D) representation of the conference participant in the current time frame or in one or more future time frames.

Assignments (1)

Continuity (1)

References Cited (30)