Generating text-to-motion animations from partially annotated datasets
A two-stage approach for learning and generating an expressive text-to-motion animation from partially annotated datasets (T2M-X). In an example implementation, T2M-X builds a unified motion dataset based on partially annotated datasets. In the first stage, T2M-X uses the unified motion dataset to train three vector-quantized variational autoencoders (VQ-VAE) for body, hand, and face, respectively, and generate high-quality motion outputs. In the second stage, T2M-X uses the high-quality motion outputs to train a multi-indexing generative pre-trained transformer (GPT) model that includes motion consistency loss and sequence length consistency for learning and then generating coordinated and expressive animations.
1 . A method of generating an animation, comprising:
training a body variational autoencoder (VAE) using a unified motion dataset that is based on a plurality of partially annotated datasets, wherein the body VAE comprises a body encoder, a body codebook, and a body decoder, and wherein the body VAE generates a body motion vector;
training a hand VAE using the unified motion dataset, wherein the hand VAE comprises a hand encoder, a hand codebook, and a hand decoder, and wherein the hand VAE generates a hand gesture vector;
training a face VAE using the unified motion dataset, wherein the face VAE comprises a face encoder, a face codebook, and a face decoder, and wherein the face VAE generates a face expression vector;
training a generative pre-trained transformer (GPT) model using the vectors generated by each VAE and a plurality of text conditions;
generating a GPT body codebook comprising a next body motion token, a GPT hand codebook comprising a next hand gesture token, and a GPT face codebook comprising a next face expression token; and
generating an animation based on the GPT codebooks, the next tokens, and a text input.
2 . The method of claim 1 , further comprising:
selecting a plurality of partially annotated datasets, each comprising at least one of a text prompt, a body motion dataset, a hand gesture dataset, or a face expression dataset; and
building the unified motion dataset in a standard format based on the selected datasets.
3 . The method of claim 1 , wherein training the GPT model comprises:
generating, for each of the plurality of text conditions, a concatenated text comprising a previous body motion token, a previous hand gesture token, a previous face expression token, wherein the next tokens are conditioned upon the previous tokens.
4 . The method of claim 1 , wherein generating the GPT codebooks comprises:
generating the GPT body codebook using a body GPT branch;
generating the GPT hand codebook using a hand GPT branch;
generating the GPT face codebook using a face GPT branch; and
providing a single joint space for processing data associated with the body GPT branch, the hand GPT branch, and the face GPT branch.
5 . The method of claim 4 , wherein training the GPT model comprises:
estimating in the single joint space a body part motion consistency loss, wherein the body part motion consistency loss is based on a contrastive loss, and a relative loss value associated with each of the GPT branches; and
imposing a sequence length consistency sampler across the GPT codebooks.
6 . The method of claim 1 , wherein training the body VAE, training the hand VAE, and training the face VAE comprises:
optimizing each VAE based on a final loss, wherein the final loss comprises a reconstruction loss, a codebook alignment loss, and a commitment loss.
7 . The method of claim 1 , wherein training the GPT model comprises:
calculating a GPT loss comprising three cross-entropy loss values, a weighted hand loss value, and a weighted face loss value.
8 . The method of claim 1 , wherein training the GPT model comprises:
generating a pose representation comprising a set of joint position vectors and a set of joint rotation vectors for each record in the unified motion dataset;
measuring a jitter associated with at least one of the next body motion token, the next hand gesture token, or the next face expression token;
applying a low-pass filter to mitigate the jitter;
selectively animating an upper body portion using a predicted set of upper-body joint rotations expressed in terms relative to the pose representation; and
selectively animating a lower body portion using a reversed set of lower-body joint rotations expressed in terms relative to the pose representation.
9 . A system for generating an animation, comprising:
a body variational autoencoder (VAE) trained using a unified motion dataset that is based on a plurality of partially annotated datasets, wherein the body VAE comprises a body encoder, a body codebook, and a body decoder, and wherein the body VAE generates a body motion vector;
a hand VAE trained using the unified motion dataset, wherein the hand VAE comprises a hand encoder, a hand codebook, and a hand decoder, and wherein the hand VAE generates a hand gesture vector; and
a face VAE trained using the unified motion dataset, wherein the face VAE comprises a face encoder, a face codebook, and a face decoder, and wherein the face VAE generates a face expression vector;
a multi-indexing generative pre-trained transformer (GPT) model trained using the vectors generated by each VAE and a plurality of text conditions;
a GPT body codebook comprising a next body motion token, a GPT hand codebook comprising a next hand gesture token, and a GPT face codebook comprising a next face expression token; and
an animation based on the GPT codebooks, the next tokens, and a text input.
10 . The system of claim 9 , wherein each of the plurality of partially annotated datasets comprises at least one of a text prompt, a body motion dataset, a hand gesture dataset, or a face expression dataset.
11 . The system of claim 9 , comprising:
a text encoder for generating, for each of the plurality of text conditions, a concatenated text comprising a previous body motion token, a previous hand gesture token, a previous face expression token, wherein the next tokens are conditioned upon the previous tokens.
12 . The system of claim 9 , comprising:
a body GPT branch for generating the GPT body codebook;
a hand GPT branch for generating the GPT hand codebook;
a face GPT branch for generating the GPT face codebook;
a single joint space for processing data associated with the body GPT branch, the hand GPT branch, and the face GPT branch.
13 . The system of claim 12 , comprising:
a body part motion consistency loss based on a contrastive loss and a relative loss value associated with each of the GPT branches; and
a sequence length consistency sampler imposed across the GPT codebooks.
14 . The system of claim 9 , comprising:
a final loss for optimizing each VAE, wherein the final loss comprises a reconstruction loss, a codebook alignment loss, and a commitment loss.
15 . The system of claim 9 , comprising:
a GPT loss associated with the GPT model, wherein the GPT loss comprises three cross-entropy loss values, a weighted hand loss value, and a weighted face loss value.
16 . The system of claim 9 , comprising:
a jitter detector for identifying a jitter associated with at least one of the next body motion token, the next hand gesture token, or the next face expression token;
a low-pass filter for mitigating the jitter;
an animation modification comprising selectively animating an upper body portion using a predicted set of upper-body joint rotations and selectively animating a lower body portion using a reversed set of lower-body joint rotations.
17 . A non-transitory computer-readable medium including instructions for generating an animation, the instructions, when executed by a processor, configure the processor to perform functions including:
selecting a plurality of partially annotated datasets, each comprising at least one of a text prompt, a body motion dataset, a hand gesture dataset, or a face expression dataset;
building a unified motion dataset in a standard format based on the selected datasets;
training a body variational autoencoder (VAE) using the unified motion dataset, wherein the body VAE comprises a body encoder, a body codebook, and a body decoder, and wherein the body VAE generates a body motion vector;
training a hand VAE using the unified motion dataset, wherein the hand VAE comprises a hand encoder, a hand codebook, and a hand decoder, and wherein the hand VAE generates a hand gesture vector;
training a face VAE using the unified motion dataset, wherein the face VAE comprises a face encoder, a face codebook, and a face decoder, and wherein the face VAE generates a face expression vector;
training a generative pre-trained transformer (GPT) model using the vectors generated by each VAE and a plurality of text conditions;
generating a GPT body codebook comprising a next body motion token, a GPT hand codebook comprising a next hand gesture token, and a GPT face codebook comprising a next face expression token; and
generating an animation based on the GPT codebooks, the next tokens, and a text input.
18 . The medium of claim 17 , wherein training the GPT model comprises:
generating, for each of the plurality of text conditions, a concatenated text comprising a previous body motion token, a previous hand gesture token, a previous face expression token, wherein the next tokens are conditioned upon the previous tokens.
19 . The medium of claim 17 , wherein training the GPT model comprises:
estimating in a single joint space a body part motion consistency loss, wherein the body part motion consistency loss is based on a contrastive loss, and a relative loss value associated with each of a body GPT branch, a hand GPT branch, and a face GPT branch.
20 . The medium of claim 17 , wherein training the body VAE, training the hand VAE, and training the face VAE comprises optimizing each VAE based on a final loss, wherein the final loss comprises a reconstruction loss, a codebook alignment loss, and a commitment loss, and
wherein training the GPT model comprises calculating a GPT loss comprising three cross-entropy loss values, a weighted hand loss value, and a weighted face loss value.