Applying artificial intelligence to generate motion information
This disclosure describes techniques that include generating, based on a description of a scene, a movie or animation that represents at least one possible version of a story corresponding to the description of the scene. This disclosure also describes techniques for training a machine learning model to generate predefined data structures from textual information, visual information, and/or other information about a story, an event, a scene, or a sequence of events or scenes within a story. This disclosure also describes techniques for using GANs to generate, from input, an animation of motion (e.g., an animation or a video clip). This disclosure also describes techniques for implementing an explainable artificial intelligence system that may provide end users with information (e.g., through a user interface) that enables an understanding of at least some of the decisions made by the AI system.
1. A method for generating information about motion, comprising:
receiving, by a computing system, text input indicating motion by an actor;
applying, by the computing system, a generative adversarial network (GAN) model trained using a training set that includes abstracted animations of motion that include less data than pixel-level representations of motion, to the text input to generate motion information, including motion sequence, without conditioning, the motion information representing the motion by the actor, wherein the GAN model is trained using less computational resources than a GAN model trained using pixel-level representations of motion; and
outputting, by the computing system, the motion information.
2. The method of claim 1 , wherein motion information includes at least one of: skeleton movement, joint angles between body parts, a mesh of a human body, a facial expression, movement of body landmarks, movement of facial feature landmarks.
3. The method of claim 1 ,
wherein applying the GAN model includes applying a GAN model trained to generate a predicted abstracted animation of motion from the text input indicating motion; and
wherein outputting the motion information includes outputting an abstracted animation of motion representing the motion by the actor.
4. The method of claim 3 ,
wherein the GAN uses at least one of a convolutional neural network (CNN) and a recurrent neural network (RNN) to generate the motion information; and
wherein the GAN combines the CNN and the RNN for motion completion.
5. The method of claim 4 , wherein the GAN includes a gradient penalty.
6. The method of claim 5 , wherein the generative adversarial network is a dense validation generative adversarial network (DVGAN).
7. The method of claim 6 , wherein training the GAN includes:
training the GAN using a discriminator that produces scores at a dense frequency of time resolutions and at a dense frequency of frames.
8. The method of claim 1 ,
wherein applying the GAN model includes applying a GAN model trained by evaluating each of the animations over a set of time resolutions and over a set of frame frequencies.
9. The method of claim 1 ,
wherein training the GAN includes producing a GAN validation score at each of the time resolutions and at each of the frame frequencies.
10. The method of claim 1 ,
wherein the abstracted animations of motion are derived from video clips drawn from movies.
11. A system comprising:
a storage device; and
processing circuitry having access to the storage device and configured to:
receive text input indicating motion by an actor,
apply a generative adversarial network (GAN) model, trained using a training set that includes abstracted animations of motion that include less data than pixel-level representations of motion, to the text input to generate motion information including motion sequence representing the motion by the actor, wherein the GAN model is trained using less computational resources than a GAN model trained using pixel-level representations of motion, and
output the motion information.
12. The system of claim 11 , wherein motion information includes at least one of: skeleton movement, joint angles between body parts, a mesh of a human body, a facial expression, movement of body landmarks, movement of facial feature landmarks.
13. The system of claim 11 , wherein to apply the GAN, the processing circuitry is further configured to:
apply a GAN model trained by evaluating each of the animations over a set of time resolutions and over a set of frame frequencies.
14. The system of claim 11 , wherein to apply the GAN model, the processing circuitry is further configured to:
apply a GAN model trained to generate a predicted abstracted animation of motion from the text input indicating motion; and
wherein outputting the motion information includes outputting an abstracted animation of motion representing the motion by the actor.
15. The system of claim 14 ,
wherein the GAN uses at least one of a convolutional neural network (CNN) and a recurrent neural network (RNN) to generate the motion information; and
wherein the GAN combines the CNN and the RNN for motion completion.
16. The system of claim 15 , wherein the GAN includes a Wasserstein GAN with a gradient penalty.
17. The system of claim 16 , wherein the generative adversarial network is a dense validation generative adversarial network (DVGAN).
18. The system of claim 16 , wherein to train the GAN, the processing circuitry is further configured to:
use a discriminator that produces scores at a dense frequency of time resolutions and at a dense frequency of frames.
19. A non-transitory computer-readable storage medium comprising instructions that, when executed, configure processing circuitry of a computing system to:
receive text input indicating motion by an actor;
apply a generative adversarial network (GAN) model, trained using a training set that includes abstracted animations of motion and trained by evaluating each of the abstracted animations of motion over a set of time resolutions and over a set of frame frequencies, to the text input to generate motion information including motion sequence representing the motion by the actor; and
output the motion information.