Multi-scale text filter conditioned generative adversarial networks
Systems and methods for processing video are provided. The method includes receiving a text-based description of active scenes and representing the text-based description as a word embedding matrix. The method includes using a text encoder implemented by neural network to output frame level textual representation and video level representation of the word embedding matrix. The method also includes generating, by a shared generator, frame by frame video based on the frame level textual representation, the video level representation and noise vectors. A frame level and a video level convolutional filter of a video discriminator are generated to classify frames and video of the frame by frame video as true or false. The method also includes training a conditional video generator that includes the text encoder, the video discriminator, and the shared generator in a generative adversarial network to convergence.
1. A method for implementing processing video, comprising:
receiving a text-based description of at least one active scene;
representing, by a processor device, the text-based description as a word embedding matrix;
using a text encoder implemented by neural network to output at least one frame level textual representation and at least one video level representation of the word embedding matrix;
generating, by a shared generator, at least one frame by frame video based on the at least one frame level textual representation, the at least one video level representation and noise vectors sampled from a Gaussian distribution;
generating a frame level and a video level convolutional filter of a video discriminator to classify frames and video of the at least one frame by frame video as true or false;
optimizing the text encoder T, based on applying:
max
T
L
T
=
E
(
v
,
t
)
~
P
data
,
real
log
(
D
(
v
,
T
(
t
)
)
)
+
E
(
v
,
t
)
~
P
data
,
fake
log
(
1
-
D
(
v
,
T
(
t
)
)
)
Wherein
max
T
L
T
is a maximum loss, p data,real denotes a real data distribution with correct video-text correspondences, p data,fake denotes a distribution with incorrect video-text correspondences, v is the video level representation, t is the frame level textual representation, and D is a discriminator; and
training a conditional video generator that includes the text encoder, the video discriminator, and the shared generator in a generative adversarial network to convergence.
2. The method as recited in claim 1 , further comprising:
using the trained conditional video generator to generate at least one further video from additional text.
3. The method as recited in claim 2 , further comprising:
generating the at least one further video for an input to a process selected from a group consisting of multimedia applications, generating synthetic datasets, model-based reinforcement learning systems, and domain adaptation.
4. The method as recited in claim 2 , further comprising:
training an artificial intelligence system based on the at least one further video.
5. The method as recited in claim 4 , wherein the artificial intelligence system is selected from a group consisting of a control system for self-driving cars and a surveillance system.
6. The method as recited in claim 2 , wherein using the trained conditional video generator to generate at least one further video further comprises:
producing a variable length video.
7. The method as recited in claim 1 , wherein the at least one active scene includes a text-based dangerous traffic scene description.
8. The method as recited in claim 1 , wherein the at least one active scene includes a suspicious scene description.
9. The method as recited in claim 1 , further comprising:
determining a photo-realistic video synthesis using a deeper generator-discriminator.
10. The method as recited in claim 1 , wherein the shared frame generator network further comprises:
a motion and content decomposed generative adversarial network (MoCoGAN) that generates a video by mapping a sequence of random vectors to a sequence of video frames.
11. A computer system for processing video, comprising:
a processor device operatively coupled to a memory device, the processor device being configured to:
receive a text-based description of at least one active scene;
represent the text-based description as a word embedding matrix; and
use a text encoder implemented by neural network to output at least one frame level textual representation and at least one video level representation of the word embedding matrix;
generate, by a shared generator, at least one frame by frame video based on the at least one frame level textual representation, the at least one video level representation and noise vectors sampled from a Gaussian distribution;
optimize the text encoder T, based on applying:
max
T
L
T
=
E
(
v
,
t
)
~
p
data
,
real
log
(
D
(
v
,
T
(
t
)
)
)
+
E
(
v
,
t
)
~
p
data
,
fake
log
(
1
-
D
(
v
,
T
(
t
)
)
)
wherein
max
T
L
T
is a maximum loss, p data,real denotes a real data distribution with correct video-text correspondences, p data,fake denotes a distribution with incorrect video-text correspondences, v is the video level representation, t is the frame level textual representation, and D is a discriminator; and
train a conditional video generator that includes the text encoder, the video discriminator, and the shared generator in a generative adversarial network to convergence.
12. The system as recited in claim 11 , wherein the processor device is further configured to:
use the trained conditional video generator to generate at least one further video from additional text.
13. The system as recited in claim 12 , wherein the processor device is further configured to:
generate the at least one further video for an input to a process selected from a group consisting of multimedia applications, generating synthetic datasets, model-based reinforcement learning systems, and domain adaptation.
14. The system as recited in claim 12 , wherein the processor device is further configured to:
train an artificial intelligence system based on the at least one further video.
15. The system as recited in claim 14 , wherein the artificial intelligence system is selected from a group consisting of a control system for self-driving cars and a surveillance system.
16. The system as recited in claim 12 , wherein, when using the trained conditional video generator to generate at least one further video, the processor device is further configured to:
produce a variable length video.
17. The system as recited in claim 11 , wherein the at least one active scene includes a text-based dangerous traffic scene description.
18. The system as recited in claim 11 , wherein the at least one active scene includes a suspicious scene description.
19. A computer program product for implementing a text filter generative adversarial network (TF-GAN), the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to perform the method comprising:
receiving a text-based description of at least one active scene;
representing, by a processor device, the text-based description as a word embedding matrix;
using a text encoder implemented by neural network to output at least one frame level textual representation and at least one video level representation of the word embedding matrix;
generating, by a shared generator, at least one frame by frame video based on the frame level textual representation, the video level representation and noise vectors sampled from a Gaussian distribution;
generating a frame level and a video level convolutional filter of a video discriminator to classify frames and video of the at least one frame by frame video as true or false;
optimizing the text encoder T, based on applying:
max
T
L
T
=
E
(
v
,
t
)
~
P
data
,
real
log
(
D
(
v
,
T
(
t
)
)
)
+
E
(
v
,
t
)
~
P
data
,
fake
log
(
1
-
D
(
v
,
T
(
t
)
)
)
wherein
max
T
L
T
.
is a maximum loss, p data,real denotes a real data distribution with correct video-text correspondences, p data,fake denotes a distribution with incorrect video-text correspondences, v is the video level representation, t is the frame level textual representation, and D is a discriminator; and
training a conditional video generator that includes the text encoder, the video discriminator, and the shared generator in a generative adversarial network to convergence.