Deep multi-scale video prediction
In one embodiment, a method includes receiving a plurality of input frames of a video sequence associated with a time t, training a convolutional network to predict one or more future frames of the video sequence from the plurality of input frames based on a generative model, and outputting a first future frame of the video sequence associated with a time t+1 as predicted by the generative model. The training may comprise using an adversarial model and an image gradient difference loss model. In addition, the training may comprise randomly selecting temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow.
1. A method comprising:
by a computing device, receiving a plurality of input frames of a video sequence associated with a time t;
by the computing device, training a convolutional network to predict one or more future frames of the video sequence from the plurality of input frames based on a generative model,
wherein the training comprises using an adversarial model and an image gradient difference loss model, and
wherein the training comprises randomly selecting temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow; and
by the computing device, outputting a first future frame of the video sequence associated with a time t+1 as predicted by the generative model.
2. The method of claim 1 , further comprising:
by the computing device, predicting a second future frame of the video sequence associated with time t+2 by
using the first future frame as an input frame of the video sequence,
recursively apply the generative model based on the first future frame, and
outputting the second future frame of the video sequence associated with the time t+2 as predicted by the generative model.
3. The method of claim 1 , wherein threshold of optical flow is determined based on a comparison of movement in the n×m grid of pixels between a time t−1 and the time t.
4. The method of claim 3 , wherein the comparison of movement comprises comparing pixel intensity associated with each pixel of the n×m grid of pixels between the time t−1 and the time t and calculating a similarity measure based on compared pixel intensities.
5. The method of claim 1 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a peak-signal-to-noise ratio between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
6. The method of claim 1 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a structural similarity index measure between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
7. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive a plurality of input frames of a video sequence associated with a time t;
train a convolutional network to predict one or more future frames of the video sequence from the plurality of input frames based on a generative model,
wherein the training comprises using an adversarial model and an image gradient difference loss model, and
wherein the training comprises randomly selecting temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow; and
output a first future frame of the video sequence associated with a time t+1 as predicted by the generative model.
8. The media of claim 7 , wherein the software is further operable when executed to:
predict a second future frame of the video sequence associated with time t+2 by
using the first future frame as an input frame of the video sequence,
recursively apply the generative model based on the first future frame, and
outputting the second future frame of the video sequence associated with the time t+2 as predicted by the generative model.
9. The media of claim 7 , wherein threshold of optical flow is determined based on a comparison of movement in the n×m grid of pixels between a time t−1 and the time t.
10. The media of claim 9 , wherein the comparison of movement comprises comparing pixel intensity associated with each pixel of the n×m grid of pixels between the time t−1 and the time t and calculating a similarity measure based on compared pixel intensities.
11. The media of claim 7 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a peak-signal-to-noise ratio between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
12. The media of claim 7 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a structural similarity index measure between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
13. A system comprising:
one or more processors; and
one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
receive a plurality of input frames of a video sequence associated with a time t;
train a convolutional network to predict one or more future frames of the video sequence from the plurality of input frames based on a generative model,
wherein the training comprises using an adversarial model and an image gradient difference loss model, and
wherein the training comprises randomly selecting temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow; and
output a first future frame of the video sequence associated with a time t+1 as predicted by the generative model.
14. The system of claim 13 , wherein the processors are further operable when executing the instructions to:
predict a second future frame of the video sequence associated with time t+2 by
using the first future frame as an input frame of the video sequence,
recursively apply the generative model based on the first future frame, and
outputting the second future frame of the video sequence associated with the time t+2 as predicted by the generative model.
15. The system of claim 13 , wherein threshold of optical flow is determined based on a comparison of movement in the n×m grid of pixels between a time t−1 and the time t.
16. The system of claim 15 , wherein the comparison of movement comprises comparing pixel intensity associated with each pixel of the n×m grid of pixels between the time t−1 and the time t and calculating a similarity measure based on compared pixel intensities.
17. The system of claim 13 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a peak-signal-to-noise ratio between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
18. The system of claim 13 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a structural similarity index measure between the one or more future frames of the video sequence and a ground truth frame of the video sequence.