Deep multi-scale video prediction
In one embodiment, a method includes receiving a plurality of input frames of a video sequence associated with a time t, predicting, using a convolutional network, one or more future frames of the video sequence from the plurality of input frames, wherein the convolutional network is trained with randomly selected temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow. In addition, the training may comprise randomly selecting temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow.
1. A method comprising, by one or more computing devices:
receiving a plurality of input frames of a video sequence associated with a time t;
predicting, using a convolutional network, one or more future frames of the video sequence from the plurality of input frames, wherein the convolutional network is trained with randomly selected temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow; and
outputting a first future frame of the video sequence associated with a time t+1 as predicted by the convolutional network.
2. The method of claim 1 , further comprising by the one or more computing devices:
predicting, using the convolutional network, a second future frame of the video sequence associated with time t+2 by using the first future frame as an input frame of the video sequence;
recursively applying the convolutional network based on the first future frame; and
outputting the second future frame of the video sequence associated with the time t+2 as predicted by the convolutional network.
3. The method of claim 1 , wherein the threshold of optical flow is determined based on a comparison of movement in the n×m grid of pixels between a time t−1 and the time t.
4. The method of claim 3 , wherein the comparison of movement comprises comparing pixel intensity associated with each pixel of the n×m grid of pixels between the time t−1 and the time t and calculating a similarity measure based on compared pixel intensities.
5. The method of claim 1 , wherein the convolutional network is further trained by evaluating a quality of the predicted one or more future frames of the video sequence by calculating a peak-signal-to-noise ratio between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
6. The method of claim 1 , wherein the convolutional network is further trained by evaluating a quality of the predicted one or more future frames of the video sequence by calculating a structural similarity index measure between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
7. The method of claim 1 , wherein the convolutional network employs an L 1 loss to train the network on the input frames.
8. The method of claim 1 , wherein the convolutional network employs an L 2 loss to train the network on the input frames.
9. The method of claim 1 , wherein the convolutional network employs a gradient difference loss to train the network on the input frames.
10. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive a plurality of input frames of a video sequence associated with a time t;
predict, using a convolutional network, one or more future frames of the video sequence from the plurality of input frames, wherein the convolutional network is trained with randomly selected temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow; and
output a first future frame of the video sequence associated with a time t+1 as predicted by the convolutional network.
11. The media of claim 10 , wherein threshold of optical flow is determined based on a comparison of movement in the n×m grid of pixels between a time t−1 and the time t.
12. The media of claim 11 , wherein the comparison of movement comprises comparing pixel intensity associated with each pixel of the n×m grid of pixels between the time t−1 and the time t and calculating a similarity measure based on compared pixel intensities.
13. The media of claim 10 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a peak-signal-to-noise ratio between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
14. The media of claim 10 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a structural similarity index measure between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
15. A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
receive a plurality of input frames of a video sequence associated with a time t;
predict, using a convolutional network, one or more future frames of the video sequence from the plurality of input frames, wherein the convolutional network is trained with randomly selected temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow; and
output a first future frame of the video sequence associated with a time t+1 as predicted by the convolutional network.
16. The system of claim 15 , wherein the processors are further operable when executing the instructions to:
predict a second future frame of the video sequence associated with time t+2 by using the first future frame as an input frame of the video sequence,
recursively apply the generative model based on the first future frame, and
outputting the second future frame of the video sequence associated with the time t+2 as predicted by the generative model.
17. The system of claim 15 , wherein threshold of optical flow is determined based on a comparison of movement in the n×m grid of pixels between a time t−1 and the time t.
18. The system of claim 17 , wherein the comparison of movement comprises comparing pixel intensity associated with each pixel of the n×m grid of pixels between the time t−1 and the time t and calculating a similarity measure based on compared pixel intensities.
19. The system of claim 15 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a peak-signal-to-noise ratio between the one or more future frames of the video sequence and a ground truth frame of the video sequence.
20. The system of claim 15 , wherein the training of the convolutional network may further comprise evaluating a quality of the predicted one or more future frames of the video sequence by calculating a structural similarity index measure between the one or more future frames of the video sequence and a ground truth frame of the video sequence.