Multi-frame video interpolation using optical flow
Video interpolation is used to predict one or more intermediate frames at timesteps defined between two consecutive frames. A first neural network model approximates optical flow data defining motion between the two consecutive frames. A second neural network model refines the optical flow data and predicts visibility maps for each timestep. The two consecutive frames are warped according to the refined optical flow data for each timestep to produce pairs of warped frames for each timestep. The second neural network model then fuses the pair of warped frames based on the visibility maps to produce the intermediate frame for each timestep. Artifacts caused by motion boundaries and occlusions are reduced in the predicted intermediate frames.
1. A computer-implemented method, comprising:
receiving forward optical flow data and backward optical flow data computed for consecutive frames including a first frame and a second frame in a sequence of frames, wherein the sequence of frames does not include a frame at a time between the first frame and the second frame;
warping the first frame according to approximated forward optical flow data corresponding to the time;
warping the second frame according to approximated backward optical flow data corresponding to the time;
predicting, by a flow interpolation neural network model, occlusion data for the consecutive frames; and
generating, by the flow interpolation neural network model, an intermediate frame at the time based on the warped first frame, the warped second frame, and the occlusion data, wherein occluded pixels in the warped first and second frames, as indicated by the occlusion data, are excluded from contributing to the intermediate frame.
2. The computer-implemented method of claim 1 , further comprising interpolating, by the flow interpolation neural network model, between the forward optical flow data and the backward optical flow data to generate the approximated forward optical flow data and the approximated backward optical flow data.
3. The computer-implemented method of claim 2 , wherein the flow computation neural network model and the flow interpolation neural network model are convolutional neural network models.
4. The computer-implemented method of claim 1 , further comprising generating, by a flow computation neural network model, the forward optical flow data and the backward optical flow data based on the consecutive frames.
5. The computer-implemented method of claim 4 , wherein the flow computation neural network model and the flow interpolation neural network model are jointly trained in an unsupervised manner.
6. The computer-implemented method of claim 4 , wherein the flow computation neural network model and the flow interpolation neural network model are jointly trained using a loss function that minimizes perceptual loss.
7. The computer-implemented method of claim 4 , wherein the flow computation neural network model and the flow interpolation neural network model are jointly trained using a loss function that minimizes reconstruction loss.
8. The computer-implemented method of claim 4 , wherein the flow computation neural network model and the flow interpolation neural network model are jointly trained using a loss function that minimizes warping loss.
9. The computer-implemented method of claim 4 , wherein the flow computation neural network model and the flow interpolation neural network model are jointly trained using a loss function that minimizes smoothness loss.
10. The computer-implemented method of claim 1 , wherein the occlusion data comprises a forward visibility map and a backward visibility map that are predicted by the flow interpolation neural network model.
11. The computer-implemented method of claim 1 , further comprising:
receiving second approximated forward optical flow data corresponding to a second time in the sequence of frames that is between the consecutive frames;
receiving second approximated backward optical flow data corresponding to the second time; and
generating, by the flow interpolation neural network model, a second intermediate frame at the second time based on the first frame warped according to the second approximated forward optical flow data, the second frame warped according to the second approximated backward optical flow data, and the occlusion data, wherein occluded pixels in the first frame warped according to the second approximated forward optical flow data and the second frame warped according to the second approximated backward optical flow data, as indicated by the occlusion data indicates, are excluded from contributing to the second intermediate frame.
12. The computer-implemented method of claim 1 , wherein the flow interpolation neural network model generates the intermediate frame corresponding to the time by linearly fusing the warped first frame and the warped second frame.
13. The computer-implemented method of claim 12 , further comprising, before the linear fusing, applying a forward visibility map to the warped first frame and applying a backward visibility map to the warped second frame.
14. A system, comprising:
a frame interpolation system configured to:
receive forward optical flow data and backward optical flow data computed for consecutive frames including a first frame and a second frame in a sequence of frames, wherein the sequence of frames does not include a frame at a time between the first frame and the second frame;
warp the first frame according to approximated forward optical flow data corresponding to the time;
warping the second frame according to approximated backward optical flow data corresponding to the time;
predict occlusion data for the consecutive frames; and
predict, by a flow interpolation neural network model, occlusion data for the consecutive frames; and
generate, by the flow interpolation neural network model, an intermediate frame at the time based on the warped first frame, the warped second frame, and the occlusion data, wherein occluded pixels in the warped first and second frames, as indicated by the occlusion data, are excluded from contributing to the intermediate frame.
15. The system of claim 14 , wherein the frame interpolation system further comprises an intermediate optical flow neural network model configured to interpolate between the forward optical flow data and the backward optical flow data to generate the approximated forward optical flow data and the approximated backward optical flow data corresponding to the time.
16. The system of claim 14 , wherein the frame interpolation system further comprises a flow computation neural network model configured to generate the forward optical flow data and the backward optical flow data based on the consecutive frames.
17. The system of claim 16 , wherein the flow computation neural network model and the flow interpolation neural network model are jointly trained in an unsupervised manner.
18. The system of claim 14 , wherein the occlusion data comprises a forward visibility map and a backward visibility map that are predicted by the flow interpolation neural network model.
19. The system of claim 14 , wherein the flow interpolation neural network model generates the intermediate frame corresponding to the time by linearly fusing the warped first frame and the warped second frame.
20. A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to:
receive forward optical flow data and backward optical flow data computed for consecutive frames including a first frame and a second frame in a sequence of frames, wherein the sequence of frames does not include a frame at a time between the first frame and the second frame;
warp the first frame according to approximated forward optical flow data corresponding to the time;
warp the second frame according to approximated backward optical flow data corresponding to the time;
predict, by a flow interpolation neural network model, occlusion data for the consecutive frames; and
generate, by the flow interpolation neural network model, an intermediate frame at the time based on the warped first frame, and the warped second frame, and the occlusion data, wherein occluded pixels in the warped first and second frames, as indicated by the occlusion data, are excluded from contributing to the intermediate frame.