Temporal feature alignment network for video inpainting

1. A method for inpainting a sequence of initial image frames, the method comprising:

receiving initial video data representing a sequence of initial image frames;

generating optical flow displacement values between neighboring image frames of the sequence of initial image frames;

generating optical flow validity masks for respective image frames of the sequence of image frames based on the optical flow displacement values by:

mapping a first pixel at a first pixel location in a first image frame to a corresponding pixel in a second pixel location in a second image frame using a backward optical flow displacement vector and pixel values of the first pixel;

mapping the corresponding pixel in the second pixel location in the second image frame to another corresponding pixel in a third pixel location in the first image frame using a forward optical flow displacement vector and the pixel values of the first pixel; and

determining a distance between the first pixel location in the first image frame and the third pixel location in the first image frame;

warp-shifting image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame using the optical flow displacement values;

selecting the warp-shifted image features from the image feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame based on one or more of the generated optical flow validity masks; and

generating a sequence of complete image frames based on warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame, the sequence of complete image frames including an inpainted version of the sequence of initial image frames.

2. The method of claim 1 ,

wherein the sequence of complete image frames is based on the selected warp-shifted image features from the feature maps of the one or more neighboring image frames and the selected image features from the image feature maps of the current image frame.

3. The method of claim 1 , wherein warp-shifting image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame includes:

temporally shifting features from a first image feature map in a first channel of a previous image frame to the current image frame;

temporally shifting features from a second image feature map in a second channel of a subsequent image frame to the current image frame;

warping the shifted image features from the first image feature map in the first channel to a time consistent with the current image frame; and

warping the shifted image features from the second image feature map in the second channel to the time consistent with the current image frame.

4. The method of claim 1 , further comprising indicating that optical flow displacement values associated with the first pixel location in the first image frame are valid when the determined distance between the first pixel location in the first image frame and the second pixel location in the first image frame is less than a threshold distance value.

5. The method of claim 1 , further comprising:

providing the optical flow displacement values and the optical flow validity mask to a trained convolutional neural network model including a plurality of encoding layers and a plurality of decoding layers; and

at one or more encoding layers of the plurality of encoding layers and one or more decoding layers of the plurality of decoding layers:

warp-shifting image features from an image feature map of a first channel of a first neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value;

warp-shifting image features from an image feature map of a second channel of a second neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value; and

concatenating the feature maps along the channel dimension.

6. The method of claim 5 , further comprising:

applying a convolution operation to the concatenated feature maps; and

combining the results of the convolution operation applied to the concatenated feature maps with a gating signal.

7. A system configured to perform video inpainting, the system comprising:

a processor; and

memory including instructions, which when executed by the processor, causes the processor to:

receive initial video data representing a sequence of initial image frames;

generate optical flow displacement values between neighboring image frames of the sequence of initial image frames;

generate optical flow validity masks for respective image frames of the sequence of image frames based on the optical flow displacement values by:

mapping a first pixel at a first pixel location in a first image frame to a corresponding pixel in a second pixel location in a second image frame using a backward optical flow displacement vector and pixel values of the first pixel;

mapping the corresponding pixel in the second pixel location in the second image frame to another corresponding pixel in a third pixel location in the first image frame using a forward optical flow displacement vector and the pixel values of the first pixel; and

determining a distance between the first pixel location in the first image frame and the third pixel location in the first image frame;

at one or more encoding layers and at one or more decoding layers of a trained convolutional neural network, warp-shift image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame using the optical flow displacement values;

select the warp-shifted image features from the image feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame based on one or more of the generated optical flow validity masks; and

generate a sequence of complete image frames based on warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame, the sequence of complete image frames including an inpainted version of the sequence of initial image frames.

8. The system of claim 7 ,

wherein the sequence of complete image frames is based on the selected warp-shifted image features from the feature maps of the one or more neighboring image frames and the selected image features from the image feature maps of the current image frame.

9. The system of claim 8 , wherein the instructions, when executed by the processor, cause the processor to:

temporally shift features from a first image feature map in a first channel of a previous image frame to the current image frame;

temporally shift features from a second image feature map in a second channel of a subsequent image frame to the current image frame;

warp the shifted image features from the first image feature map in the first channel to a time consistent with the current image frame; and

warp the shifted image features from the second image feature map in the second channel to the time consistent with the current image frame.

10. The system of claim 7 , wherein the instructions, when executed by the processor, cause the processor to indicate that optical flow displacement values associated with the first pixel location in the first image frame are valid when the determined distance between the first pixel location in the first image frame and the second pixel location in the first image frame is less than a threshold distance value.

11. The system of claim 7 , wherein the instructions, when executed by the processor, cause the processor to:

provide the optical flow displacement values and the optical flow validity mask to the trained convolutional neural network; and

at one or more encoding layers of the plurality of encoding layers and one or more decoding layers of the plurality of decoding layers:

warp-shift image features from an image feature map of a first channel of a first neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value;

warp-shift image features from an image feature map of a second channel of a second neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value; and

concatenate the feature maps along the channel dimension.

12. The system of claim 11 , wherein the instructions, when executed by the processor, cause the processor to:

apply a convolution operation to the concatenated feature maps; and

combine the results of the convolution operation applied to the concatenated feature maps with a gating signal.

13. The system of claim 7 , wherein the instructions, when executed by the processor, cause the processor to provide the sequence of complete image frames to a computing system.

14. A non-transitory computer-readable storage medium including instructions, which when executed by a processor, cause the processor to:

receive initial video data representing a sequence of initial image frames;

generate optical flow displacement values between neighboring image frames of the sequence of initial image frames;

generate optical flow validity masks for respective image frames of the sequence of image frames based on the optical flow displacement values by:

mapping a first pixel at a first pixel location in a first image frame to a corresponding pixel in a second pixel location in a second image frame using a backward optical flow displacement vector and pixel values of the first pixel;

mapping the corresponding pixel in the second pixel location in the second image frame to another corresponding pixel in a third pixel location in the first image frame using a forward optical flow displacement vector and the pixel values of the first pixel; and

determining a distance between the first pixel location in the first image frame and the third pixel location in the first image frame;

warp-shift image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame using the optical flow displacement values;

select warp-shifted image features from the feature maps of the one or more neighboring image frames and image features from the image feature maps of the current image frame based on one or more of the optical flow validity masks; and

generate a sequence of complete image frames based on the selected warp-shifted image features from the feature maps of the one or more neighboring image frames and the selected image features from the image feature maps of the current image frame, the sequence of complete image frames including an inpainted version of the sequence of initial image frames.

15. The non-transitory computer-readable storage medium of claim 14 , wherein warp-shifting image features from image feature maps of one or more neighboring image frames to image feature maps of a current image frame includes:

temporally shifting features from a first image feature map in a first channel of a previous image frame to the current image frame;

temporally shifting features from a second image feature map in a second channel of a subsequent image frame to the current image frame;

warping the shifted image features from the first image feature map in the first channel to a time consistent with the current image frame; and

warping the shifted image features from the second image feature map in the second channel to the time consistent with the current image frame.

16. The non-transitory computer-readable storage medium of claim 14 , further comprising instructions, which when executed by the processor, cause the processor to indicate that optical flow displacement values associated with the first pixel location in the first image frame are valid when the determined distance between the first pixel location in the first image frame and the second pixel location in the first image frame is less than a threshold distance value.

17. The non-transitory computer-readable storage medium of claim 14 , further comprising instructions, which when executed by the processor, cause the processor to:

provide the optical flow displacement values and the optical flow validity mask to a trained convolutional neural network model including a plurality of encoding layers and a plurality of decoding layers;

warp-shift image features from an image feature map of a first channel of a first neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value at one or more encoding layers of the plurality of encoding layers and one or more decoding layers of the plurality of decoding layers;

warp-shift image features from an image feature map of a second channel of a second neighboring image frame to the image feature maps of the current image frame using the optical flow displacement value at one or more encoding layers of the plurality of encoding layers and one or more decoding layers of the plurality of decoding layers; and

concatenate the feature maps along the channel direction.