Video matting
The present disclosure describes techniques of improving video matting. The techniques comprise extracting features from each frame of a video by an encoder of a model, wherein the video comprises a plurality of frames; incorporating, by a decoder of the model, into any particular frame temporal information extracted from one or more frames previous to the particular frame, wherein the particular frame and the one or more previous frames are among the plurality of frames of the video, and the decoder is a recurrent decoder; and generating a representation of a foreground object included in the particular frame by the model, wherein the model is trained using segmentation dataset and matting dataset.
1. A method of improving video matting, comprising:
extracting features from each frame of a video by an encoder of a model, wherein the video comprises a plurality of frames;
incorporating, by a decoder of the model, into any particular frame temporal information extracted from one or more frames previous to the particular frame, wherein the particular frame and the one or more previous frames are among the plurality of frames of the video, and the decoder is a recurrent decoder; and
generating a representation of a foreground object included in the particular frame by the model, wherein the model is trained using segmentation dataset and matting dataset, and wherein an application of the segmentation dataset to the model and an application of the matting dataset to the model are interleaved in a process of training the model.
2. The method of claim 1 , further comprising:
generating a low-resolution representative of each frame by downsampling each frame.
3. The method of claim 2 , further comprising:
encoding the low-resolution representative using the encoder, wherein the encoder comprises a plurality of convolution and pooling layers.
4. The method of claim 1 , wherein the decoder comprises a plurality of convolutional gated recurrent unit (ConvGRU) for incorporating the temporal information.
5. The method of claim 1 , wherein the model further comprises a Deep Guided Filter (DGF) for processing a high-resolution video.
6. The method of claim 1 , wherein the model is further trained by applying at least one loss function, and wherein the at least one loss function comprises a Least Absolute Deviations (L1) loss, a Laplacian loss, and a temporal coherence loss, or a foreground prediction loss.
7. The method of claim 1 , wherein the foreground object included in the particular frame is a human being.
8. A system of improving video matting, comprising:
at least one processor; and
at least one memory communicatively coupled to the at least one processor and storing instructions that upon execution by the at least one processor cause the system to perform operations, the operations comprising:
extracting features from each frame of a video by an encoder of a model, wherein the video comprises a plurality of frames;
incorporating, by a decoder of the model, into any particular frame temporal information extracted from one or more frames previous to the particular frame, wherein the particular frame and the one or more previous frames are among the plurality of frames of the video, and the decoder is a recurrent decoder; and
generating a representation of a foreground object included in the particular frame by the model, wherein the model is trained using segmentation dataset and matting dataset, and wherein an application of the segmentation dataset to the model and an application of the matting dataset to the model are interleaved in a process of training the model.
9. The system of claim 8 , the operations further comprising:
generating a low-resolution representative of each frame by downsampling each frame.
10. The system of claim 9 , further comprising:
encoding the low-resolution representative using the encoder, wherein the encoder comprises a plurality of convolution and pooling layers.
11. The system of claim 8 , wherein the decoder comprises a plurality of convolutional gated recurrent unit (ConvGRU) for incorporating the temporal information.
12. The system of claim 8 , wherein the model further comprises a Deep Guided Filter (DGF) for processing a high-resolution video.
13. The system of claim 8 , wherein the model is further trained by applying at least one loss function, and wherein the at least one loss function comprises a Least Absolute Deviations (L1) loss, a Laplacian loss, and a temporal coherence loss, or a foreground prediction loss.
14. The system of claim 8 , wherein the foreground object included in the particular frame is a human being.
15. A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations, the operation comprising:
extracting features from each frame of a video by an encoder of a model, wherein the video comprises a plurality of frames;
incorporating, by a decoder of the model, into any particular frame temporal information extracted from one or more frames previous to the particular frame, wherein the particular frame and the one or more previous frames are among the plurality of frames of the video, and the decoder is a recurrent decoder; and
generating a representation of a foreground object included in the particular frame by the model, wherein the model is trained using segmentation dataset and matting dataset, and wherein an application of the segmentation dataset to the model and an application of the matting dataset to the model are interleaved in a process of training the model.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the model further comprises a Deep Guided Filter (DGF) for processing a high-resolution video.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the decoder comprises a plurality of convolutional gated recurrent unit (ConvGRU) for incorporating the temporal information.