Video matting based on foreground-background constraint propagation
A method for propagating user-provided foreground-background constraint information for a first video frame to subsequent frames allows extraction of moving foreground objects with minimal user interaction. Video matting is performed wherein constraints derived from user input with respect to a first frame are propagated to subsequent frames using the estimated alpha matte of each frame. The matte of a frame is processed in order to arrive at a rough foreground-background segmentation which is then used for estimating the matte of the next frame. At each frame, the propagated constraints are used by an image matting method for estimating the corresponding matte which is in turn used for propagating the constraints to the next frame, and so on.
1. A computer implemented method for determining foreground and background constraints for a video stream comprising:
receiving foreground and background constraints for a video frame via user input;
generating a matte for the video frame at a frame t using the foreground and background constraints;
generating foreground constraints for a subsequent video frame at a frame t+1 using the matte; and
generating background constraints for the subsequent video frame at a frame t+1 using the matte, wherein generating background constraints for the subsequent video frame includes:
generating a background score field using the matte; and
performing a thresholding operation on the background score field to generate a binary field.
2. The method of claim 1 , wherein generating foreground constraints for the subsequent video frame includes:
performing a thresholding operation on the matte to generate a binary field.
3. The method of claim 2 , wherein generating foreground constraints for the subsequent video frame includes:
performing a morphological erosion operation on the binary field.
4. The method of claim 1 , wherein generating background constraints for the subsequent video frame includes:
performing a morphological erosion operation on the binary field.
5. The method of claim 1 , wherein the background score field includes background scores for pixels of the video frame, the background score for a pixel being directly related to the pixel's proximity to a foreground pixel.
6. The method of claim 1 , wherein the background score field includes background scores for pixels of the video frame, the background score for a pixel being inversely related to a corresponding value for the pixel in the matte.
7. The method of claim 1 , wherein generating the background score field includes:
generating a distance transform from the foreground constraints; and
generating a weighted combination of an inverse of the matte and an inverse of the distance transform.
8. The method of claim 7 , wherein generating the weighted combination includes weighting the inverse of the matte by a weight w and the inverse of the distance transform by a weight 1−w, wherein 0.5≦w ≦0.9.
9. A digital video editor apparatus having computer instructions in memory, which, when executed, cause the digital video editor apparatus to determine foreground and background constraints for a video stream by performing the steps of:
receiving foreground and background constraints for a video frame via a user input;
generating a matte for the video frame at a frame t using the foreground and background constraints;
generating foreground constraints for a subsequent video frame at a frame t+1 using the matte; and
generating background constraints for the subsequent video frame at a frame t+ 1 using the matte, wherein generating background constraints for the subsequent video frame includes:
generating a background score field using the matte; and
performing a thresholding operation on the background score field to generate a binary field.
10. The digital video editor apparatus of claim 9 , wherein generating foreground constraints for the subsequent video frame includes:
performing a thresholding operation on the matte to generate a binary field.
11. The digital video editor apparatus of claim 9 , wherein generating foreground constraints for the subsequent video frame includes:
performing a morphological erosion operation on the binary field.
12. The digital video editor apparatus of claim 9 , wherein generating background constraints for the subsequent video frame includes:
performing a morphological erosion operation on the binary field.
13. The digital video editor apparatus of claim 9 , wherein the background score field includes background scores for pixels of the video frame, the background score for a pixel being directly related to the pixel's proximity to a foreground pixel.
14. The digital video editor apparatus of claim 9 , wherein the background score field includes background scores for pixeis of the video frame, the background score for a pixel being inversely related to a corresponding value for the pixel in the matte.
15. The digital video editor apparatus of claim 9 , wherein generating the background score field includes: generating a distance transform from the foreground constraints; and generating a weighted combination of an inverse of the matte and an inverse of the distance transform.
16. The digital video editor apparatus of claim 15 , wherein generating the weighted combination includes weighting the inverse of the matte by a weight w and the inverse of the distance transform by a weight 1−w, wherein 0.5 ≦w≦0.9.