Scene segmentation using model subtraction

1. A computer-implemented method comprising:

providing a frame of a scene, the scene including a scene background, the scene background being a chroma screen;

providing a model of the scene background based on fitting an intensity plane to the chroma screen in the frame;

determining a frame background using the model based on mixing a chroma of the chroma screen with the intensity plane;

subtracting the frame background from the frame to obtain an approximate segmentation; and

training a segmentation network using the approximate segmentation.

2. The computer-implemented method of claim 1 , further comprising:

segmenting, after training the segmentation network, a second frame from the scene using the segmentation network.

3. The computer-implemented method of claim 1 , wherein training the segmentation network using the approximate segmentation comprises:

tagging a first portion of the frame, included in the approximate segmentation, with a subject tag;

tagging a second portion of the frame, excluded from the approximate segmentation, with a background tag;

generating a segmentation inference from the segmentation network using the frame; and

evaluating the segmentation inference with at least one of the subject tag and the background tag.

4. The computer-implemented method of claim 3 , wherein training the segmentation network using the approximate segmentation comprises:

morphologically dilating the approximate segmentation to define the second portion of the frame; and

morphologically eroding the approximate segmentation to define the first portion of the frame.

5. The computer-implemented method of claim 1 , wherein:

providing the model of the scene background comprises capturing a three-dimensional model of the scene background; and

determining the frame background from the model comprises generating the frame background using the three-dimensional model and a camera pose of the frame.

6. The computer-implemented method of claim 1 , wherein determining the frame background from the model comprises:

registering the model with a set of trackable tags in the scene background;

deriving a camera pose for the frame using the set of trackable tags; and

deriving the frame background using the model and the camera pose of the frame.

7. The computer-implemented method of claim 1 , wherein:

providing the model of the scene background comprises: (i) capturing a clean plate scene background; (ii) selecting a set of key frames from the clean plate scene background; and (iii) training, using the clean plate scene background, an optical flow network; and

determining the frame background from the model comprises inferring the frame background using the optical flow network.

8. The computer-implemented method of claim 7 , wherein:

training the optical flow network includes synthesizing occlusions on the clean plate scene background.

9. A non-transitory computer-readable medium storing instructions to execute a computer-implemented method, the method comprising:

providing a frame of a scene, the scene including a scene background, the scene background being a chroma screen;

providing a model of the scene background based on fitting an intensity plane to the chroma screen in the frame;

determining a frame background from the model based on mixing a chroma of the chroma screen with the intensity plane;

subtracting the frame background from the frame to obtain a frame foreground; and

training a segmentation network using the frame foreground.

10. The non-transitory computer-readable medium of claim 9 , the method further comprising:

segmenting, after training the segmentation network, a second frame from the scene using the segmentation network.

11. The non-transitory computer-readable medium of claim 9 , wherein training the segmentation network using the frame foreground comprises:

tagging a first portion of the frame, included in the frame foreground, with a subject tag;

tagging a second portion of the frame, excluded from the frame foreground, with a background tag;

generating a segmentation inference from the segmentation network using the frame; and

evaluating the segmentation inference with at least one of the subject tag and the background tag.

12. The non-transitory computer-readable medium of claim 11 , wherein training the segmentation network using the frame foreground comprises:

morphologically dilating the frame foreground to define the second portion of the frame; and

morphologically eroding the frame foreground to define the first portion of the frame.

13. The non-transitory computer-readable medium of claim 9 , wherein:

providing the model of the scene background comprises capturing a three-dimensional model of the scene background; and

determining the frame background from the model comprises generating the frame background using the three-dimensional model and a camera pose of the frame.

14. The non-transitory computer-readable medium of claim 9 , wherein determining the frame background from the model comprises:

registering the model with a set of trackable tags in the scene background;

deriving a camera pose for the frame using the set of trackable tags; and

deriving the frame background using the model and the camera pose of the frame.

15. The non-transitory computer-readable medium of claim 9 , wherein:

providing the model of the scene background comprises: (i) capturing a clean plate scene background; (ii) selecting a set of key frames from the clean plate scene background; and (iii) training, using the clean plate scene background, an optical flow network; and

determining the frame background from the model comprises inferring the frame background using the optical flow network.

16. The non-transitory computer-readable medium of claim 15 , wherein:

training the optical flow network includes synthesizing occlusions on the clean plate scene background.