Patch expansion for segmentation network training

1. A computer-implemented method comprising:

associating a first primary element from a scene with a background tag;

associating a second primary element from the scene with a subject tag;

generating a background texture using the first primary element, wherein the generating the background texture includes:

defining a first patch of the scene, wherein the patch includes the first primary element; and

expand the first patch to a larger dimension, wherein the larger dimension is larger than an input size of a segmentation network;

generating a subject texture using the second primary element;

combining the background texture and the subject texture into a synthesized frame; and

training a segmentation network using the background tag, the subject tag, and the synthesized frame.

2. The computer-implemented method of claim 1 , further comprising: after the training of the segmentation network, segmenting a first frame from the scene using the segmentation network, wherein the first primary element is a first pixel from a second frame from the scene, and wherein the second primary element is a second pixel from the second frame from the scene.

3. The computer-implemented method of claim 1 , wherein the associating steps comprise:

displaying a frame from the scene; and

receiving a selection directed to the frame while the frame is displayed.

4. The computer-implemented method of claim 1 , wherein the generating of the background texture step comprises:

defining a patch from the scene, wherein the patch includes the first primary element; and

expanding the patch to a larger dimension, wherein the larger dimension is larger than an input size for the segmentation network.

5. The computer-implemented method of claim 1 , wherein the generating of the subject texture step comprises:

defining a patch from the scene, wherein the patch includes the second primary element; and

expanding the patch to a larger dimension, wherein the larger dimension is larger than an input size of the segmentation network.

6. The computer-implemented method of claim 1 , wherein the combining step comprises:

generating a background for the synthesized frame using the background texture; and

generating a subject mask for the synthesized frame using the subject texture, wherein the synthesized frame includes the background and the subject mask.

7. The computer-implemented method of claim 6 , wherein the combining step further comprises:

tagging the background using the background tag; and

tagging the subject mask with the subject tag.

8. The computer-implemented method of claim 7 , wherein:

the segmentation network is a convolutional neural network;

the tagging steps make the synthesized frame a supervisor for a training routine; and

the training the segmentation network step uses the background tag and the subject tag in that the supervisor is labeled using the background tag and the subject tag.

9. The computer-implemented method of claim 6 , wherein the generating the subject mask comprises:

providing a mask for the subject; and

filling the mask using the subject texture.

10. The computer-implemented method of claim 9 , wherein:

the mask is an alpha mask;

the first primary element is a first pixel; and

the second primary element is a second pixel.

11. The computer-implemented method of claim 9 , wherein the providing the mask for the subject comprises:

providing a mask dictionary; and

receiving a selection of the mask from the mask dictionary.

12. The computer-implemented method of claim 9 , wherein the providing the mask for the subject comprises:

providing a mask dictionary, wherein the mask dictionary is indexed by a set of subject identifiers;

identifying the subject using a classifier and the subject texture, wherein the identifying produces a subject identifier; and

selecting the mask from the mask dictionary using the subject identifier.

13. A computer-implemented method comprising:

associating a first primary element from a first frame in a scene with a background tag;

associating a second primary element from the first frame with a subject tag;

generating a background texture using the first primary element, the generating of the background texture includes:

defining a first patch from the first frame, wherein the first patch includes the first primary element; and

expanding the first patch to a larger dimension, wherein the larger dimension is larger than an input size for a segmentation network;

generating a foreground texture using the second primary element;

combining the background texture and the foreground texture into a synthesized frame; and

training a segmentation network using the background tag, the subject tag, and the synthesized frame.

14. The computer-implemented method of claim 13 , further comprising:

training the segmentation network using the background tag, the subject tag, and the synthesized frame; and

after the training of the segmentation network, segmenting a second frame using the segmentation network, wherein the second frame is from the scene.

15. The computer-implemented method of claim 14 , wherein the associating the first primary element step comprise:

displaying the first frame from the scene; and

receiving a selection directed to the first frame while the first frame is displayed.

16. The computer-implemented method of claim 14 , wherein the generating of the subject texture step comprises:

defining a patch from the first frame, wherein the patch includes the second primary element; and

expanding the patch to a larger dimension, wherein the larger dimension is larger than an input size of the segmentation network.

17. The computer-implemented method of claim 13 , wherein the combining step comprises:

generating a background for the synthesized frame using the background texture; and

generating a subject mask for the synthesized frame using the foreground texture, wherein the synthesized frame includes the background and the subject mask.

18. The computer-implemented method of claim 17 , wherein the combining step further comprises:

tagging the background using the background tag; and

tagging the subject mask with the subject tag.

19. The computer-implemented method of claim 18 , wherein:

the segmentation network is a convolutional neural network;

the tagging steps make the synthesized frame a supervisor for a training routine; and

the training the segmentation network step uses the background tag and the subject tag in that the supervisor is labeled using the background tag and the subject tag.

20. The computer-implemented method of claim 17 , wherein the generating the subject mask comprises:

providing a mask for the subject; and

filling the mask using the foreground texture.

21. The computer-implemented method of claim 20 , wherein:

the mask is an alpha mask;

the first primary element is a first pixel; and

the second primary element is a second pixel.

22. The computer-implemented method of claim 20 , wherein the providing the mask for the subject comprises:

providing a mask dictionary; and

receiving a selection of the mask from the mask dictionary.

23. The computer-implemented method of claim 20 , wherein the providing the mask for the subject comprises:

providing a mask dictionary, wherein the mask dictionary is indexed by a set of subject identifiers;

identifying the subject using a classifier and the foreground texture, wherein the identifying produces a subject identifier; and

selecting the mask from the mask dictionary using the subject identifier.

24. A computer-implemented method comprising:

associating a first primary element from a first frame in a scene with a background;

associating a second primary element from the first frame with a subject;

generating a background texture using the first primary element, wherein the generating the background texture includes:

defining a first patch of the first frame, wherein the patch includes the first primary element; and

expand the first patch to a larger dimension, wherein the larger dimension is larger than an input size of a segmentation network;

generating a foreground texture using the second primary element;

combining, in a synthesized frame, the background texture and the foreground texture; and

training a segmentation network using the synthesized frame.

25. The computer-implemented method of claim 24 , further comprising:

after the training of the segmentation network, segmenting a second frame using the segmentation network, wherein the second frame is from the scene.

26. The computer-implemented method of claim 25 , wherein the associating the first primary element step comprise:

displaying the first frame from the scene; and

receiving a selection directed to the first frame while the first frame is displayed.

27. The computer-implemented method of claim 24 , wherein the combining step comprises:

generating a background for the synthesized frame using the background texture; and

generating a subject mask for the synthesized frame using the foreground texture,

wherein the synthesized frame includes the background and the subject mask.

28. The computer-implemented method of claim 27 , wherein the generating the subject mask comprises:

providing a mask for the subject; and

filling the mask using the foreground texture.

29. The computer-implemented method of claim 28 , wherein the providing the mask for the subject comprises:

providing a mask dictionary; and

receiving a selection of the mask from the mask dictionary.

30. The computer-implemented method of claim 28 , wherein the providing the mask for the subject comprises:

providing a mask dictionary, wherein the mask dictionary is indexed by a set of subject identifiers;

identifying the subject using a classifier and the foreground texture, wherein the identifying produces a subject identifier; and

selecting the mask from the mask dictionary using the subject identifier.