User guided iterative frame and scene segmentation via network overtraining

1. A computer-implemented method comprising:

selecting a frame from a scene;

generating a first frame segmentation using: (i) the frame; and (ii) a segmentation network;

displaying on a device: (i) the frame; and (ii) the first frame segmentation overlaid on the frame;

receiving a correction input directed to the frame;

training the segmentation network using the correction input;

iterating through the generating segmentation step, displaying segmentation step, receiving correction input step, and training the segmentation step until a number of iterations reaches a predetermined threshold, the predetermined threshold being determined based on a statistical variation within the scene;

generating, after training the segmentation network using the correction input, a revised frame segmentation using: (i) the frame; and (ii) the segmentation network; and

displaying on the device: (i) the frame; and (ii) the revised frame segmentation overlaid on the frame.

2. The computer-implemented method of claim 1 , further comprising:

scanning the scene for a set of statistical variation points using a frame selector,

wherein the selecting the frame step is conducted: (i) using the set of statistical variation points; and (ii) so that the frame is from a frame located between two statistical variation points in the set of statistical variation points.

3. The computer-implemented method of claim 1 , further comprising:

dilating the first frame segmentation to form an outer boundary of a tri-map;

eroding the first frame segmentation to form an inner boundary of the tri-map; and

presenting a region of the frame for receiving the correction input using the tri-map.

4. The computer-implemented method of claim 1 , wherein:

training the segmentation network using the correction input includes tagging a portion of the frame identified by the correction input with a tag for the segmentation target; and

the portion of the frame and the tag are used as a supervisor in a training routine for the segmentation network.

5. The computer-implemented method of claim 1 , further comprising:

selecting a second frame from the scene;

generating a second frame segmentation using: (i) the second frame; and (ii) the segmentation network;

displaying on the device: (i) the second frame; and (ii) the second frame segmentation overlain on the second frame;

receiving a frame skip input; and

selecting a third frame from the scene, wherein the selecting the third frame step is conducted as a response to the receiving the frame skip input.

6. The computer-implemented method of claim 1 , further comprising:

displaying a prompt to select a segmentation target, wherein the correction input is provided in response to the prompt, wherein training the segmentation using the correction input includes tagging a portion of the frame identified by the correction input with a tag for the segmentation target, and wherein the portion of the frame and the tag are used as a supervisor in a training routine for the segmentation network.

7. A computer-implemented method comprising:

selecting a frame from a scene;

generating a first frame segmentation using: (i) the frame; and (ii) a segmentation network;

displaying on a device: (i) the frame; and (ii) the first frame segmentation overlaid on the frame;

receiving a correction input on the frame;

training the segmentation network using the correction input; and

training the segmentation network for the scene by iterating the selecting, generating, displaying, receiving, and training steps using one of: the same frame; or a series of frames from the scene, wherein the training step, the generating the revised frame segmentation step, and the displaying the frame and revised frame segmentation step are all conducted as a single response to the receiving the correction input step until a number of iterations reaches a predetermined threshold, the predetermined threshold being determined based on a statistical variation within the scene.

8. The computer-implemented method of claim 7 , further comprising:

generating, after training the segmentation network using the correction input, a second frame segmentation using: (i) a second frame; and (ii) the segmentation network, wherein the second frame is from the scene, and wherein the generating the second frame segmentation step is conducted as part of the single response.

9. The computer-implemented method of claim 7 , further comprising:

displaying a prompt to select a segmentation target, wherein the correction input is provided in response to the prompt, wherein training the segmentation using the correction input includes tagging a portion of the frame identified by the correction input with a tag for the segmentation target, and wherein the portion of the frame and the tag are used as a supervisor in a training routine for the segmentation network.

10. A computer-implemented method comprising:

selecting a frame from a scene;

scanning the scene for a set of statistical variation points, wherein the selecting the frame step is conducted: (i) using the set of statistical variation points; and (ii) so that the frame is from a frame located between two statistical variation points in the set of statistical variation points;

generating a first frame segmentation using: (i) the frame; and (ii) a segmentation network;

displaying on a device: (i) the frame; and (ii) the first frame segmentation overlaid on the frame;

receiving a correction input on the frame;

training the segmentation network using the correction input; and

training the segmentation network for the scene by iterating the selecting, generating, displaying, receiving, and training steps using one of: the same frame; or a series of frames from the scene.

11. A computer-implemented method comprising:

selecting a frame from a scene;

generating a first frame segmentation using: (i) the frame; and (ii) a segmentation network;

displaying on a device: (i) the frame; and (ii) the first frame segmentation overlaid on the frame;

receiving a correction input on the frame;

training the segmentation network using the correction input;

training the segmentation network for the scene by iterating the selecting, generating, displaying, receiving, and training steps using one of: the same frame; or a series of frames from the scene;

dilating the first frame segmentation to form an outer boundary of a tri-map;

eroding the first frame segmentation to form an inner boundary of the tri-map; and

presenting a region of the frame for receiving the correction input using the tri-map.

12. A computer-implemented method comprising:

selecting a frame from a scene;

generating a first frame segmentation using: (i) the frame; and (ii) a segmentation network;

displaying on a device: (i) the frame; and (ii) the first frame segmentation overlaid on the frame;

receiving a correction input on the frame;

training the segmentation network using the correction input;

training the segmentation network for the scene by iterating the selecting, generating, displaying, receiving, and training steps using one of: the same frame; or a series of frames from the scene; and

receiving a frame skip input instead of an additional correction input during an iteration of the training the segmentation network for the scene step, wherein the training the segmentation network using the correction input step is skipped during the iteration.

13. A device comprising:

a display;

a frame selector instantiated on the device, wherein the frame selector is programmed to select a frame from a scene;

a segmentation editor instantiated on the device, wherein the segmentation editor is programmed to, in response to the frame selector selecting the frame, display on the display: (i) the frame; and (ii) a frame segmentation overlaid on the frame;

a correction interface configured to receive a correction input directed to the frame;

wherein the device is programmed to:

provide the correction input to a trainer for a segmentation network;

receive a revised frame segmentation from the segmentation network after the trainer has applied the correction input to the segmentation network; and

display the revised frame segmentation overlaid on the frame; and

iterate through the receive the correction input, and training the corrected input until a number of iterations reaches a predetermined threshold, the predetermined threshold being determined based on a statistical variation within the scene.

14. The device of claim 13 , wherein the frame selector is programmed to:

scan the scene for a set of statistical variation points, wherein the selecting the frame step is conducted: (i) using the set of statistical variation points; and (ii) so that the frame is from a frame located between two statistical variation points in the set of statistical variation points.

15. The device of claim 13 , wherein:

the correction interface includes a prompt to select a segmentation target;

the correction input is provided in response to the prompt;

the trainer is a supervised learning trainer; and

a portion of the frame identified by the correction input and a segmentation target type form a supervisor for the trainer.

16. A non-transitory computer-readable medium comprising executable instructions, the executable instructions being executable by one or more processors to perform a method, the method comprising:

selecting a frame from a scene;

generating a first frame segmentation using: (i) the frame; and (ii) a segmentation network;

displaying on a device: (i) the frame; and (ii) the first frame segmentation overlaid on the frame;

receiving a correction input directed to the frame;

training the segmentation network using the correction input;

iterating through the generating segmentation step, displaying segmentation step, receiving correction input step, and training the segmentation step until a number of iterations reaches a predetermined threshold, the predetermined threshold being determined based on a statistical variation within the scene;

generating, after training the segmentation network using the correction input, a revised frame segmentation using: (i) the frame; and (ii) the segmentation network; and

displaying on the device: (i) the frame; and (ii) the revised frame segmentation overlaid on the frame.

17. The non-transitory computer-readable medium of claim 16 , the executable instructions that are executable by the one or more processors to further:

scanning the scene for a set of statistical variation points using a frame selector, wherein the selecting the frame step is conducted: (i) using the set of statistical variation points; and (ii) so that the frame is from a frame located between two statistical variation points in the set of statistical variation points.

18. The non-transitory computer-readable medium of claim 16 , the executable instructions that are executable by the one or more processors to further:

dilating the first frame segmentation to form an outer boundary of a tri-map;

eroding the first frame segmentation to form an inner boundary of the tri-map; and

presenting a region of the frame for receiving the correction input using the tri-map.

19. The non-transitory computer-readable medium of claim 16 , the executable instructions that are executable by the one or more processors to further:

training the segmentation network using the correction input includes tagging a portion of the frame identified by the correction input with a tag for the segmentation target; and

the portion of the frame and the tag are used as a supervisor in a training routine for the segmentation network.

20. A non-transitory computer-readable medium comprising executable instructions, the executable instructions being executable by one or more processors to perform a method, the method comprising:

selecting a frame from a scene;

scanning the scene for a set of statistical variation points, wherein the selecting the frame step is conducted: (i) using the set of statistical variation points; and (ii) so that the frame is from a frame located between two statistical variation points in the set of statistical variation points;

generating a first frame segmentation using: (i) the frame; and (ii) a segmentation network;

displaying on a device: (i) the frame; and (ii) the first frame segmentation overlaid on the frame;

receiving a correction input on the frame;

training the segmentation network using the correction input; and

training the segmentation network for the scene by iterating the selecting, generating, displaying, receiving, and training steps using one of: the same frame; or a series of frames from the scene.

21. A non-transitory computer-readable medium comprising executable instructions, the executable instructions being executable by one or more processors to perform a method, the method comprising:

selecting a frame from a scene;

dilating the first frame segmentation to form an outer boundary of a tri-map;

eroding the first frame segmentation to form an inner boundary of the tri-map;

presenting a region of the frame for receiving the correction input using the tri-map;

generating a first frame segmentation using: (i) the frame; and (ii) a segmentation network;

displaying on a device: (i) the frame; and (ii) the first frame segmentation overlaid on the frame;

receiving a correction input on the frame;

training the segmentation network using the correction input; and

training the segmentation network for the scene by iterating the selecting, generating, displaying, receiving, and training steps using one of: the same frame; or a series of frames from the scene.

22. A non-transitory computer-readable medium comprising executable instructions, the executable instructions being executable by one or more processors to perform a method, the method comprising:

selecting a frame from a scene;

generating a first frame segmentation using: (i) the frame; and (ii) a segmentation network;

displaying on a device: (i) the frame; and (ii) the first frame segmentation overlaid on the frame;

receiving a correction input on the frame;

training the segmentation network using the correction input;

training the segmentation network for the scene by iterating the selecting, generating, displaying, receiving, and training steps using one of: the same frame; or a series of frames from the scene; and

receiving a frame skip input instead of an additional correction input during an iteration of the training the segmentation network for the scene step, wherein the training the segmentation network using the correction input step is skipped during the iteration.