System for automatic object mask and hotspot tracking
Systems and methods provide editing operations in a smart editing system that may generate a focal point within a mask of an object for each frame of a video segment and perform editing effects on the frames of the video segment to quickly provide users with natural video editing effects. An eye-gaze network may produce a hotspot map of predicted focal points in a video frame. These predicted focal points may then be used by a gaze-to-mask network to determine objects in the image and generate an object mask for each of the detected objects. This process may then be repeated to effectively track the trajectory of objects and object focal points in videos. Based on the determined trajectory of an object in a video clip and editing parameters, the editing engine may produce editing effects relative to an object for the video clip.
1. A method comprising:
generating a mask of an object in a video segment based on an initial frame of the video segment and a first focal region associated with the object in the initial frame;
for each frame of a plurality of frames of the video segment, generating, based on the frame and the mask of the object, using a neural network, a corresponding focal region associated with the mask of the object that tracks the object through the plurality of frames; and
executing an operation based on the corresponding focal region for one or more frames of the plurality of frames.
2. The method of claim 1 , further comprising generating the mask using a masking network.
3. The method of claim 1 , further comprising generating the first focal region, using a focal region prediction network and based on the initial frame of the video segment, by extracting spatial features of the initial frame during an encoding process and decoding the spatial features.
4. The method of claim 1 , wherein the neural network comprises a first branch configured to generate the corresponding focal region and a second branch configured to generate a corresponding mask of the object in the frame.
5. The method of claim 1 , further comprising generating the mask by:
receiving, at a decoder block of a masking network, spatial information extracted by an encoder block of the masking network via a skip connection; and
integrating the spatial information with a representation of the first focal region.
6. The method of claim 1 , further comprising generating the mask of the object by:
extracting, by an encoder block of a masking network, image features of the initial frame;
receiving the image features at a decoder branch via a first skip connection from the encoder block;
receiving a representation of the first focal region at the decoder branch via a second skip connection;
generating an integrated input that integrates the image features and the representation of the first focal region; and
generating, by the decoder branch, the mask of the object based on the integrated input.
7. The method of claim 1 , further comprising generating the corresponding focal region for the frame by: generating a template image of the object by cropping the initial frame based on the mask, and feeding the template image and the frame into the neural network.
8. A computer system comprising:
one or more hardware processors and memory configured to provide computer program instructions to the one or more hardware processors;
a segmentation and hotspot module configured to use the one or more hardware processors and the memory to:
generate a mask of an object in a video segment based on an initial frame of the video segment and a first focal region associated with the object in the initial frame; and
for each frame of a plurality of frames of the video segment, generate, based on the frame and the mask of the object, using a neural network, a corresponding focal region associated with the mask of the object that tracks the object through the plurality of frames; and
a crop suggestion module configured to use the one or more hardware processors and the memory to execute an operation based on the corresponding focal region for one or more frames of the plurality of frames.
9. The computer system of claim 8 , further comprising a masking network configured to generate the mask.
10. The computer system of claim 8 , further comprising a focal region prediction network configured to generate the first focal region based on the initial frame of the video segment by extracting spatial features of the initial frame during an encoding process and decoding the spatial features.
11. The computer system of claim 8 , wherein the neural network comprises a first branch to generate the corresponding focal region and a second branch configured to generate a corresponding mask of the object in the frame.
12. The computer system of claim 8 , further comprising a masking network configured to generate the mask by:
receiving, at a decoder block of the masking network, spatial information extracted by an encoder block of the masking network via a skip connection; and
integrating the spatial information with a representation of the first focal region.
13. The computer system of claim 8 , further comprising a masking network configured to generate the mask by:
extracting, by an encoder block of the masking network, image features of the initial frame;
receiving the image features at a decoder branch via a first skip connection from the encoder block;
receiving a representation of the first focal region at the decoder branch via a second skip connection;
generating an integrated input that integrates the image features and the representation of the first focal region; and
generating, by the decoder branch, the mask of the object based on the integrated input.
14. The computer system of claim 8 , wherein the segmentation and hotspot module is further configured to generate the corresponding focal region for the frame based on the mask by generating a template image of the object by cropping the initial frame based on the mask, and feeding the template image and the frame into the neural network.
15. The computer system of claim 8 , wherein the crop suggestion module is further configured to execute an editing operation based on the corresponding focal region for the one or more frames.
16. One or more non-transitory computer-readable storage media storing instructions executable by a computing device to cause the computing device to perform operations comprising:
generating a mask of an object in a video segment based on an initial frame of the video segment and a first focal region associated with the object in the initial frame;
for each frame of a plurality of frames of the video segment, generating, based on the frame and the mask of the object, using a neural network, a corresponding focal region associated with the mask of the object that tracks the object through the plurality of frames; and
executing an operation based on the corresponding focal region for one or more frames of the plurality of frames.
17. The one or more non-transitory computer-readable storage media of claim 16 , the operations further comprising generating the mask using a masking network and generating the first focal region using a focal region prediction network and based on the initial frame of the video segment by extracting spatial features of the initial frame during an encoding process and decoding the spatial features.
18. The one or more non-transitory computer-readable storage media of claim 16 , the operations further comprising generating the mask by:
receiving, at a decoder block of a masking network, spatial information extracted by an encoder block of the masking network via a skip connection; and
integrating the spatial information with a representation of the first focal region.
19. The one or more non-transitory computer-readable storage media of claim 16 , the operations further comprising generating the mask by:
extracting, by an encoder block of a masking network, image features of the initial frame;
receiving the image features at a decoder branch via a first skip connection from the encoder block;
receiving a representation of the first focal region at the decoder branch via a second skip connection;
generating an integrated input that integrates the image features and the representation of the first focal region; and
generating, by the decoder branch, the mask of the object based on the integrated input.
20. The one or more non-transitory computer-readable storage media of claim 16 , wherein the operations further comprise generating the corresponding focal region for the frame based on the mask comprises generating a template image of the object by cropping the initial frame based on the mask, and feeding the template image and the frame into the neural network.