Self-supervised compositional feature representation for video understanding
A method of compositional feature representation learning for video understanding is described. The method includes individually processing a sequence of video frames received as an input of a feature map network to generate a plurality of feature maps. The method also includes binding the plurality of feature maps to a fixed set of slot variables using an attention model according to a motion segmentation signal. The method further includes combining slot states corresponding to the fixed set of slot variables into a combined feature map. The method also includes decoding the combined feature map to form a reconstructed sequence of video frames, in which objects discovered in the reconstructed sequence of video frames are identified.
1 . A method of compositional feature representation learning for video understanding, comprising:
individually processing a sequence of video frames received as an input of a feature map network to generate a plurality of feature maps;
binding the plurality of feature maps to a fixed set of slot variables using an attention model according to a motion segmentation signal;
combining slot states corresponding to the fixed set of slot variables into a combined feature map;
decoding the combined feature map to form a reconstructed sequence of video frames, in which dynamic objects discovered in the reconstructed sequence of video frames are identified by 2D bounding boxes estimated from 2D instance masks estimated to focus on the dynamic objects discovered instead of non-dynamic objects in the reconstructed sequence of video frames; and
controlling an ego vehicle to follow a planned trajectory adjusted for collision avoidance in response to the dynamic objects discovered in the reconstructed sequence of video frames in a scene captured by and surrounding the ego vehicle.
2 . The method of claim 1 , in which binding the plurality of feature maps comprises separating the moving objects from static objects in the plurality of feature maps based on the motion segmentation signal.
3 . The method of claim 1 , further comprising using motion supervision for training the attention model to assign the fixed set of slot variables distinguishing a foreground of a sequence of video frames from a background of the sequence of video frames.
4 . The method of claim 1 , further comprises segmenting the objects discovered in the sequence of video frames.
5 . The method of claim 1 , further comprises tracking the objects discovered in the sequence of video frames.
6 . The method of claim 1 , in which decoding the combined feature map comprises embedding features within the reconstructed sequence of video frames.
7 . A non-transitory computer-readable medium having program code recorded thereon of compositional feature representation learning for video understanding, the program code being executed by a processor and comprising:
program code to individually process a sequence of video frames received as an input of a feature map network to generate a plurality of feature maps;
program code to bind the plurality of feature maps to a fixed set of slot variables using an attention model according to a motion segmentation signal;
program code to combine slot states corresponding to the fixed set of slot variables into a combined feature map;
program code to decode the combined feature map to form a reconstructed sequence of video frames, in which dynamic objects discovered in the reconstructed sequence of video frames are identified by 2D bounding boxes estimated from 2D instance masks estimated to focus on the dynamic objects discovered instead of non-dynamic objects in the reconstructed sequence of video frames; and
program code to control an ego vehicle to follow a planned trajectory adjusted for collision avoidance in response to the dynamic objects discovered in the reconstructed sequence of video frames in a scene captured by and surrounding the ego vehicle.
8 . The non-transitory computer-readable medium of claim 7 , in which the program code to bind the plurality of feature maps comprises program code to separate the moving objects from static objects in the plurality of feature maps based on the motion segmentation signal.
9 . The non-transitory computer-readable medium of claim 7 , further comprising program code to use motion supervision for training the attention model to assign the fixed set of slot variables, distinguishing a foreground of a sequence of video frames from a background of the sequence of video frames.
10 . The non-transitory computer-readable medium of claim 7 , further comprises program code to segment the objects discovered in the sequence of video frames.
11 . The non-transitory computer-readable medium of claim 7 , further comprises program code to track the objects discovered in the sequence of video frames.
12 . The non-transitory computer-readable medium of claim 7 , in which the program code to decode the combined feature map comprises program code to embed features within the reconstructed sequence of video frames.
13 . A system of compositional feature representation learning for video understanding, the system comprising:
an individual feature map generation module to individually process a sequence of video frames received as an input of a feature map network to generate a plurality of feature maps;
an attention model to bind the plurality of feature maps to a fixed set of slot variables using the attention model according to a motion segmentation signal;
a combined feature map generation module to combine slot states corresponding to the fixed set of slot variables into a combined feature map;
an object discovery module to decode the combined feature map to form a reconstructed sequence of video frames, in which dynamic objects discovered in the reconstructed sequence of video frames are identified by 2D bounding boxes estimated from 2D instance masks estimated to focus on the dynamic objects discovered instead of non-dynamic objects in the reconstructed sequence of video frames; and
a controller to control an ego vehicle to follow a planned trajectory adjusted for collision avoidance in response to the dynamic objects discovered in the reconstructed sequence of video frames in a scene captured by and surrounding the ego vehicle.
14 . The system of claim 13 , in which the attention model is trained to separate the moving objects from static objects in the plurality of feature maps based on the motion segmentation signal.
15 . The system of claim 14 , in which the attention model is further trained to assign the fixed set of slot variables, distinguishing a foreground of a sequence of video frames from a background of the sequence of video frames using motion supervision.