Online flow guided memory networks for object detection in video

1. A computer-implemented method for object detection utilizing an online flow guided memory network (OFGMN), the method comprising:

receiving, by a processor, a plurality of videos, each of the plurality of videos including a plurality of frames;

generating, by the processor with a feature extraction network, a frame feature map for a current frame of the plurality of frames;

determining, by the processor, a memory feature map from the frame feature map and a previous memory feature map from a previous frame by warping the previous memory feature map, the warping being updated at each of the plurality of frames and being configured to perform only a single warp computation per-frame, the memory feature map being generated by aggregating the frame feature map with the warped previous memory feature map;

spatially aligning hidden memory in the memory feature map with an observer and scene motion in particular content of the plurality of frames of the plurality of videos by (h t =RNN (warp(h t-1 , D t,t-1 ), x t )), where h t represents the frame feature map for a current frame, RNN represents a Recurrent Neural Network, h t-1 represents a previous frame from the current frame, D t,t-1 represents a displacement for each pixel between frames t and t−1, and x t represents a spatial alignment at the current frame;

predicting, by the processor with a task network, an object from the memory feature map; and

controlling an operation of a processor-based machine to react in accordance with the object.

2. The computer-implemented method as recited in claim 1 , wherein the generating includes employing a residual neural network.

3. The computer-implemented method as recited in claim 1 , wherein the determining includes utilizing a displacement network.

4. The computer-implemented method as recited in claim 3 , wherein the displacement network includes an optical flow convolutional neural network.

5. The computer-implemented method as recited in claim 3 , wherein the displacement network generates displacement vectors from the current frame and the previous frame.

6. The computer-implemented method as recited in claim 1 , wherein the aggregating includes employing simple averaging.

7. The computer-implemented method as recited in claim 1 , wherein the aggregating includes utilizing a learned adaptive weighting.

8. The computer-implemented method as recited in claim 1 , wherein the warping includes warping the previous memory feature map with displacement vectors from a displacement network to form a warped memory feature map.

9. The computer-implemented method as recited in claim 1 , wherein the warping includes employing bilinear sampling.

10. The computer-implemented method as recited in claim 1 , wherein the predicting includes utilizing a task convolutional neural network.

11. A computer program product for object detection utilizing an online flow guided memory network (OFGMN), the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

receiving, by a processor, a plurality of videos, each of the plurality of videos including a plurality of frames;

generating, by the processor with a feature extraction network, a frame feature map for a current frame of the plurality of frames;

determining, by the processor, a memory feature map from the frame feature map and a previous memory feature map from a previous frame by warping the previous memory feature map, the warping being updated at each of the plurality of frames and being configured to perform only a single warp computation per-frame, the memory feature map being generated by aggregating the frame feature map with the warped previous memory feature map;

spatially aligning hidden memory in the memory feature map with an observer and scene motion in particular content of the plurality of frames of the plurality of videos by (h t =RNN (warp(h t-1 , D t,t-1 ), x t )), where h t represents the frame feature map for a current frame, RNN represents a Recurrent Neural Network, h t-1 represents a previous frame from the current frame, D t,t-1 represents a displacement for each pixel between frames t and t−1, and x t represents a spatial alignment at the current frame;

predicting, by the processor with a task network, an object from the memory feature map; and

controlling an operation of a processor-based machine to react in accordance with the object.

12. An object detector to detect objects in video, the object detector comprising:

a processing system including a processor device and memory coupled to the processor device, the processing system having program code configured to:

receive a plurality of videos, each of the plurality of videos including a plurality of frames;

generate, with a feature extraction network, a frame feature map for a current frame of the plurality of frames;

determine a memory feature map from the frame feature map and a previous memory feature map from a previous frame by warping the previous memory feature map, the warping being updated at each of the plurality of frames and being configured to perform only a single warp computation per-frame, the memory feature map being generated by aggregating the frame feature map with the warped previous memory feature map;

spatially align hidden memory in the memory feature map with an observer and scene motion in particular content of the plurality of frames of the plurality of videos by (h t =RNN (warp(h t-1 , D t,t-1 ), x t )), where h t represents the frame feature map for a current frame, RNN represents a Recurrent Neural Network, h t-1 represents a previous frame from the current frame, D t,t-1 represents a displacement for each pixel between frames t and t−1, and x t represents a spatial alignment at the current frame;

predict, with a task network, an object from the memory feature map; and

control an operation of a processor-based machine to react in accordance with the object.

13. The object detector as recited in claim 12 , wherein the feature extraction network includes a residual neural network.

14. The object detector as recited in claim 12 , further comprising a displacement network that includes an optical flow convolutional neural network.

15. The object detector as recited in claim 14 , wherein the displacement network generates displacement vectors from the current frame and the previous frame.

16. The object detector as recited in claim 12 , wherein the warped memory feature map is formed by warping a previous memory feature map with displacement vectors from a displacement network.

17. The object detector as recited in claim 16 , wherein the warped memory feature map is formed with bilinear sampling.

18. The object detector as recited in claim 12 , wherein the aggregation system utilizes either simple averaging or a learned adaptive weighting.