Gradient-embedded video anomaly detection
A sequence of frames of a video can be received. For a given frame in the sequence of frames, a gradient-embedded frame is generated corresponding to the given frame. The gradient-embedded frame incorporates motion information. The motion information can be represented as disturbance in the gradient-embedded frame. A plurality of such gradient-embedded frames can be generated corresponding to a plurality of the sequence of frames. Based on the plurality of gradient-embedded frames, a neural network such as a generative adversarial network is trained to learn to suppress the disturbance in the gradient-embedded frame and to generate a substitute frame. In inference stage, anomaly in a target video frame can be detected by comparing it to a corresponding substitute frame generated by the neural network.
1. A computer-implemented method comprising:
receiving a sequence of frames of a video;
for a given frame in the sequence of frames, generating a gradient-embedded frame corresponding to the given frame, the gradient-embedded frame incorporating motion information;
generating a substitute frame corresponding to the given frame by running a neural network, the neural network trained to generate a substitute frame by learning to suppress disturbance in a training set of gradient-embedded frames; and
comparing the substitute frame and the given frame to detect anomaly in the given frame.
2. The method of claim 1 , wherein the neural network includes a generative adversarial network.
3. The method of claim 2 , wherein the generative adversarial network is trained based on optimizing an objective function including adversarial loss.
4. The method of claim 2 , wherein the generative adversarial network is trained based on optimizing an objective function including contextual loss.
5. The method of claim 2 , wherein the generative adversarial network is trained based on optimizing an objective function including gradient loss.
6. The method of claim 2 , wherein the generative adversarial network is trained based on optimizing an objective function including optical loss.
7. The method of claim 1 , wherein the comparing includes generating an anomaly score incorporating motion and appearance differences between the substitute frame and the given frame.
8. The method of claim 1 , wherein the neural network is trained based on a training data set including video frames without anomaly.
9. The method of claim 1 , wherein the generating a gradient-embedded frame includes determining an inter-frame gradient between the given frame and a neighbor frame, the neighbor frame being a frame interval away from the given frame in the sequence of frames, and adding the inter-frame gradient to the given frame conditioned on a threshold.
10. The method of claim 9 , wherein the frame interval is configurable.
11. A computer-implemented method comprising:
receiving a sequence of frames of a video as a training data set;
for a given frame in the sequence of frames, generating a gradient-embedded frame corresponding to the given frame, the gradient-embedded frame incorporating motion information, the motion information represented as disturbance in the gradient-embedded frame, wherein a plurality of gradient-embedded frames are generated corresponding to a plurality of the sequence of frames; and
based on the plurality of gradient-embedded frames, training a neural network to learn to suppress the disturbance in the gradient-embedded frame to generate a substitute frame.
12. The method of claim 11 , wherein the sequence of frames of the video in the training data set includes frames without anomaly.
13. The method of claim 11 , wherein the neural network includes a generative adversarial network.
14. The method of claim 13 , wherein the generative adversarial network is trained based on optimizing an objective function including at least one of adversarial loss, contextual loss gradient loss, and optical loss.
15. The method of claim 11 , wherein the trained neural network is run to detect anomaly in a given set of target video frames, the trained neural network receiving as input a corresponding set of target gradient-embedded frames and generating at least one substitute frame corresponding to at least one of the target gradient-embedded frames.
16. The method of claim 15 , further including determining motion and appearance differences between the at least one substitute frame and the at least one of the target gradient-embedded frames to detect anomaly.
17. The method of claim 11 , wherein the generating a gradient-embedded frame includes determining an inter-frame gradient between the given frame and a neighbor frame, the neighbor frame being a frame interval away from the given frame in the sequence of frames, and adding the inter-frame gradient to the given frame conditioned on a threshold.
18. A system comprising:
a processor; and
a memory device coupled with the processor;
the processor configured to:
receive a sequence of frames of a video as a training data set;
for a given frame in the sequence of frames, generate a gradient-embedded frame corresponding to the given frame, the gradient-embedded frame incorporating motion information among a set of neighboring frames of the given frame, the motion information represented as disturbance in the gradient-embedded frame, wherein a plurality of gradient-embedded frames are generated corresponding to a plurality of the sequence of frames; and
based on the plurality of gradient-embedded frames, train a generative adversarial network to learn to suppress the disturbance in the gradient-embedded frame to generate a substitute frame.
19. The system of claim 18 , wherein to generate the gradient-embedded frame, the processor is configured to determine an inter-frame gradient between the given frame and a neighbor frame, the neighbor frame being a frame interval away from the given frame in the sequence of frames, and to add the inter-frame gradient to the given frame conditioned on a threshold.
20. The system of claim 18 , wherein the generative adversarial network is trained using sequence of frames of the video having no anomaly and used to detect anomaly on video data having anomaly.