Neural network based position estimation of target object of interest in video frames
Visual target tracking is task of locating a target in consecutive frame of a video. Conventional systems observe target behavior frames of the video. However, dealing with this problem is very challenging when video has illumination variations, occlusion, change in size and view of the object due to relative motion between camera and object. Embodiments of the present disclosure addresses this problem by implementing Neural Network (NN), its features and their corresponding gradients. Present disclosure explicitly guides the NN by feeding target object of interest (ToI) defined by a bounding box in the first frame of the video. With this guidance, NN generates target activation map via convolutional features map and their gradient maps, thus giving tentative location of the ToI to further exploit to locate target object precisely by using correlation filter(s) and peak location estimator, thus repeating process for every frame of video to track ToI accurately.
1. A processor implemented method, comprising:
receiving, at a neural network, a first input comprising a video frame, and a second input comprising a target object of interest in the video frame, wherein a corresponding class label is generated based on the second input;
receiving, at a first part of the neural network, a current frame to generate one or more corresponding convolutional feature maps;
generating, by a softmax layer of the neural network, a probability score for the generated one or more corresponding convolutional feature maps;
generating by a cost function, a cost value based on the generated corresponding class label and the corresponding probability score;
generating one or more gradient maps using the cost value and the generated one or more corresponding convolutional feature maps;
generating one or more target activation maps based on the generated one or more gradient maps and the generated one or more corresponding convolutional feature maps; and
identifying a tentative location of the target object of interest in the video frame based on the one or more target activation maps, wherein the step of identifying the tentative location of the target object of interest in the video frame based on the one or more target activation maps comprises:
sampling, the generated one or more corresponding convolution feature maps into a size of the video frame;
converting the generated one or more gradient maps to a single dimensional vector;
generating the one or more target activation maps based on the size of the video frame and the single dimensional vector; and
identifying the tentative location of the target object of interest in the video frame using the generated one or more target activation maps.
2. The processor implemented method of claim 1 , further comprising identifying, by a peak location estimator, a position of the target object in the video frame based on the tentative location.
3. The processor implemented method of claim 2 , wherein the step of identifying, by a peak estimator, a position of the target object comprises:
extracting one or more patches from one or more neighborhoods of the tentative location of the generated one or more target activation maps;
generating, by using a correlation filter, a correlation filter response for each frame at center of the target object of interest;
identifying a peak value based on the correlation filter response generated for each frame; and
estimating the position of the target object based on the identified peak value in the correlation filter response for each frame.
4. The processor implemented method of claim 1 , wherein the probability score is generated by the softmax layer by utilizing a second part of the neural network.
5. The processor implemented method of claim 3 , further comprising training the correlation filter based on the identified peak value and at least one of a corresponding patch or one or more features extracted from the corresponding patch; and updating one or more parameters pertaining to the correlation filter.
6. A system comprising:
a memory storing instructions;
one or more communication interfaces; and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive, by a neural network, a first input comprising a video frame, and a second input comprising a target object of interest in the video frame, wherein a corresponding class label is generated based on the second input;
receive, at a first part of the neural network, a current frame to generate one or more corresponding convolutional feature maps;
generate, by using a softmax layer of the neural network, a probability score for the generated one or more corresponding convolutional feature maps;
generate by a cost function of the neural network, a cost value based on the generated corresponding class label and the corresponding probability score;
generate one or more gradient maps using the cost value and the generated one or more corresponding convolutional feature maps;
generate, by using a target activation map generator, one or more target activation maps based on the generated one or more gradient maps and the generated one or more corresponding convolutional feature maps; and
identify a tentative location of the target object of interest in the video frame based on the one or more target activation maps, wherein the tentative location of the target object of interest in the video frame is identified based on the one or more target activation maps by:
sampling, by using a sampler, the generated one or more corresponding convolution feature maps into a size of the video frame and converting the generated one or more gradient maps to a single dimensional vector;
generating, by the target activation map generator, the one or more target activation maps based on the size of the video frame and the single dimensional vector; and
identifying the tentative location of the target object of interest in the video frame using the generated one or more target activation maps.
7. The system of claim 6 , further comprising a peak location estimator that identifies a position of the target object in the video frame based on the tentative location.
8. The system of claim 7 , wherein the position of the target object is identified by:
extracting, by using a pre-process and patch of interest extractor, one or more patches from one or more neighborhoods of the tentative location of the generated one or more target activation maps;
generating, by using a correlation filter, a correlation filter response for each frame at center of the target object of interest;
identifying, by using a peak estimator, a peak value based on the correlation filter response generated for each frame; and
estimating the position of the target object based on the identified peak value in the correlation filter response for each frame.
9. The system of claim 6 , wherein the probability score is generated by the softmax layer by utilizing a second part of the neural network.
10. The system of claim 8 , wherein the one or more hardware processors are configured by the instructions to: train the correlation filter based on the identified peak value and at least one of a corresponding or one or more features extracted from the corresponding patch; and update one or more parameters pertaining to the correlation filter.
11. One or more non-transitory machine readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
receiving, at a neural network, a first input comprising a video frame, and a second input comprising a target object of interest in the video frame, wherein a corresponding class label is generated based on the second input;
receiving, at a first part of the neural network, a current frame to generate one or more corresponding convolutional feature maps;
generating, by a softmax layer of the neural network, a probability score for the generated one or more corresponding convolutional feature maps;
generating by a cost function, a cost value based on the generated corresponding class label and the corresponding probability score;
generating one or more gradient maps using the cost value and the generated one or more corresponding convolutional feature maps;
generating one or more target activation maps based on the generated one or more gradient maps and the generated one or more corresponding convolutional feature maps; and
identifying a tentative location of the target object of interest in the video frame based on the one or more target activation maps, wherein the step of identifying the tentative location of the target object of interest in the video frame based on the one or more target activation maps comprises:
sampling, the generated one or more corresponding convolution feature maps into a size of the video frame;
converting the generated one or more gradient maps to a single dimensional vector;
generating the one or more target activation maps based on the size of the video frame and the single dimensional vector; and
identifying the tentative location of the target object of interest in the video frame using the generated one or more target activation maps.
12. The one or more non-transitory machine readable information storage mediums of claim 11 , wherein the instructions which when executed by the one or more hardware processors further cause identifying, by a peak location estimator, a position of the target object in the video frame based on the tentative location.
13. The one or more non-transitory machine readable information storage mediums of claim 12 , wherein the step of identifying, by a peak estimator, a position of the target object comprises:
extracting one or more patches from one or more neighborhoods of the tentative location of the generated one or more target activation maps;
generating, by using a correlation filter, a correlation filter response for each frame at center of the target object of interest;
identifying a peak value based on the correlation filter response generated for each frame; and
estimating the position of the target object based on the identified peak value in the correlation filter response for each frame.
14. The one or more non-transitory machine readable information storage mediums of claim 11 , wherein the probability score is generated by the softmax layer by utilizing a second part of the neural network.
15. The one or more non-transitory machine readable information storage mediums of claim 13 , wherein the instructions which when executed by the one or more hardware processors further cause training the correlation filter based on the identified peak value and at least one of a corresponding patch or one or more features extracted from the corresponding patch; and updating one or more parameters pertaining to the correlation filter.