Efficient and fine-grained video retrieval
A computer-implemented method executed by at least one processor for performing mini-batching in deep learning by improving cache utilization is presented. The method includes temporally localizing a candidate clip in a video stream based on a natural language query, encoding a state, via a state processing module, into a joint visual and linguistic representation, feeding the joint visual and linguistic representation into a policy learning module, wherein the policy learning module employs a deep learning network to selectively extract features for select frames for video-text analysis and includes a fully connected linear layer and a long short-term memory (LSTM), outputting a value function from the LSTM, generating an action policy based on the encoded state, wherein the action policy is a probabilistic distribution over a plurality of possible actions given the encoded state, and rewarding policy actions that return clips matching the natural language query.
1. A computer-implemented method executed by at least one processor for performing mini-batching in deep learning by improving cache utilization, the method comprising:
temporally localizing a candidate clip in a video stream based on a natural language query;
encoding a state, via a state processing module, into a joint visual and linguistic representation;
feeding the joint visual and linguistic representation into a policy learning module, wherein the policy learning module employs a deep learning network to selectively extract features for select frames for video-text analysis and includes a fully connected linear layer and a long short-term memory (LSTM);
outputting a value function from the LSTM;
generating an action policy based on the encoded state, wherein the action policy is a probabilistic distribution over a plurality of possible actions given the encoded state; and
rewarding policy actions that return clips matching the natural language query,
wherein a Hadamard multiplication is performed between an attention vector and a mean-pooled result to generate the joint visual and linguistic representation.
2. The method of claim 1 , wherein an agent is trained to guide a fixed-size window around the candidate clip of the video stream to find the clips matching the natural language query.
3. The method of claim 1 , wherein the policy learning module learns to fast-forward and rewind the video stream to locate the matching clips.
4. The method of claim 1 , wherein the natural language query is encoded by a gated recurrent unit, the gate recurrent unit outputting a query embedding vector.
5. The method of claim 4 , wherein the query embedding vector is fed into a fully connected linear layer with a sigmoid activation function.
6. The method of claim 5 , wherein an output of the fully connected linear layer is the attention vector.
7. The method of claim 6 , wherein the candidate clip of the video stream is fed into a three-dimensional convolution network to extract spatio-temporal features from a fifth convolutional layer.
8. The method of claim 7 , wherein the spatio-temporal features are mean-pooled across frames of the video stream to output the mean-pooled result.
9. A system for performing mini-batching in deep learning by improving cache utilization, the system comprising:
a memory; and
a processor in communication with the memory, wherein the processor runs program code to:
temporally localize a candidate clip in a video stream based on a natural language query;
encode a state, via a state processing module, into a joint visual and linguistic representation;
feed the joint visual and linguistic representation into a policy learning module, wherein the policy learning module employs a deep learning network to selectively extract features for select frames for video-text analysis and includes a fully connected linear layer and a long short-term memory (LSTM);
output a value function from the LSTM;
generate an action policy based on the encoded state, wherein the action policy is a probabilistic distribution over a plurality of possible actions given the encoded state; and
reward policy actions that return clips matching the natural language query,
wherein a Hadamard multiplication is performed between an attention vector and a mean-pooled result to generate the joint visual and linguistic representation.
10. The system of claim 9 , wherein an agent is trained to guide a fixed-size window around the candidate clip of the video stream to find the clips matching the natural language query.
11. The system of claim 9 , wherein the policy learning module learns to fast-forward and rewind the video stream to locate the matching clips.
12. The system of claim 9 , wherein the natural language query is encoded by a gated recurrent unit, the gate recurrent unit outputting a query embedding vector.
13. The system of claim 12 , wherein the query embedding vector is fed into a fully connected linear layer with a sigmoid activation function.
14. The system of claim 13 , wherein an output of the fully connected linear layer is the attention vector.
15. The system of claim 14 , wherein the candidate clip of the video stream is fed into a three-dimensional convolution network to extract spatio-temporal features from a fifth convolutional layer.
16. The system of claim 15 , wherein the spatio-temporal features are mean-pooled across frames of the video stream to output the mean-pooled result.
17. A non-transitory computer-readable storage medium comprising a computer-readable program for performing mini-batching in deep learning by improving cache utilization, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
temporally localizing a candidate clip in a video stream based on a natural language query;
encoding a state, via a state processing module, into a joint visual and linguistic representation;
feeding the joint visual and linguistic representation into a policy learning module, wherein the policy learning module employs a deep learning network to selectively extract features for select frames for video-text analysis and includes a fully connected linear layer and a long short-term memory (LSTM);
outputting a value function from the LSTM;
generating an action policy based on the encoded state, wherein the action policy is a probabilistic distribution over a plurality of possible actions given the encoded state; and
rewarding policy actions that return clips matching the natural language query,
wherein a Hadamard multiplication is performed between an attention vector and a mean-pooled result to generate the joint visual and linguistic representation.
18. The non-transitory computer-readable storage medium of claim 17 ,
wherein the natural language query is encoded by a gated recurrent unit, the gate recurrent unit outputting a query embedding vector;
wherein the query embedding vector is fed into a fully connected linear layer with a sigmoid activation function, the output of the fully connected linear layer being the attention vector; and
wherein the candidate clip of the video stream is fed into a three-dimensional convolution network to extract spatio-temporal features from a fifth convolutional layer, the spatio-temporal features mean-pooled across frames of the video stream to output the mean-pooled result.