Unified embedding with metric learning for zero-exemplar event detection

1. A method of event detection using a neural network, comprising:

propagating each text block of a plurality of text blocks through the neural network to obtain a textual representation of the each text block in a joint space of textual and video representations, each of the plurality of text blocks describing one of a plurality of pre-defined events;

propagating each training video of a plurality of training videos through the neural network to obtain a visual representation of the each training video in the joint space, each of the plurality of training videos associated with one of the plurality of text blocks;

adjusting parameters of the neural network to reduce, for each pair of associated text block and associated training video, a distance in the joint space between a textual representation of the associated text block and a visual representation of the associated training video,

wherein the parameters of the neural network are adjusted based on a loss function, wherein the loss function is further configured to increase a distance in the joint space between a first textual representation of a first event and a second textual representation of a second event, the first event different from the second event;

propagating a text query through the neural network to obtain a textual representation of the text query in the joint space;

propagating each video of a plurality of videos through the neural network to obtain a visual representation of the each video in the joint space; and

ranking each video of the plurality of videos based on a distance in the joint space between the textual representation of the text query and the visual representation of the each video, wherein the ranking of each video is based on the increase of the distance of different events in the joint space by the loss function.

2. The method of claim 1 , further comprising selecting a set of videos from the plurality of videos based on the ranking, wherein visual representations of the set of videos are closest to the textual representation of the text query in the joint space.

3. The method of claim 1 , wherein the text query is provided to the neural network unaltered by semantic query generation.

4. The method of claim 1 , wherein the propagating the each video of the plurality of videos comprises:

extracting a representation for each frame of the each video; and

summing up representations of all frames of the each video to obtain a representation of the each video.

5. The method of claim 1 , wherein the visual representation of each video in the joint space is obtained without using metadata or title of the video.

6. An apparatus for event detection using a neural network, comprising:

means for propagating each text block of a plurality of text blocks through the neural network to obtain a textual representation of the each text block in a joint space of textual and video representations, each of the plurality of text blocks describing one of a plurality of pre-defined events;

means for propagating each training video of a plurality of training videos through the neural network to obtain a visual representation of the each training video in the joint space, each of the plurality of training videos associated with one of the plurality of text blocks;

means for adjusting parameters of the neural network to reduce, for each pair of associated text block and associated training video, a distance in the joint space between a textual representation of the associated text block and a visual representation of the associated training video,

wherein the parameters of the neural network are adjusted based on a loss function, wherein the loss function is further configured to increase a distance in the joint space between a first textual representation of a first event and a second textual representation of a second event, the first event different from the second event;

means for propagating a text query through the neural network to obtain a textual representation of the text query in the joint space;

means for propagating each video of a plurality of videos through the neural network to obtain a visual representation of the each video in the joint space; and

means for ranking each video of the plurality of videos based on a distance in the joint space between the textual representation of the text query and the visual representation of the each video, wherein the rank of each video is based on the increase of the distance of different events in the joint space by the loss function.

7. The apparatus of claim 6 , further comprising means for selecting a set of videos from the plurality of videos based on the ranking, wherein visual representations of the set of videos are closest to the textual representation of the text query in the joint space.

8. The apparatus of claim 6 , wherein the text query is provided to the neural network unaltered by semantic query generation.

9. The apparatus of claim 6 , wherein the means for propagating the each video of the plurality of videos is configured to:

extract a representation for each frame of the each video; and

sum up representations of all frames of the each video to obtain a representation of the each video.

10. An apparatus for event detection using a neural network, comprising:

a memory; and

at least one processor coupled to the memory and configured to:

propagate each text block of a plurality of text blocks through the neural network to obtain a textual representation of the each text block in a joint space of textual and video representations, each of the plurality of text blocks describing one of a plurality of pre-defined events;

propagate each training video of a plurality of training videos through the neural network to obtain a visual representation of the each training video in the joint space, each of the plurality of training videos associated with one of the plurality of text blocks;

adjust parameters of the neural network to reduce, for each pair of associated text block and associated training video, a distance in the joint space between a textual representation of the associated text block and a visual representation of the associated training video,

propagate a text query through the neural network to obtain a textual representation of the text query in the joint space;

propagate each video of a plurality of videos through the neural network to obtain a visual representation of the each video in the joint space; and

rank each video of the plurality of videos based on a distance in the joint space between the textual representation of the text query and the visual representation of the each video, wherein the rank of each video is based on the increase of the distance of different events in the joint space by the loss function;

wherein the parameters of the neural network are adjusted based on a loss function, wherein the loss function is further configured to increase a distance in the joint space between a first textual representation of a first event and a second textual representation of a second event, the first event different from the second event.

11. The apparatus of claim 10 , wherein the at least one processor is further configured to select a set of videos from the plurality of videos based on the ranking, wherein visual representations of the set of videos are closest to the textual representation of the text query in the joint space.

12. The apparatus of claim 10 , wherein the text query is provided to the neural network unaltered by semantic query generation.

13. The apparatus of claim 10 , wherein, to propagate the each video of the plurality of videos, the at least one processor is configured to:

extract a representation for each frame of the each video; and

sum up representations of all frames of the each video to obtain a representation of the each video.

14. The apparatus of claim 10 , wherein the visual representation of each video in the joint space is obtained without using metadata or title of the video.