Techniques for dense video descriptions
Techniques and apparatus for generating dense natural language descriptions for video content are described. In one embodiment, for example, an apparatus may include at least one memory and logic, at least a portion of the logic comprised in hardware coupled to the at least one memory, the logic to receive a source video comprising a plurality of frames, determine a plurality of regions for each of the plurality of frames, generate at least one region-sequence connecting the determined plurality of regions, apply a language model to the at least one region-sequence to generate description information comprising a description of at least a portion of content of the source video. Other embodiments are described and claimed.
1. An apparatus, comprising:
at least one memory; and
logic, at least a portion of the logic comprised in hardware coupled to the at least one memory, the logic to:
receive a source video comprising a plurality of frames;
determine a plurality of regions for the plurality of frames;
generate at least one region-sequence connecting the determined plurality of regions based on at least one selection criterion, the at least one selection criterion comprises a coherency selection criterion configured to maximize a cosine similarity between the plurality of regions of the at least one-region sequence; and
apply a language model to the at least one region-sequence to generate description information comprising a description of at least a portion of content of the source video.
2. The apparatus of claim 1 , the logic to manage a search space to generate the at least one region-sequence using a greedy sub-selection process.
3. The apparatus of claim 1 , the logic to generate a captioned video comprises at least one of the plurality of frames annotated with the at least one region-sequence and the description information.
4. The apparatus of claim 1 , the description information comprises a natural language description of at least one of the plurality of regions.
5. The apparatus of claim 1 , the logic is further to determine the at least one region-sequence based on a second selection criterion, the second selection criterion comprises an informativeness selection criterion configured to maximize information in the at least one region-sequence.
6. The apparatus of claim 1 , the logic is further to determine the at least one region-sequence based on a third selection criterion, the third selection criterion comprises a divergency selection criterion configured to maximally separate the plurality of regions of the at least one region-sequence in terms of divergence.
7. The apparatus of claim 1 , the logic is further to process at least one frame of the plurality of frames via a computational model to generate a response map comprising at least one anchor point representing at least one region of the plurality of regions.
8. The apparatus of claim 7 , the logic is further to process the at least one frame via a computational model comprising a convolutional neural network (CNN).
9. The apparatus of claim 7 , the logic is further to process the at least one frame via a computational model comprising a convolutional neural network (CNN) trained with a multi-instance multi-label learning (MIMLL) process.
10. The apparatus of claim 1 , the language model comprises a sequence-to-sequence learning framework comprising a plurality of long-short term memory networks (LSTMs).
11. A method, comprising:
receiving a source video comprising a plurality of frames;
determining a plurality of regions for each of the plurality of frames;
generating at least one region-sequence connecting the determined plurality of regions based on at least one selection criterion, the at least one selection criterion comprises a coherency selection criterion configured to maximize a cosine similarity between the plurality of regions of the at least one-region sequence; and
applying a language model to the at least one region-sequence to generate description information comprising a description of at least a portion of content of the source video.
12. The method of claim 11 , comprising managing a search space to generate the at least one region-sequence using a greedy sub-selection process.
13. The method of claim 11 , further comprising generating a captioned video comprising at least one of the plurality of frames annotated with the at least one region-sequence and the description information.
14. The method of claim 11 , wherein determining the at least one region-sequence comprises determining the at least one region-sequence based on a second selection criterion, the second selection criterion comprises an informativeness selection criterion configured to maximize information in the at least one region-sequence.
15. The method of claim 11 , wherein determining the at least one region-sequence comprises determining the at least one region-sequence based on a third selection criterion, the third selection criterion comprises a divergency selection criterion configured to maximally separate the plurality of regions of the at least one region-sequence in terms of divergence.
16. The method of claim 11 , further comprising processing at least one frame of the plurality of frames via a computational model comprising a convolutional neural network (CNN).
17. A non-transitory computer-readable storage medium that stores executable computer instructions for execution by processing circuitry of a computing device, the instructions to cause the computing device to:
receive a source video comprising a plurality of frames;
determine a plurality of regions for each of the plurality of frames;
generate at least one region-sequence connecting the determined plurality of regions based on at least one selection criterion, the at least one selection criterion comprises a coherency selection criterion configured to maximize a cosine similarity between the plurality of regions of the at least one-region sequence; and
apply a language model to the at least one region-sequence to generate description information comprising a description of at least a portion of content of the source video.
18. The non-transitory computer-readable storage medium of claim 17 , the executable computer instructions to cause the computing device to manage a search space to generate the at least one region-sequence using a greedy sub-selection process.
19. The non-transitory computer-readable storage medium of claim 17 , the executable computer instructions to cause the computing device to generate a captioned video comprising at least one of the plurality of frames annotated with the at least one region-sequence and the description information.
20. The non-transitory computer-readable storage medium of claim 17 , the executable computer instructions to cause the computing device to process at least one frame of the plurality of frames via a computational model comprising a convolutional neural network (CNN).