Graph convolutional networks for video grounding
Method and apparatus that includes receiving a query describing an aspect in a video, the video including a plurality of frames, identifying multiple proposals that potentially correspond to the query where each of the proposals includes a subset of the plurality of frames, ranking the proposals using a graph convolution network that identifies relationships between the proposals, and selecting, based on the ranking, one of the proposals as a video segment that correlates to the query.
1. A method comprising:
receiving a query describing an aspect in a video, the video comprising a plurality of frames;
identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames;
generating a graph based on the query and the multiple proposals;
ranking the proposals using a graph convolution network (GCN) that identifies relationships between the proposals, wherein the graph is input into the graph convolution network; and
selecting, based on the ranking, one of the proposals as a video segment that correlates to the query.
2. The method of claim 1 , wherein generating the graph comprises:
identifying visual features in the proposals using a visual feature encoder; and
generating query features from the query using a recurrent neural network (RNN).
3. The method of claim 2 ,
wherein the graph comprises nodes and edges based on the visual features and the query features.
4. The method of claim 3 , wherein ranking the proposals comprises:
updating node features for the nodes in the graph; and
calculating edge weights for the edges in the graph.
5. The method of claim 3 , wherein ranking the proposals further comprises:
performing node aggregation; and
ranking the proposals based on the node aggregation and results from processing the graph using the GCN.
6. The method of claim 1 , wherein at least two proposals of the multiple proposals comprise overlapping frames of the plurality of frames in the video.
7. The method of claim 1 , wherein at least two proposals of the multiple proposals comprise subsets of the plurality of frames in the video that do not overlap.
8. A system, comprising:
a processor; and
memory comprising a program, which when executed by the processor performs an operation, the operation comprising:
receiving a query describing an aspect in a video, the video comprising a plurality of frames;
identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames;
generating a graph based on the query and the multiple proposals;
ranking the proposals using a GCN that identifies relationships between the proposals, wherein the graph is input into the graph convolution network; and
selecting, based on the ranking, one of the proposals as a video segment that correlates to the query.
9. The system of claim 8 , wherein generating the graph comprises:
identifying visual features in the proposals using a visual feature encoder; and
generating query features from the query using a recurrent neural network (RNN).
10. The system of claim 9 ,
wherein the graph comprises nodes and edges based on the visual features and the query features.
11. The system of claim 10 , wherein ranking the proposals comprises:
updating node features for the nodes in the graph; and
calculating edge weights for the edges in the graph.
12. The system of claim 10 , wherein ranking the proposals further comprises:
performing node aggregation; and
ranking the proposals based on the node aggregation and results from processing the graph using the GCN.
13. The system of claim 8 , wherein at least two proposals of the multiple proposals comprise overlapping frames of the plurality of frames in the video.
14. The system of claim 8 , wherein at least two proposals of the multiple proposals comprise subsets of the plurality of frames in the video that do not overlap.
15. A computer program product for identifying a video segment that correlates to a query, the computer program product comprising:
a computer readable storage medium having computer readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation, the operation comprising:
receiving the query, the query describing an aspect in a video comprising a plurality of frames;
identifying multiple proposals that potentially correspond to the query, wherein each of the proposals comprises a subset of the plurality of frames;
generating a graph based on the query and the multiple proposals;
ranking the proposals using a GCN that identifies relationships between the proposals, wherein the graph is input into the graph convolution network; and
selecting, based on the ranking, one of the proposals as the video segment that correlates to the query.
16. The computer program product of claim 15 , wherein generating the graph comprises:
identifying visual features in the proposals using a visual feature encoder; and
generating query features from the query using a recurrent neural network (RNN).
17. The computer program product of claim 16 ,
wherein the graph comprises nodes and edges based on the visual features and the query features.
18. The computer program product of claim 17 , wherein ranking the proposals comprises:
updating node features for the nodes in the graph; and
calculating edge weights for the edges in the graph.
19. The computer program product of claim 17 , wherein ranking the proposals further comprises:
performing node aggregation; and
ranking the proposals based on the node aggregation and results from processing the graph using the GCN.
20. The computer program product of claim 15 , wherein at least two proposals of the multiple proposals comprise overlapping frames of the plurality of frames in the video.