Bi-directional spatial-temporal reasoning for video-grounded dialogues
Systems and methods are provided for performing a video-grounded dialogue task by a neural network model using bi-directional spatial-temporal reasoning. According to some embodiments, the systems and methods implement a dual network architecture or framework. This framework includes one network or reasoning module that learns dependencies between text and video in the direction of spatial→temporal, and another network or reasoning module that learns in the direction of temporal→spatial. The output of the multimodal reasoning modules may be combined to learn dependencies between language features in dialogues. The result joint representation is used as a contextual feature to the decoding components which allow the model to semantically generate meaningful responses to the users. In some embodiments, pointer networks are extended to the video-grounded dialogue task to allow the model to point to specific tokens from multiple source sequences to generate responses.
1. A method for performing a video-grounded dialogue task by a neural network model, the method comprising:
receiving features for a video and text sequences, wherein the text sequences comprise a dialogue history between the model and a human user regarding the video and a current query by the human user;
encoding the video features and text sequences into continuous representations;
using the encoded continuous representations, learning dependencies between the text sequences and the video features in the direction of temporal to spatial;
using the encoded continuous representations, learning dependencies between the text sequences and the video features in the direction of spatial to temporal; and
using the learned dependencies in the direction of temporal to spatial and in the direction of spatial to temporal, generating a textual response in natural language for the current query.
2. The method of claim 1 , comprising dynamically combining outputs of the learned dependencies to generate a joint query-aware contextual vector.
3. The method of claim 1 , wherein generating a textual response comprises performing video attention and textual attention using the learned dependencies to incorporate relevant information for the model.
4. The method of claim 3 , comprising generating one or more tokens to materialize the textual response in natural language.
5. The method of claim 4 , wherein generating a textual response comprises performing a linear transformation to obtain a distribution over a vocabulary set.
6. The method of claim 1 , comprising outputting the textual response in natural language to the human user in response to the current query.
7. The method of claim 1 , wherein the video features comprise one or more of video data, audio data, and caption data related to at least one video.
8. The method of claim 1 , wherein encoding the text sequences comprises performing token-level embedding.
9. The method of claim 1 , wherein encoding the video features comprises extracting spatial-temporal visual features.
10. The method of claim 1 , wherein learning dependencies comprises using the current query to select information along temporal steps of the video features.
11. A non-transitory machine-readable medium comprising executable code which when executed by one or more processors associated with a computer are adapted to cause the one or more processors to perform a method for performing a video-grounded dialogue task by a neural network model comprising:
receiving features for a video and text sequences, wherein the text sequences comprise a dialogue history between the model and a human user regarding the video and a current query by the human user;
encoding the video features and text sequences into continuous representations;
using the encoded continuous representations, learning dependencies between the text sequences and the video features in the direction of temporal to spatial;
using the encoded continuous representations, learning dependencies between the text sequences and the video features in the direction of spatial to temporal; and
using the learned dependencies in the direction of temporal to spatial and in the direction of spatial to temporal, generating a textual response in natural language for the current query.
12. The non-transitory machine-readable medium of claim 11 , comprising dynamically combining outputs of the learned dependencies to generate a joint query-aware contextual vector.
13. The non-transitory machine-readable medium of claim 11 , wherein generating a textual response comprises performing video attention and textual attention using the learned dependencies to incorporate relevant information for the model.
14. The non-transitory machine-readable medium of claim 13 , comprising generating one or more tokens to materialize the textual response in natural language.
15. The non-transitory machine-readable medium of claim 14 , wherein generating a textual response comprises performing a linear transformation to obtain a distribution over a vocabulary set.
16. The non-transitory machine-readable medium of claim 11 , comprising outputting the textual response in natural language to the human user in response to the current query.
17. The non-transitory machine-readable medium of claim 11 , wherein the video features comprise one or more of video data, audio data, and caption data related to at least one video.
18. The non-transitory machine-readable medium of claim 11 , wherein encoding the text sequences comprises performing token-level embedding.
19. The non-transitory machine-readable medium of claim 11 , wherein encoding the video features comprises extracting spatial-temporal visual features.
20. The non-transitory machine-readable medium of claim 11 , wherein learning dependencies comprises using the current query to select information along temporal steps of the video features.