Question answering apparatus and method
A question answering method that is performed by a question answering apparatus includes: receiving a data set including video content and question-answer pairs; generating input time-series sequences from the video content of the input data set and also generating a question-answer time-series sequence from the question-answer pair of the input data set; calculating weights by associating the input time-series sequence with the question-answer time-series sequence and also calculating first result values by performing operations on the calculated weights and the input time-series sequences; calculating second result values by paying attention to portions of the input time-series sequences that are directly related to characters appearing in questions and answers; and calculating third result values by concatenating the time-series sequences, the first result values, the second result values, and Boolean flags and selecting a final answer based on the third result values.
1. A question answering method that is performed by a question answering apparatus, the question answering method comprising:
a step of receiving a data set including video content and question-answer pairs based on a user input;
a preprocessing step of generating input time-series sequences from the video content of the input data set and also generating a question-answer time-series sequence from the question-answer pair of the input data set;
a step of calculating weights by associating the input time-series sequence with the question-answer time-series sequence and also calculating first result values by performing operations on the calculated weights and the input time-series sequences;
a step of calculating second result values by paying attention to portions of the input time-series sequences that are directly related to characters appearing in questions and answers;
a step of calculating third result values by concatenating the time-series sequences, the first result values, the second result values, and Boolean flags and selecting a final answer based on the third result values; and
a step of outputting the final answer,
wherein the preprocessing step further comprises:
a step of generating time-series data by concatenating the pieces of data included in the data set in sequence;
a step of generating a feature vector including a related character by extracting a word vector and an image feature vector from the time-series data and concatenating the extracted vectors with related character information of the time-series data as an one-hot vector; and
a step of generating a time-series sequence having a contextual flow by inputting the feature vector including a related character to a bidirectional Long/Short Term Memory (bi-LSTM) model.
2. The question answering method of claim 1 , wherein the data set comprises question-answer pairs, a script in which utterers are indicated, visual metadata (behaviors and emotions), and visual bounding boxes.
3. The question answering method of claim 1 , wherein the step of calculating second result values comprises a step of calculating third result values using dot-product attention and multi-head attention.
4. A question answering apparatus comprising:
a storage unit configured to store a program that performs question answering; and
a control unit including at least one processor;
wherein when a data set including video content and question-answer pairs is received by executing the program based on a user input, the control unit:
generates input time-series sequences from the video content of the input data set, and also generates a question-answer time-series sequence from the question-answer pair of the input data set;
calculates weights by associating the input time-series sequence with the question-answer time-series sequence, and also calculates first result values by performing operations on the calculated weights and the input time-series sequences;
calculates second result values by paying attention to portions of the input time-series sequences that are directly related to characters appearing in questions and answers;
calculates third result values by concatenating the time-series sequences, the first result values, the second result values, and Boolean flags, and selects a final answer based on the third result values; and
outputs the final answer,
wherein when generating the input time-series sequences and the question-answer time-series sequence from the input data set, the control unit:
generates time-series data by concatenating the pieces of data included in the data set in sequence;
generates a feature vector including a related character by extracting a word vector and an image feature vector from the time-series data and concatenating the extracted vectors with related character information of the time-series data as an one-hot vector; and
generates a time-series sequence having a contextual flow by inputting the feature vector including a related character to a bidirectional Long/Short Term Memory (bi-LSTM) model.
5. The question answering apparatus of claim 4 , wherein the data set comprises question-answer pairs, a script in which utterers are indicated, visual metadata (behaviors and emotions), and visual bounding boxes.
6. The question answering apparatus of claim 4 , wherein when calculating second result values, the control unit calculates third result values using dot-product attention and multi-head attention.
7. A non-transitory computer-readable storage medium having stored thereon a program that performs the method set forth in claim 1 .
8. A computer program that is executed by a question answering apparatus and stored in a non-transitory computer-readable medium to perform the method set forth in claim 1 .