Granted Patent B2

US 12,327,084 · App. 17/954,767 · Granted Jun 10, 2025

Inventors: Bohao Feng (Beijing, CN); Yuxin Liu (Beijing, CN)

Assignee: BEIJING BAIDU NETCOM SCIENCE TECHNOLOGY CO., LTD.

G06F40/279G06F40/30

Abstract

Claims (47)

1. A computer-implemented method for video question answering, comprising:

determining M key frames for a video corresponding to a to-be-answered question, M being a positive integer greater than 1 and less than or equal to a number of video frames in the video, including: for any video frame in the video, extracting object information in the video frame respectively, and generating a sentence describing the objects in the video frame using conjunctions in a corpus and the object information; in response to the generated sentence conforming to a grammatical rule, outputting the generated sentence as description information of the video frame, and otherwise, updating the conjunctions until a sentence conforming to the grammatical rule is generated and outputting the generated sentence as description information of the video frame; acquiring scores of correlation between the description information of respective video frames and the question; and sorting the video frames in the descending order of the scores of correlation, and using the first M video frames after the sort as the key frames; and

acquiring vector representations of the key frames; and determining the answer corresponding to the question according to the vector representations of the key frames, including: determining a question type of the question, if the question is an intuitive question, determining the corresponding answer using the vector representation of each key frame and the question, wherein the intuitive question is a question which can be answered directly using the information in the video; and if the question is a non-intuitive question, determining the corresponding answer using the vector representation of each key frame, the question and a corresponding knowledge graph, wherein the non-intuitive question is a question which may not be answered directly using the information in the video, and the knowledge graph comprises a generic knowledge graph and/or a special knowledge graph constructed from the video,

the method further comprising:

for any key frame, acquiring a product of the vector representation of a previous key frame and a predetermined fourth weight parameter and a product of the vector representation of the key frame and a predetermined fifth weight parameter; adding the obtained two products and a predetermined second bias vector, and performing a hyperbolic tangent neural network activation function operation on the added sum; and multiplying an operation result by a predetermined sixth weight parameter, and acquiring the timing attention weight corresponding to the key frame based on the obtained product, the previous key frame being located before the key frame and closest to the key frame in time,

wherein the determining the answer corresponding to the question according to the vector representations of the key frames comprises: determining the answer corresponding to the question according to the updated vector representations of the key frames.

2. The method according to claim 1 , wherein the acquiring vector representations of the key frames comprises:

performing the following processing operations on any key frame:

extracting a target region of the key frame;

extracting features of the key frame to obtain a feature vector corresponding to the key frame;

extracting features of any extracted target region to obtain a feature vector corresponding to the target region, and generating a vector representation of the target region according to the feature vector corresponding to the target region; and

generating the vector representation of the key frame according to the vector representation of each extracted target region and the feature vector corresponding to the key frame.

3. The method according to claim 2 , wherein the generating a vector representation of the target region according to the feature vector corresponding to the target region comprises:

splicing the feature vector corresponding to the target region and at least one of the following vectors: a feature vector corresponding to the key frame where the target region is located, a text vector corresponding to the key frame where the target region is located, and an audio vector corresponding to the video; and acquiring the vector representation of the target region based on a splicing result;

the text vector is obtained after text information extracted from the key frame where the target region is located is transformed to a vector, and the audio vector is obtained after text information corresponding to an audio of the video is transformed to a vector.

4. The method according to claim 3 , wherein the acquiring the vector representation of the target region based on a splicing result comprises:

acquiring a spatial attention weight corresponding to the target region, multiplying the spatial attention weight by the splicing result corresponding to the target region, and using a multiplying result as the vector representation of the target region.

5. The method according to claim 4 , wherein the acquiring a spatial attention weight corresponding to the target region comprises:

determining the spatial attention weight corresponding to the target region according to a question vector and the splicing result corresponding to the target region, the question vector being obtained after text information corresponding to the question is transformed to a vector.

6. The method according to claim 5 , wherein the determining the spatial attention weight corresponding to the target region according to a question vector and the splicing result corresponding to the target region comprises:

acquiring a product of the question vector and a predetermined first weight parameter and a product of the splicing result and a predetermined second weight parameter;

adding the obtained two products and a predetermined first bias vector, and performing a hyperbolic tangent neural network activation function operation on an added sum; and

multiplying an operation result by a predetermined third weight parameter, and acquiring the spatial attention weight corresponding to the target region based on the obtained product.

7. The method according to claim 2 , wherein the generating the vector representation of the key frame according to the vector representation of each extracted target region and the feature vector corresponding to the key frame comprises:

splicing the vector representation of each target region and the feature vector corresponding to the key frame, and using a splicing result as the vector representation of the key frame.

8. An electronic device, comprising:

at least one processor; and

a memory communicatively connected with the at least one processor;

wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a method for video question answering, wherein the method comprises:

determining M key frames for a video corresponding to a to-be-answered question, M being a positive integer greater than 1 and less than or equal to a number of video frames in the video, including: for any video frame in the video, extracting object information in the video frame respectively, and generating a sentence describing the objects in the video frame using conjunctions in a corpus and the object information; in response to the generated sentence conforming to a grammatical rule, outputting the generated sentence as description information of the video frame, and otherwise, updating the conjunctions until a sentence conforming to the grammatical rule is generated and outputting the generated sentence as description information of the video frame; acquiring scores of correlation between the description information of respective video frames and the question; and sorting the video frames in the descending order of the scores of correlation, and using the first M video frames after the sort as the key frames; and

acquiring vector representations of the key frames; and determining the answer corresponding to the question according to the vector representations of the key frames, including: determining a question type of the question, if the question is an intuitive question, determining the corresponding answer using the vector representation of each key frame and the question, wherein the intuitive question is a question which can be answered directly using the information in the video; and if the question is a non-intuitive question, determining the corresponding answer using the vector representation of each key frame, the question and a corresponding knowledge graph, wherein the non-intuitive question is a question which may not be answered directly using the information in the video, and the knowledge graph comprises a generic knowledge graph and/or a special knowledge graph constructed from the video,

the method further comprising:

for any key frame, acquiring a product of the vector representation of a previous key frame and a predetermined fourth weight parameter and a product of the vector representation of the key frame and a predetermined fifth weight parameter; adding the obtained two products and a predetermined second bias vector, and performing a hyperbolic tangent neural network activation function operation on the added sum; and multiplying an operation result by a predetermined sixth weight parameter, and acquiring the timing attention weight corresponding to the key frame based on the obtained product, the previous key frame being located before the key frame and closest to the key frame in time,

wherein the determining the answer corresponding to the question according to the vector representations of the key frames comprises: determining the answer corresponding to the question according to the updated vector representations of the key frames.

9. The electronic device according to claim 8 ,

wherein the acquiring vector representations of the key frames comprises: performing the following processing operations on any key frame: extracting a target region of the key frame; extracting features of the key frame to obtain a feature vector corresponding to the key frame; extracting features of any extracted target region to obtain a feature vector corresponding to the target region, and generating a vector representation of the target region according to the feature vector corresponding to the target region; and generating the vector representation of the key frame according to the vector representation of each extracted target region and the feature vector corresponding to the key frame.

10. The electronic device according to claim 9 ,

wherein the generating a vector representation of the target region according to the feature vector corresponding to the target region comprises: splicing the feature vector corresponding to the target region and at least one of the following vectors: a feature vector corresponding to the key frame where the target region is located, a text vector corresponding to the key frame where the target region is located, and an audio vector corresponding to the video; and acquire the vector representation of the target region based on a splicing result;

the text vector is obtained after text information extracted from the key frame where the target region is located is transformed to a vector, and the audio vector is obtained after text information corresponding to an audio of the video is transformed to a vector.

11. The electronic device according to claim 10 ,

wherein the acquiring the vector representation of the target region based on a splicing result comprises: acquiring a spatial attention weight corresponding to the target region, multiplying the spatial attention weight by the splicing result corresponding to the target region, and using a multiplying result as the vector representation of the target region.

12. A non-transitory computer readable storage medium with computer instructions stored thereon, wherein the computer instructions are used for causing a method for video question answering, wherein the method comprises:

determining M key frames for a video corresponding to a to-be-answered question, M being a positive integer greater than 1 and less than or equal to a number of video frames in the video, including: for any video frame in the video, extracting object information in the video frame respectively, and generating a sentence describing the objects in the video frame using conjunctions in a corpus and the object information; in response to the generated sentence conforming to a grammatical rule, outputting the generated sentence as description information of the video frame, and otherwise, updating the conjunctions until a sentence conforming to the grammatical rule is generated and outputting the generated sentence as description information of the video frame; acquiring scores of correlation between the description information of respective video frames and the question; and sorting the video frames in the descending order of the scores of correlation, and using the first M video frames after the sort as the key frames; and

acquiring vector representations of the key frames; and determining the answer corresponding to the question according to the vector representations of the key frames, including: determining a question type of the question, if the question is an intuitive question, determining the corresponding answer using the vector representation of each key frame and the question, wherein the intuitive question is a question which can be answered directly using the information in the video; and if the question is a non-intuitive question, determining the corresponding answer using the vector representation of each key frame, the question and a corresponding knowledge graph, wherein the non-intuitive question is a question which may not be answered directly using the information in the video, and the knowledge graph comprises a generic knowledge graph and/or a special knowledge graph constructed from the video,

the method further comprising:

for any key frame, acquiring a product of the vector representation of a previous key frame and a predetermined fourth weight parameter and a product of the vector representation of the key frame and a predetermined fifth weight parameter; adding the obtained two products and a predetermined second bias vector, and performing a hyperbolic tangent neural network activation function operation on the added sum; and multiplying an operation result by a predetermined sixth weight parameter, and acquiring the timing attention weight corresponding to the key frame based on the obtained product, the previous key frame being located before the key frame and closest to the key frame in time,

wherein the determining the answer corresponding to the question according to the vector representations of the key frames comprises: determining the answer corresponding to the question according to the updated vector representations of the key frames.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (43)