QUERY OF VIDEO SUBJECT MATTER
Disclosed are systems and methods that convert digital video data, such as two-dimensional digital video data, into a natural language text description describing the subject matter represented in the video. For example, the disclosed implementations may process video data in real-time, near real-time, or after the video data is created and generate a text-based video narrative describing the subject matter of the video. In addition, the disclosed implementations may also support a question and answer session in which a user may submit queries about the subject matter of one or more videos and the disclosed implementations will present natural language responses based on the subject matter of the video and any corresponding context.
1 . A computer-implemented method, comprising:
for each of a plurality of segments of a video:
processing the segment to generate a plurality of embeddings, each embedding of the plurality of embeddings corresponding to at least a portion of a subject matter represented in the segment of the plurality of segments of the video; and
representing the plurality of embeddings as a feature embedding for the segment of the video;
receiving a query about the subject matter of the video;
processing the query to generate a query embedding;
projecting the query embedding in a vector space with each of the plurality of feature embeddings to determine one or more feature embeddings that are responsive to the query;
determining, for each of the one or more feature embeddings that are responsive to the query and based at least on the plurality of embeddings included in the feature embedding, a descriptive text that is descriptive of the feature embedding and the subject matter of the segment corresponding to the feature embedding; and
generating, based at least in part on the descriptive text determined for each of the one or more feature embeddings, a natural language response that is responsive to the query representative of at least a portion of the subject matter of the video.
2 . The computer-implemented method of claim 1 , further comprising:
determining a context for the query, wherein the context is determined based at least in part on one or more of the query, a query and answer session in which the query is presented, the video, or a user; and
wherein generating the natural language response is further based at least in part on the context.
3 . The computer-implemented method of claim 1 , wherein processing the segment to generate the plurality of embeddings, further includes:
processing the segment with a local semantic attention transformer to generate a plurality of local embeddings, wherein the plurality of local embeddings are included in the plurality of embeddings;
processing the segment with a global semantic attention transformer to generate a plurality of global embeddings, wherein the plurality of global embeddings are included in the plurality of embeddings;
processing the segment with a temporal semantic attention transformer to generate a plurality of temporal embeddings, wherein the plurality of temporal embeddings are included in the plurality of embeddings; and
processing the segment with an activity semantic attention transformer to generate a plurality of activity embeddings, wherein the plurality of activity embeddings are included in the plurality of embeddings.
4 . The computer-implemented method of claim 3 , wherein the query includes at least one of a text or an image.
5 . The computer-implemented method of claim 3 , wherein a temporal embedding of the plurality of temporal embeddings is indicative of a dependency of at least one feature of the segment over a duration of the segment.
6 . The computer-implemented method of claim 1 , further comprising:
receiving a second query about the subject matter of the video;
processing the query to determine the query is a request for a report relating to at least a portion of the subject matter of the video; and
generating, in a natural language format and based at least in part on at least a portion of the feature embeddings generated for each segment of the video, the report.
7 . A system, comprising:
one or more processors; and
a memory storing program instructions that, when executed by the one or more processors, cause the one or more processors to at least:
receive a query about a subject matter of a video;
process the query to generate a query embedding;
project the query embedding in a vector space with each of a plurality of feature embeddings to determine one or more feature embeddings that are responsive to the query, wherein each of the plurality of feature embeddings is representative of a segment of a plurality of segments of the video;
determine, for each of the one or more feature embeddings that are responsive to the query and based at least on a plurality of embeddings included in the feature embedding, a descriptive text that is descriptive of the feature embedding and the subject matter of the segment corresponding to the feature embedding; and
generate, based at least in part on the descriptive text determined for each of the one or more feature embeddings, a natural language response that is responsive to the query representative of at least a portion of the subject matter of the video.
8 . The system of claim 7 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
determine a context for the query, wherein the context is determined based at least in part on one or more of the query, a query and answer session in which the query is presented, the video, or a user; and
wherein generation of the natural language response is further based at least in part on the context.
9 . The system of claim 7 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
for each of a plurality of segments of the video:
process the segment with a local semantic attention transformer to generate a plurality of local embeddings, wherein the plurality of local embeddings are included in the feature embedding for the segment of the video;
process the segment with a global semantic attention transformer to generate a plurality of global embeddings, wherein the plurality of global embeddings are included in the feature embedding for the segment of the video;
process the segment with a temporal semantic attention transformer to generate a plurality of temporal embeddings, wherein the plurality of temporal embeddings are included in the feature embedding for the segment of the video; and
process the segment with an activity semantic attention transformer to generate a plurality of activity embeddings, wherein the plurality of activity embeddings are included in the feature embedding for the segment of the video.
10 . The system of claim 7 ,
wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least determine an activity of a plurality of activities that is occurring in the subject matter of the video; and
wherein the program instructions that cause the one or more processors to generate the natural language response further include program instructions that, when executed by the one or more processors, further cause the one or more processors to at least, generate the natural language response based at least in part on the activity and the feature embeddings.
11 . The system of claim 7 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
present the natural language response at least one of audibly, visually, or haptically.
12 . The system of claim 7 , wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
receive a second query about the subject matter of the video;
process the query to determine the query is a request for a report relating to at least a portion of the subject matter of the video; and
generate, in a natural language format and based at least in part on at least a portion of the feature embeddings generated for each segment of the video, the report.
13 . The system of claim 7 ,
wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least determine, based at least in part on the query embedding, a second plurality of segments of a second video that represent subject matter that is responsive to the query; and
wherein the program instructions that cause the one or more processors to generate the natural language response, further include program instructions that, when executed by the one or more processors, further cause the one or more processors to at least determine, based at least in part on the feature embeddings generated for each of the plurality of segments of the video and feature embeddings generated for each of the second plurality of segments, the natural language response.
14 . A method, comprising:
processing a plurality of segments of a video to produce, for each segment, a feature embedding corresponding to the segment;
process a query about a subject matter of the video to generate a query embedding;
projecting the query embedding in a vector space with each of a plurality of feature embeddings to determine a feature embedding of the plurality of feature embeddings that is responsive to the query;
determining, for the feature embedding that is responsive to the query and based at least on a plurality of embeddings included in the feature embedding, a descriptive text that is descriptive of the feature embedding and the subject matter of the segment corresponding to the feature embedding; and
generating, based at least in part on the descriptive text determined for the feature embedding, a natural language response that is responsive to the query representative of at least a portion of the subject matter of the video.
15 . The method of claim 14 , further comprising:
determining a context based at least in part on one or more of the query, a query and answer session in which the query is presented, the video, or a user; and
wherein generating the natural language response is based at least in part on the feature embeddings generated for each of the plurality of segments and the context.
16 . The method of claim 14 , further comprising:
for each of a plurality of segments of the video:
processing the segment with a local semantic attention transformer to generate a plurality of local embeddings, wherein the plurality of local embeddings are included in the feature embedding for the segment of the video;
processing the segment with a global semantic attention transformer to generate a plurality of global embeddings, wherein the plurality of global embeddings are included in the feature embedding for the segment of the video;
processing the segment with a temporal semantic attention transformer to generate a plurality of temporal embeddings, wherein the plurality of temporal embeddings are included in the feature embedding for the segment of the video; and
processing the segment with an activity semantic attention transformer to generate a plurality of activity embeddings, wherein the plurality of activity embeddings are included in the feature embedding for the segment of the video.
17 . The method of claim 14 , further comprising:
determining an activity of a plurality of activities that is occurring in the subject matter of the video; and
wherein generating the natural language response further includes generating the natural language response based at least in part on the activity and the feature embeddings.
18 . The method of claim 14 , further comprising:
determining a context for the query, wherein the context is determined based at least in part on one or more of the query, a query and answer session in which the query is presented, the video, or a user; and
wherein generating the natural language response is further based at least in part on the context.
19 . The method of claim 14 , further comprising:
receiving a second query about the subject matter of the video;
processing the query to determine the query is a request for a report relating to at least a portion of the subject matter of the video; and
generating, in a natural language format and based at least in part on at least a portion of the feature embeddings generated for each segment of the video, the report.
20 . The method of claim 14 , further comprising:
determining, based at least in part on the query embedding, a second plurality of segments of a second video that represent subject matter that is responsive to the query; and
wherein generating the natural language response, further includes determining, based at least in part on the feature embeddings generated for each of the plurality of segments of the video and feature embeddings generated for each of the second plurality of segments, the natural language response.