Video scene describer
Examples of the present disclosure describe a video scene describer. The video scene describer receives video content data as input and provides audio description (AD) data as output. The video scene describer utilizes one or more components using artificial intelligence (AI) and/or algorithms to analyze and describe the video content data. For example, the video scene describer may include a video indexer component to identify and describe particular aspects of the video content data and generates video insights data based on the analysis. The video indexer provides video insights data to a large language model (LLM) component. The video scene describer may additionally include a visual-language model system, which includes a visual encoder, a relation aggregator, a transformer encoder, and/or transformer. The LLM component synthesizes the video insights data and video embedding data, along with any prompt (e.g., a request or question) or dialogue context, to provide the AD data.
1 . A system comprising:
a processing system; and
memory comprising executable instructions that when executed, perform operations, comprising:
receiving, by a visual-language model system, video content data;
generating, by the visual-language model system, video embedding data based at least in part on the video content data;
receiving video insights data comprising speech-to-text (STT) data, optical character recognition (OCR) data, and facial recognition data;
providing the video embedding data and the video insights data to a large language model (LLM) component;
receiving, from the LLM component, audio description (AD) data based at least in part on the video embedding data and the video insights data, wherein the AD data is received from the LLM component in an audio format and describes at least one insight of the video insights data; and
providing the AD data to a device.
2 . The system of claim 1 , wherein the LLM component generates the AD data based on an auto-recursive algorithm, the operations further comprising:
generating at least one previous AD data, wherein at least one of the video embedding data, the video insights data, or the AD data correspond to a given shot, and wherein the at least one previous AD data corresponds to at least one previous shot before the given shot.
3 . The system of claim 2 , wherein generating the AD data using the auto-recursive algorithm comprises generating the AD data based on at least one of the video embedding data, the video insights data, or the at least one previous AD data.
4 . The system of claim 1 , wherein the video embedding data comprises at least one of audio embeddings or RGB embeddings.
5 . The system of claim 1 , wherein the video insights data further comprises at least one of transcripts, objects, clothing, age, gender, emotion, or landmarks.
6 . The system of claim 1 , wherein the visual-language model system comprises at least one of a visual encoder, a transformer encoder, a relation aggregator, or a transformer.
7 . The system of claim 6 , wherein the transformer encoder uses cross attention to combine visual and language data of video content data into a unified video embedding.
8 . The system of claim 1 , wherein the video embedding data comprises at least one vector of numbers.
9 . The system of claim 1 , wherein generating the AD data comprises concatenating the video embedding data with the video insights data using a plurality of delimiters.
10 . The system of claim 1 , wherein generating the AD data comprises performing cross-attention on the video embedding data and the video insights data.
11 . The system of claim 1 , wherein the AD data comprises a textual description of at least one of:
audio elements of the video content data;
visual elements of the video content data;
explicit elements of the video content data; or
implicit elements of the video content data.
12 . A system comprising:
a processing system; and
memory comprising executable instructions that when executed, perform operations, comprising:
receiving, by a visual-language model system, video content data;
providing, to a large language model (LLM) component, video embedding data of the video content data;
receiving, from a video indexer component, video insights data of the video content data;
providing, to the LLM component, input data comprising the video embedding data, the video insights data, and first audio description (AD) data of the video content data;
receiving, from the LLM component, second AD data of the video content data based at least in part on the input data, wherein the second AD data is received from the LLM component in an audio format and describes at least one insight of the video insights data; and
providing the second AD data to a device.
13 . The system of claim 12 , wherein the at least one of the video embedding data, the video insights data, or the second AD data correspond to a given shot, and wherein the first AD data corresponds to at least one previous shot before the given shot.
14 . The system of claim 12 , wherein the video embedding data, the video insights data, and the second AD data correspond to a given frame, and wherein the first AD data corresponds to previous frames before the given frame.
15 . The system of claim 12 , the operations further comprising:
providing the second AD data to a narrator tool.
16 . A system comprising:
a processing system; and
memory comprising executable instructions that when executed, perform operations, comprising:
receiving, by a visual-language model system, video content data;
generating, by the visual-language model system, video embedding data based on the video content data;
receiving, from a video indexer component, video insights data based on the video content data;
providing, to a large language model (LLM) component, the video embedding data and the video insights data;
creating concatenated data by concatenating, by the LLM component, the video embedding data with the video insights data using a plurality of delimiters; and
providing, by the LLM component, audio description (AD) data for presentation based on the concatenated data, wherein the AD data is received from the LLM component in an audio format and describes at least one insight of the video insights data.
17 . The system of claim 16 , wherein the plurality of delimiters indicate the separation of the video insights data from the video embedding data.
18 . The system of claim 16 , wherein the concatenating further comprises concatenating at least one previous AD data with the video embedding data and the video insights data.
19 . The system of claim 16 , wherein the video insights data comprises at least one of optical character recognition (OCR) data, speech-to-text (STT) data, audio effects data, emotion data, keywords, object tracking data, or topics inference data.
20 . The system of claim 16 , wherein the video embedding data comprises at least one of audio embeddings or RGB embeddings.