Interactive real-time video search based on knowledge graph

1 . A computer-implemented method comprising:

obtaining, by a media processing server including one or more machine learning models, a user query, the user query including a request for a live multimedia stream and one or more descriptions associated with the live multimedia stream;

obtaining, by the media processing server, live multimedia data comprising an image portion and an audio portion in response to obtaining the user query, wherein the live multimedia data is associated with the one or more descriptions included in the user query;

generating, by the media processing server, a knowledge graph of the live multimedia data using the one or more machine learning models based on the image portion, the audio portion, and the one or more descriptions included in the user query, wherein the knowledge graph includes a plurality of entities and relationships between entities;

dynamically updating, by the media processing server, the knowledge graph while the live multimedia data is being streamed;

obtaining, by the media processing server, additional data from a source external to the live multimedia data, wherein the additional data comprises additional entities and additional relationships between the additional entities;

obtaining, by the media processing server, a feedback data associated with a previous user query and a previous response to the previous user query;

dynamically adjusting, by the media processing server, weights of the one or more machine learning models using the feedback data;

generating, by the media processing server using the one or more machine learning models with adjusted weights, an augmented knowledge graph by iteratively augmenting the dynamically updated knowledge graph with additional entities and additional relationships between the additional entities of the additional data, wherein the augmented knowledge graph is incorporated with the and the feedback data; and

providing, by the media processing server, a response to the user query based on the augmented knowledge graph.

2 . The computer-implemented method of claim 1 , further comprising:

identifying one or more relevant entities in the augmented knowledge graph based on the user query,

wherein the response is further provided based on the one or more relevant entities, the one or more relevant entities including at least one selected from a group including objects, people, and places.

3 . The computer-implemented method of claim 2 , wherein the response to the user query indicates a determined timespan of the live multimedia data during which the one or more relevant entities are included.

4 . The computer-implemented method of claim 1 , wherein the additional data is obtained based on the user query.

5 . The computer-implemented method of claim 1 , wherein the additional data relates to a particular entity of the plurality of entities in the knowledge graph, wherein the particular entity is an object identified in the image portion of the live multimedia data or a topic identified in the audio portion of the live multimedia data.

6 . The computer-implemented method of claim 1 , wherein the live multimedia data is associated with an enterprise, and wherein the additional data is obtained from a private data source of the enterprise.

7 . The computer-implemented method of claim 1 , wherein the additional data comprises different multimedia data.

8 . The computer-implemented method of claim 1 , wherein the one or more machine learning models include an object recognition model that analyzes video data of the image portion using the one or more descriptions included in the user query to identify the entities and the relationships between entities, and a bidirectional encodings from transformers (BERT) model that analyzes a text transcript of the audio portion to identify the entities and the relationships between the entities.

9 . A system comprising:

one or more computer processors;

one or more computer readable storage media; and

program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising instructions to:

obtain a user query, the user query including a request for a live multimedia stream and one or more descriptions associated with the live multimedia stream;

obtain live multimedia data comprising an image portion and an audio portion in response to obtaining the user query, wherein the live multimedia data is associated with the one or more descriptions included in the user query;

generate a knowledge graph of the live multimedia data using one or more machine learning models based on the image portion, the audio portion, and the one or more descriptions included in the user query, wherein the knowledge graph includes a plurality of entities and relationships between entities;

dynamically update the knowledge graph while the live multimedia data is being streamed;

obtain additional data from a source external to the live multimedia data, wherein the additional data comprises additional entities and additional relationships between the additional entities;

obtain a feedback data associated with a previous user query and a previous response to the previous user query and dynamically adjust weights of the one or more machine learning models using the feedback data;

generate an augmented knowledge graph, using the dynamically adjusted one or more machine learning models, by iteratively augmenting the dynamically updated knowledge graph with additional entities and additional relationships between the additional entities of the additional data, wherein the augmented knowledge graph is incorporated with the feedback data; and

provide a response to the user query based on the augmented knowledge graph.

10 . The system of claim 9 , further comprising instructions to:

identify one or more relevant entities in the augmented knowledge graph based on the user query,

wherein the response is further provided based on the one or more relevant entities, the one or more relevant entities including at least one selected from a group including objects, people, and places.

11 . The system of claim 10 , wherein the response to the user query indicates a determined timespan of the live multimedia data during which the one or more relevant entities are included.

12 . The system of claim 9 , wherein the additional data is obtained based on the user query.

13 . The system of claim 9 , wherein the additional data relates to a particular entity of the plurality of entities in the knowledge graph, wherein the particular entity is an object identified in the image portion of the live multimedia data or a topic identified in the audio portion of the live multimedia data.

14 . One or more non-transitory computer readable storage media having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform operations including:

obtaining a user query, the user query including a request for a live multimedia stream and one or more descriptions associated with the live multimedia stream;

obtaining live multimedia data comprising an image portion and an audio portion in response to obtaining the user query, wherein the live multimedia data is associated with the one or more descriptions included in the user query;

generating a knowledge graph of the live multimedia data using one or more machine learning models based on the image portion, the audio portion, and the one or more descriptions included in the user query, wherein the knowledge graph includes a plurality of entities and relationships between entities;

dynamically updating the knowledge graph while the live multimedia data is being streamed;

obtaining additional data from a source external to the live multimedia data, wherein the additional data comprises additional entities and additional relationships between the additional entities;

obtaining a feedback data associated with a previous user query and a previous response to the previous user query;

dynamically adjusting weights of the one or more machine learning models using the feedback data;

generating an augmented knowledge graph, using the dynamically adjusted one or more machine learning models, by iteratively augmenting the dynamically updated knowledge graph with additional entities and additional relationships between the additional entities of the additional data, wherein the augmented knowledge graph is incorporated with the feedback data; and

providing a response to the user query based on the augmented knowledge graph.

15 . The one or more non-transitory computer readable storage media of claim 14 , wherein the program instructions further cause the computer to perform operations including:

identifying one or more relevant entities in the augmented knowledge graph based on the user query,

wherein the response is further provided based on the

one or more relevant entities, the one or more relevant entities including at least one selected from a group including objects, people, and places.

16 . The one or more non-transitory computer readable storage media of claim 15 , wherein the response to the user query indicates a determined timespan of the live multimedia data during which the one or more relevant entities are included.

17 . The one or more non-transitory computer readable storage media of claim 14 , wherein the additional data is obtained based on the user query.

18 . The computer-implemented method of claim 2 , further comprising:

providing, by the media processing server, a visual knowledge graph comprising the one or more relevant entities and relationships between the one or more relevant entities obtained from the augmented knowledge graph and based on the user query.

19 . The system of claim 10 , further comprising instructions to:

provide a visual knowledge graph comprising the one or more relevant entities and relationships between one or more relevant entities obtained from the augmented knowledge graph and based on the user query.

20 . The one or more non-transitory computer readable storage media of claim 15 , wherein the program instructions further cause the computer to perform operations including:

providing a visual knowledge graph comprising the one or more relevant entities and relationships between the one or more relevant entities obtained from the augmented knowledge graph and based on the user query.