Question response generation using language models and live stream video data
A media detection system receives a video corresponding to a fixed field of view. The media detection system may receive user input indicating one or more object types to identify or a subset of the video within which to identify objects. The media detection system applies one or more machine-learned classifiers to frames of the video and creates a summary video that includes the background of the video and identified instances for simultaneous playback within the fixed field of view. The media detection system may also identify instances of objects in a live video stream and use the identified instances to respond to user questions. The media detection system applies a language model to questions to identify the subject matter of the questions, identifies content within the live video stream associated with the subject matter, and uses the identified content to respond to the user's question.
1 . A computer-implemented method, the method comprising:
receiving a question asked by a user;
applying a language model to the received question to identify a subject matter of the question;
generating a response shell based on the received question, the response shell including an answer placeholder associated with the identified subject matter of the question;
determining an average state associated with the identified subject matter of the question;
identifying a live video stream corresponding to the identified subject matter of the question;
applying one or more machine-learned classifiers to frames of the live video stream to identify content within the live video stream associated with the identified subject matter;
determining a current state associated with the identified subject matter of the question based on the identified content within the live video stream;
modifying the response shell by replacing the answer placeholder with information representative of a comparison between the average state associated with the identified subject matter and the current state associated with the identified subject matter to produce a response; and
presenting the response to the user.
2 . The computer-implemented method of claim 1 , wherein the subject matter of the question pertains to one or more of traffic, weather, crowd size, or event status.
3 . The computer-implemented method of claim 1 , wherein the response shell comprises conversational text.
4 . The computer-implemented method of claim 1 , wherein identifying a live video stream comprises querying one or more live video streams based on one or more of a location, camera type, or keywords associated with the one or more live video streams.
5 . The computer-implemented method of claim 1 , wherein the live video stream corresponds to a fixed field of view such that a background for the live video stream is consistent for an above-threshold portion of the live video stream.
6 . The computer-implemented method of claim 1 , wherein information representative of the identified content comprises information describing the type, quantity, or state of the identified content.
7 . The computer-implemented method of claim 1 , wherein presenting the response to the user comprises presenting a text or speech response.
8 . A non-transitory computer-readable storage medium storing executable computer instructions that when executed by a hardware processor are configured to cause the hardware processor to perform steps comprising:
receiving a question asked by a user;
applying a language model to the received question to identify a subject matter of the question;
generating a response shell based on the received question, the response shell including an answer placeholder associated with the identified subject matter of the question;
determining an average state associated with the identified subject matter of the question;
identifying a live video stream corresponding to the identified subject matter of the question;
applying one or more machine-learned classifiers to frames of the live video stream to identify content within the live video stream associated with the identified subject matter;
determining a current state associated with the identified subject matter of the question based on the identified content within the live video stream;
modifying the response shell by replacing the answer placeholder with information representative of a comparison between the average state associated with the identified subject matter and the current state associated with the identified subject matter to produce a response; and
presenting the response to the user.
9 . The non-transitory computer-readable storage medium of claim 8 , wherein the subject matter of the question pertains to one or more of traffic, weather, crowd size, or event status.
10 . The non-transitory computer-readable storage medium of claim 8 , wherein the response shell comprises conversational text.
11 . The non-transitory computer-readable storage medium of claim 8 , wherein instructions for identifying a live video stream comprise instructions for querying one or more live video streams based on one or more of a location, camera type, or keywords associated with the one or more live video streams.
12 . The non-transitory computer-readable storage medium of claim 8 , wherein the live video stream corresponds to a fixed field of view such that a background for the live video stream is consistent for an above-threshold portion of the live video stream.
13 . The non-transitory computer-readable storage medium of claim 8 , wherein information representative of the identified content comprises information describing the type, quantity, or state of the identified content.
14 . The non-transitory computer-readable storage medium of claim 8 , wherein instructions for presenting the response to the user comprise instructions for presenting a text or speech response.
15 . A system, comprising:
a computer processor; and
a non-transitory memory storing executable computer instructions that when executed by the computer processor are configured to cause the computer processor to perform steps comprising:
receiving a question asked by a user;
applying a language model to the received question to identify a subject matter of the question;
generating a response shell based on the received question, the response shell including an answer placeholder associated with the identified subject matter of the question;
determining an average state associated with the identified subject matter of the question;
identifying a live video stream corresponding to the identified subject matter of the question;
applying one or more machine-learned classifiers to frames of the live video stream to identify content within the live video stream associated with the identified subject matter;
determining a current state associated with the identified subject matter of the question based on the identified content within the live video stream;
modifying the response shell by replacing the answer placeholder with information representative of a comparison between the average state associated with the identified subject matter and the current state associated with the identified subject matter to produce a response; and
presenting the response to the user.
16 . The system of claim 15 , wherein the subject matter of the question pertains to one or more of traffic, weather, crowd size, or event status.
17 . The system of claim 15 , wherein the response shell comprises conversational text.
18 . The system of claim 15 , wherein instructions for identifying a live video stream comprise instructions for querying one or more live video streams based on one or more of a location, camera type, or keywords associated with the one or more live video streams.