Multimodal content recognition and contextual advertising and content delivery
The present disclosure is directed to providing supplemental content to one or more client devices requesting multimedia content. The supplemental content may be determined or selected based on the availability of one or more device channels, such as a display, speakers, or other component of the one or more client devices capable of providing an output. Furthermore, the supplemental content may be determined or selected based on the portions of the requested multimedia content that are the most prominent or significant, such as any audio content, any video content, and/or any textual content. The supplemental content may be provided to the one or more client devices such that it is displayed before, during, or after the display of the requested multimedia content.
1 . A computerized method comprising:
monitoring for changes in availability of output channels for a device;
detecting an output channel availability change in the device providing a multimedia content stream having an audio portion output via a speaker of the device and a video portion output via a display of the device, wherein the output channel availability change is a change between video output and audio output, wherein detecting the output channel availability change includes identifying that either the video output or the audio output is no longer available;
after detecting the output channel availability change, selecting supplemental content based on a supplemental content type comprising one of supplemental audio content and supplemental video content such that the supplemental content is still playable by an available output channel of the device after the output channel availability change; and
providing a content stream comprising the multimedia content and the selected supplemental content having the supplemental content type.
2 . The method of claim 1 , further comprising:
detecting frame changes at the device;
determining the video portion of the multimedia content stream is being provided at the device based on detection of the frame changes; and
selecting the supplemental video content based on determining the video portion of the multimedia content stream is being provided.
3 . The method of claim 1 , further comprising:
detecting no image changes at the device;
determining the audio portion of the multimedia content stream is being provided at the device based on detecting no image changes; and
selecting the supplemental audio content based on determining the audio portion of the multimedia content stream is being provided.
4 . The method of claim 1 , further comprising:
based on detecting the change in the output device channel availability, selecting a same supplemental content having a different supplemental content type; and
changing the content stream to include the same supplemental content using the different supplemental content type.
5 . The method of claim 1 , further comprising:
receiving a request from the device to change the supplemental content type; and
responsive to receiving the request, providing a same supplemental content having a different supplemental content type.
6 . The method of claim 1 , further comprising:
determining that no output device channels are available to provide the audio portion and the video portion; and
based on determining no output device channels are available, stopping the content stream.
7 . One or more non-transitory computer-readable media storing executable instructions which, when executed by a processor, cause the processor to perform operations comprising:
monitoring for changes in availability of output channels for a device;
for a multimedia content stream configured to be provided via a plurality of output device channels of the device, detecting an output channel availability change in the device between a first output device channel and a second output device channel that is providing multimedia content of the multimedia content stream, wherein detecting the output channel availability change includes identifying that either the first output device channel or the second output device channel is no longer available;
after detecting the output channel availability change in the device, selecting supplemental content having a supplemental content type, each of a plurality of supplemental content types for the supplemental content corresponds to a different output device channel of the device and the supplemental content is selected such that the supplemental content is still playable by an available output channel of the device after the output channel availability change; and
providing a content stream comprising the multimedia content and the selected supplemental content having the supplemental content type.
8 . The media of claim 7 , further comprising determining whether the first output device channel or the second output device channel is providing the multimedia content based on frame changes at a display of the device.
9 . The media of claim 7 , further comprising determining whether the first output device channel or the second output device channel is providing the multimedia content is based on no identified image changes at a display of the device.
10 . The media of claim 7 , further comprising scoring the supplemental content, and selecting the supplemental content based on a score for the supplemental content.
11 . The media of claim 7 , further comprising:
based on detecting the change in the output device channel availability, selecting a same supplemental content having a different supplemental content type; and
changing the content stream to include the same supplemental content using the different supplemental content type.
12 . The media of claim 7 , further comprising:
receiving a request from the device to change the supplemental content type; and
responsive to receiving the request, providing a same supplemental content having a different supplemental content type.
13 . The media of claim 7 , further comprising:
determining that no output device channels are available to provide the supplemental content type; and
based on determining no output device channels are available, stopping the content stream.
14 . A device comprising:
a speaker;
a display;
at least one processor; and
a memory storing instructions which, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
monitoring for changes in availability of output channels for the device;
detecting, for a multimedia content stream configured to provide audio via the speaker and video via the display, an output channel availability change in the device between an audio output and a video output, wherein detecting the output channel availability change includes identifying that either the video output or the audio output is no longer available;
after detecting the output channel availability change, receiving supplemental content having a supplemental content type comprising one of supplemental audio content and supplemental video content, such that the supplemental content is still playable by an available output channel of the device after the output channel availability change; and
providing a content stream comprising the multimedia content and the received supplemental content having the supplemental content type.
15 . The device of claim 14 , further comprising determining the video output of the multimedia content stream is being provided based on frame changes at the display.
16 . The device of claim 14 , further comprising determining the audio output of the multimedia content stream is being provided based on detecting no image changes at the display.
17 . The device of claim 14 , further comprising:
providing an indication of the output channel availability change;
responsive to the indication, receiving a same supplemental content having a different supplemental content type; and
providing the content stream that includes the same supplemental content using a different device output channel selected from one of the speaker and the display.
18 . The device of claim 14 , further comprising:
providing a request to change the supplemental content type for the supplemental content; and
responsive to the request, providing a same supplemental content using a different device output channel selected from one of the speaker and the display.