Dynamic video overlays
A client device accesses a video input stream from an intermediate device for display. The client device analyzes the video input stream to determine that the video input stream matches a template corresponding to a screen portion. Based on the video input stream matching the template, a video output stream is generated and caused to be presented on a display. In some example embodiments, the analysis is performed while the client device is replacing video content received from a content source via the intermediate device. For example, commercials transmitted from a national content provider to a smart TV via a set-top box may be replaced with targeted commercials. During the replacement, menus generated by the set-top box may be detected and the replacement video altered by the smart TV to include the menus.
1. A system, comprising:
a display;
memory that stores instructions and a plurality of templates; and
one or more processors of a client device configured by the instructions to perform operations comprising:
accessing a video input stream that includes first video content from a content provider and second video content generated by a set-top box device upstream of the one or more processors, wherein, in a frame of the video input stream, the first video content corresponds to a first screen portion of the display and the second video content corresponds to a second screen portion of the display;
generating, from the first video content, a query fingerprint;
accessing, based on a comparison of the query fingerprint and a reference fingerprint, replacement video content provided by a replacement content source, wherein the replacement video content differs from the first video content, wherein the replacement video content is accessed separately from the video input stream;
identifying, based on the set-top box device, a template of the plurality of templates, wherein the identified template indicates that the second screen portion comprises an overlay for displaying the second video content generated by the set-top box device;
comparing the video input stream to the identified template to determine that the frame of the video input stream corresponds to the identified template;
responsive to a determination that the frame of the video input stream corresponds to the identified template, generating a video output stream comprising the second video content for the second screen portion and the video replacement content for the first screen portion to mimic a presentation of the first video content and the second video content in the video input stream; and
causing the video output stream to be presented on the display.
2. The system of claim 1 , wherein accessing the replacement video content comprises:
identifying, based on the comparison, the first video content provided by the content provider; and
identifying the replacement video content based on the identification of the first video content provided by the content provider.
3. The system of claim 1 , wherein accessing the replacement video content comprises requesting the replacement video content from the replacement content source.
4. The system of claim 1 , wherein the system is a smart television.
5. The system of claim 1 , wherein generating the video output stream comprises copying the second video content in the second screen portion from the video input stream to the video output stream on a pixel-by-pixel basis.
6. The system of claim 1 , wherein comparing the video input stream to the identified template comprises:
downsampling the frame of the video input stream to form a downsampled frame;
determining a cross-correlation between the downsampled frame and the identified template;
performing a comparison of the cross-correlation to a threshold; and
determining, based on the comparison, that the frame of the video input stream corresponds to the identified template.
7. The system of claim 1 , wherein the operations further comprise:
accessing an identifier of the set-top box device; and
selecting a subset of the plurality of templates based on the identifier of the set-top box device,
wherein comparing the video input stream to the identified template to determine that the frame of the video input stream corresponds to the identified template is a sub-step of comparing each template of the subset of the plurality of templates to determine whether the frame of the video input stream corresponds to the template.
8. A method comprising:
accessing, at a display device, a video input stream that includes first video content from a content provider and second video content generated by a set-top box device upstream of the display device, wherein, in a frame of the video input stream, the first video content corresponds to a first screen portion of the display device and the second video content corresponds to a second screen portion of the display decice;
generating, from the first video content, a query fingerprint;
accessing, based on a comparison of the query fingerprint and a reference fingerprint, replacement video content provided by a replacement content source, wherein the replacement video content differs from the first video content, wherein the replacement video content is accessed separately from the video input stream;
identifying, based on the set-top box device, a template of the plurality of templates, wherein the identified template indicates that the second screen portion comprises an overlay for displaying the second video content generated by the set-top box device;
comparing the video input stream to the identified template to determine that the frame of the video input stream corresponds to the identified template;
responsive to a determination that the frame of the video input stream corresponds to the identified template, generating a video output stream comprising the second video content for the second screen portion and the replacement video content for the first screen portion to mimic a presentation of the first video content and the second video content in the video input stream; and
causing the video output stream to be presented on the display device.
9. The method of claim 8 , wherein accessing the replacement video content comprises:
identifying, based on the comparison, the first video content provided by the content provider; and
identifying the replacement video content based on the identification of the first video content provided by the content provider.
10. The method of claim 8 , wherein accessing the replacement video content comprises requesting the replacement video content from the replacement content source.
11. The method of claim 8 , wherein the display device is a smart television.
12. The method of claim 8 , wherein generating the video output stream comprises copying the second video content in the second screen portion from the video input stream to the video output stream on a pixel-by-pixel basis.
13. The method of claim 8 , wherein comparing the video input stream to the identified template comprises:
downsampling the frame of the video input stream to form a downsampled frame;
determining a cross-correlation between the downsampled frame and the identified template;
performing a comparison of the cross-correlation to a threshold; and
determining, based on the comparison, that the frame of the video input stream corresponds to the identified template.
14. The method of claim 8 , further comprising:
accessing an identifier of the set-top box device; and
selecting a subset of the plurality of templates based on the identifier of the set-top box device,
wherein comparing the video input stream to the identified template to determine that the frame of the video input stream corresponds to the identified template is a sub-step of comparing each template of the subset of the plurality of templates to determine whether the frame of the video input stream corresponds to the template.
15. A non-transitory machine-readable medium that stores instructions which, when executed by one or more processors of a client device, cause the client device to perform operations comprising:
accessing, by the one or more processors, a video input stream that includes first video content from a content provider and second video content generated by a set-top box device upstream of the one or more processors, wherein, in a frame of the video input stream, the first video content corresponds to a first screen portion of a display and the second video content corresponds to a second screen portion of the display;
generating, from the first video content, a query fingerprint;
accessing, based on a comparison of the query fingerprint and a reference fingerprint, replacement video content provided by a replacement content source, wherein the replacement video content differs from the first video content, wherein the replacement video content is accessed separately from the video input stream;
identifying, based on the set-top box device, a template of the plurality of templates, wherein the identified template indicates that the second screen portion comprises an overlay for displaying the second video content generated by the set-top box device;
comparing the video input stream to the identified template to determine that the frame of the video input stream corresponds to the identified template;
responsive to a determination that the frame of the video input stream corresponds to the identified template, generating a video output stream comprising the second video content for the second screen portion and the replacement video content for the first screen portion to mimic a presentation of the first video content and the second video content in the video input stream; and
causing the video output stream to be presented on the display.
16. The non-transitory machine-readable medium of claim 15 , wherein accessing the replacement video content comprises:
identifying, based on the comparison, the first video content provided by the content provider; and
identifying the replacement video content based on the identification of the first video content provided by the content provider.
17. The non-transitory machine-readable medium of claim 15 , wherein accessing the replacement video content comprises requesting the replacement video content from the replacement content source.
18. The non-transitory machine-readable medium of claim 15 , wherein generating the video output stream comprises copying the second video content in the second screen portion from the video input stream to the video output stream on a pixel-by-pixel basis.
19. The non-transitory machine-readable medium of claim 15 , wherein comparing the video input stream to the identified template comprises:
downsampling the frame of the video input stream to form a downsampled frame;
determining a cross-correlation between the downsampled frame and the identified template;
performing a comparison of the cross-correlation to a threshold; and
determining, based on the comparison, that the frame of the video input stream corresponds to the identified template.
20. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
accessing an identifier of the set-top box device; and
selecting a subset of the plurality of templates based on the identifier of the set-top box device,
wherein comparing the video input stream to the identified template to determine that the frame of the video input stream corresponds to the identified template is a sub-step of comparing each template of the subset of the plurality of templates to determine whether the frame of the video input stream corresponds to the template.