Dynamic video overlays
A client device accesses a video input stream from an intermediate device for display. The client device analyzes the video input stream to determine that the video input stream matches a template indicating a portion of the video input stream that is created by the intermediate device. Based on the video input stream matching the template, a video output stream generated by the client device is modified to include the portion of the video input stream generated by the intermediate device. For example, commercials transmitted from a national content provider to a smart TV via a set-top box may be replaced with targeted commercials. During the replacement, informational messages or menus generated by the set-top box may be detected and the replacement video altered by the smart TV to include the messages or menus generated by the set-top box.
1. A method comprising:
causing, by one or more processors of a client device, a video output stream to be presented on a display of the client device;
while causing the video output stream to be presented on the display:
accessing, by the one or more processors, a video input stream comprising first video content from a content provider and second video content generated by a set-top box upstream of the one or more processors, wherein, in a frame of the video input stream, the first video content corresponds to a first screen portion of the display and the second video content corresponds to a second screen portion of the display, wherein the video output stream comprises third video content from a replacement content source, wherein the third video content differs from the first video content and is accessed separately from the video input stream, wherein the third video content is selected in response to a video identification system receiving a query fingerprint from a query fingerprint generator of the client device and the video identification system identifying a match between the query fingerprint and at least one reference fingerprint generated by a reference fingerprint generator, wherein the query fingerprint is generated based on a frame of the first video content, and wherein the video identification system and the reference fingerprint generator are remote from the client device;
accessing, from a database that stores a plurality of sets of templates, a template corresponding to the second screen portion of the display, wherein the database maps a plurality of set-top box identifiers to the plurality of sets of templates, wherein the template corresponding to the second screen portion of the display is accessed from a set of templates based on an identifier of the set-top box;
analyzing, by the one or more processors, the video input stream to determine that at least a portion of the second video content of the frame of the video input stream matches the template;
responsive to determining that the at least the portion of the second video content of the frame of the video input stream matches the template, modifying, by the one or more processors, the video output stream to include, in the second screen portion of the display, the second video content generated by the set-top box; and
causing, by the one or more processors, the modified video output stream, including the second video content in the second screen portion and the third video content in the first screen portion, to be presented on the display.
2. The method of claim 1 , further comprising: accessing a second template corresponding to a third screen portion of the display for the second video content generated by the set-top box;
analyzing the video input stream to determine that at least a portion of the second video content of a second frame of the video input stream matches the second template; and
based on the at least the portion of the second video content of the second frame of the video input stream matching the second template, ceasing modification of the video output stream.
3. The method of claim 1 , wherein the modifying of the video output stream to include, in the second screen portion, the second video content generated by the set-top box comprises: copying the second video content in the second screen portion from the video input stream to the video output stream on a pixel-by-pixel basis.
4. The method of claim 1 , wherein the modifying of the video output stream to include, in the second screen portion, the second video content generated by the set-top box comprises: identifying information contained in the second screen portion; and
modifying the video output stream to include the identified information.
5. The method of claim 1 , further comprising:
identifying the first video content of the video input stream from the content provider; and
requesting, based on the identified first video content from the content provider, replacement video content from the replacement content source; and
responsive to requesting replacement video content, receiving the third video content from the replacement content source.
6. The method of claim 1 , wherein the analyzing of the video input stream to determine that at least the portion of the second video content of the frame of the video input stream matches the template comprises determining that the at least the portion of the second video content of the frame of the video input stream contains an icon added by the set-top box.
7. The method of claim 1 , wherein the analyzing of the video input stream to determine that at least the portion of the second video content of the frame of the video input stream matches the template comprises determining that the at least the portion of the second video content of the frame of the video input stream contains predetermined text added by the set-top box.
8. A system comprising: a memory that stores instructions; and one or more processors, of a client device, configured by the instructions to perform operations comprising:
causing a video output stream to be presented on a display of the client device;
while causing the video output stream to be presented on the display:
accessing a video input stream comprising first video content from a content provider and second video content generated by a set-top box upstream of the one or more processors, wherein, in a frame of the video input stream, the first video content corresponds to a first screen portion of the display and the second video content corresponds to a second screen portion of the display, wherein the video output stream comprises third video content from a replacement content source, wherein the third video content differs from the first video content and is accessed separately from the video input stream, wherein the third video content is selected in response to a video identification system receiving a query fingerprint from a query fingerprint generator of the client device and the video identification system identifying a match between the query fingerprint and at least one reference fingerprint generated by a reference fingerprint generator, wherein the query fingerprint is generated based on a frame of the first video content, and wherein the video identification system and the reference fingerprint generator are remote from the client device;
accessing, from a database that stores a plurality of sets of templates, a template corresponding to the second screen portion of the display, wherein the database maps a plurality of set-top box identifiers to the plurality of sets of templates, wherein the template corresponding to the second screen portion of the display is accessed from a set of templates based on an identifier of the set-top box;
analyzing the video input stream to determine that at least a portion of the second video content of the frame of the video input stream matches the template;
responsive to determining that the at least the portion of the second video content of the frame of the video input stream matches the template, modifying the video output stream to include, in the second screen portion of the display, the second video content generated by the set-top box; and causing the modified video output stream, including the second video content in the second screen portion and the third video content in the first screen portion, to be presented on the display.
9. The system of claim 8 , wherein the operations further comprise: accessing a second template corresponding to a third screen portion of the display for the second video content generated by the set-top box; analyzing the video input stream to determine that at least a portion of the second video content of a second frame of the video input stream matches the second template; and based on the at least the portion of the second video content of the second frame of the video input stream matching the second template, ceasing modification of the video output stream.
10. The system of claim 8 , wherein the modifying of the video output stream to include, in the second screen portion, the second video content generated by the set-top box comprises: copying the second video content in the second screen portion from the video input stream to the video output stream on a pixel-by-pixel basis.
11. The system of claim 8 , wherein the modifying of the video output stream to include, in the second screen portion, the second video content generated by the set-top box comprises: identifying information contained in the second screen portion; and modifying the video output stream to include the identified information.
12. The system of claim 8 , wherein: the operations further comprise: identifying the first video content of the video input stream from the content provider; and requesting, based on the identified first video content from the content provider, replacement video content from the replacement content source;
and responsive to requesting replacement video content, receiving the third video content from the replacement content source.
13. The system of claim 8 , wherein the analyzing of the video input stream to determine that at least the portion of the second video content of the frame of the video input stream matches the template comprises determining that the at least the portion of the second video content of the frame of the video input stream contains an icon added by the-set-top box.
14. The system of claim 8 , wherein the analyzing of the video input stream to determine that at least the portion of the second video content of the frame of the video input stream matches the template comprises determining that the at least the portion of the second video content of the frame of the video input stream contains predetermined text added by the set-top box.
15. A non-transitory machine-readable medium that stores instructions that, when executed by one or more processors of a client device, cause the one or more processors to perform operations comprising: causing a video output stream to be presented on a display of the client device; while causing the video output stream to be presented on the display:
accessing a video input stream comprising first video content from a content provider and second video content generated by a set-top box upstream of the one or more processors, wherein, in a frame of the video input stream, the first video content corresponds to a first screen portion of the display and the second video content corresponds to a second screen portion of the display, wherein the video output stream comprises third video content from a replacement content source, wherein the third video content differs from the first video content and is accessed separately from the video input stream, wherein the third video content is selected in response to a video identification system receiving a query fingerprint from a query fingerprint generator of the client device and the video identification system identifying a match between the query fingerprint and at least one reference fingerprint generated by a reference fingerprint generator, wherein the query fingerprint is generated based on a frame of the first video content, and wherein the video identification system and the reference fingerprint generator are remote from the client device;
accessing, from a database that stores a plurality of sets of templates, a template corresponding to the second screen portion of the display, wherein the database maps a plurality of set-top box identifiers to the plurality of sets of templates, wherein the template corresponding to the second screen portion of the display is accessed from a set of templates based on an identifier of the set-top box;
analyzing the video input stream to determine that at least a portion of the second video content of the frame of the video input stream matches the template;
responsive to determining that the at least the portion of the second video content of the frame of the video input stream matches the template, modifying the video output stream to include, in the second screen portion of the display, the second video content generated by the set-top box; and
causing the modified video output stream, including the second video content in the second screen portion and the third video content in the first screen portion, to be presented on the display.
16. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise: accessing a second template corresponding to a third screen portion of the display for the second video content generated by the set-top box; analyzing the video input stream to determine that at least a portion of the second video content of a second frame of the video input stream matches the second template; and based on the at least the portion of the second video content of the second frame of the video input stream matching the second template, ceasing modification of the video output stream.
17. The non-transitory machine-readable medium of claim 15 , wherein the modifying of the video output stream to include, in the second screen portion, the second video content generated by the set-top box comprises: copying the second video content in the second screen portion from the video input stream to the video output stream on a pixel-by-pixel basis.
18. The non-transitory machine-readable medium of claim 15 , wherein the modifying of the video output stream to include, in the second screen portion, the second video content generated by the set-top box comprises: identifying information contained in the second screen portion; and modifying the video output stream to include the identified information.
19. The non-transitory machine-readable medium of claim 15 , wherein: the operations further comprise: identifying the first video content of the video input stream from the content provider; and requesting, based on the identified first video content from the content provider, replacement video content from the replacement content source; and responsive to requesting replacement video content, receiving the third video content from the replacement content source.
20. The non-transitory machine-readable medium of claim 15 , wherein the analyzing of the video input stream to determine that at least the portion of the second video content of the frame of the video input stream matches the template comprises determining that the at least the portion of the second video content of the frame of the video input stream contains an icon added by the set-top box.