Dynamic video overlays
View Patent ↗A client device accesses a video input stream from an intermediate device for display. The client device analyzes the video input stream to determine that the video input stream matches a template indicating a semi-transparent overlay. Based on the video input stream matching the template, a video output stream is generated and caused to be presented on a display. In some example embodiments, the analysis is performed while the client device is replacing video content received from a content source via the intermediate device. For example, commercials transmitted from a national content provider to a smart TV via a set-top box may be replaced with targeted commercials. During the replacement, semi-transparent menus generated by the set-top box may be detected and the replacement video altered by the smart TV to include the menus.
1. A system comprising:
a memory having instructions embodied thereon; and
one or more processors configured by the instructions to perform operations comprising:
accessing, from an intermediate device upstream of the one or more processors, a video input stream that includes first content and a semi-transparent overlay overlaid at least partially on the first content in an overlay zone, wherein the first content is from a content provider and the semi-transparent overlay is overlaid on the first content by the intermediate device;
accessing a template that includes an indication of the overlay zone in the video input stream;
analyzing, using the template, the video input stream to determine that the video input stream includes the semi-transparent overlay in the overlay zone;
accessing replacement content for replacing the first content outside of the overlay zone;
while accessing the video input stream, generating a video output stream that includes a modified semi-transparent overlay overlaid on the replacement content in the overlay zone; and
causing the video output stream, including the replacement content outside of the overlay zone and the modified semi-transparent overlay in the overlay zone, to be presented on a display.
2. The system of claim 1 , wherein the operations further comprise:
modifying the semi-transparent overlay to cancel out the first content presented in the overlay zone, using operations comprising:
computing a difference between the first content and the video input stream for each pixel in the overlay zone; and
replacing each pixel in the overlay zone based on the difference and a threshold.
3. The system of claim 2 , wherein the operations further comprise:
generating a fingerprint of the first content; and
identifying the first content based on the fingerprint.
4. The system of claim 2 , wherein:
the template indicates a degree of transparency; and
the replacing of each pixel in the overlay zone is further based on the degree of transparency.
5. The system of claim 4 , wherein:
the replacing of each pixel in the overlay zone is further based on a corresponding pixel of the first content.
6. The system of claim 1 , wherein:
the indication of the overlay zone in the video input stream comprises coordinates of boundaries of the overlay zone.
7. The system of claim 1 , wherein:
the indication of the overlay zone in the video input stream comprises a bitmap.
8. The system of claim 1 , wherein the operations further comprise:
modifying the semi-transparent overlay to cancel out the first content presented in the overlay zone, using operations comprising:
identifying an average color of a background of the overlay zone in the video input stream;
comparing a color of each pixel of a frame of the first content lacking the semi-transparent overlay to the average color of the background; and
for each of the pixels having a color within a predetermined range of the average color of the background, changing the color of the pixel to the average color of the background.
9. The system of claim 1 , wherein the operations further comprise:
identifying a text portion of the semi-transparent overlay; and
modifying the semi-transparent overlay to cancel out the first content while maintaining the text portion of the semi-transparent overlay without change.
10. The system of claim 9 , wherein the identifying of the text portion of the semi-transparent overlay comprises:
identifying, based on the template, a region that may contain text; and
determining, based on color values of pixels within the region and a threshold, which pixels within the region are included in the text portion of the semi-transparent overlay.
11. The system of claim 1 , wherein:
the template indicates a color; and
the generating of the video output stream that includes the modified semi-transparent overlay overlaid on the replacement content comprises semi-transparently applying the color to at least a portion of the overlay zone.
12. The system of claim 11 , wherein:
the template indicates an information type of text to be presented; and
the generating of the video output stream that includes the modified semi-transparent overlay overlaid on the replacement content further comprises adding text to at least a portion of the overlay zone, the added text based on the information type indicated in the template.
13. A method comprising:
accessing, by one or more processors, a video input stream that includes first content and a semi-transparent overlay overlaid at least partially on the first content in an overlay zone, wherein the first content is from a content provider and the semi-transparent overlay is overlaid on the first content by an intermediate device that is upstream of the one or more processors;
accessing, from a database that stores a plurality of templates, a template that includes an indication of the overlay zone in the video input stream;
analyzing, using the template, the video input stream to determine that the video input stream includes the semi-transparent overlay in the overlay zone;
accessing replacement content for replacing the first content outside of the overlay zone;
while accessing the video input stream, generating, by one or more processors, a video output stream that includes a modified semi-transparent overlay overlaid on the replacement content in the overlay zone; and
causing, by the one or more processors, the video output stream, including the replacement content outside of the overlay zone and the modified semi-transparent overlay in the overlay zone, to be presented on a display.
14. The method of claim 13 , further comprising:
modifying the semi-transparent overlay to cancel out the first content presented in the overlay zone, using operations comprising:
computing a difference between the first content and the video input stream for each pixel in the overlay zone; and
replacing each pixel in the overlay zone based on the difference and a threshold.
15. The method of claim 14 , further comprising:
generating a fingerprint of the first content; and
identifying the first content based on the fingerprint.
16. The method of claim 14 , wherein:
the template indicates a degree of transparency; and
the replacing of each pixel in the overlay zone is further based on the degree of transparency.
17. The method of claim 16 , wherein:
the replacing of each pixel in the overlay zone is further based on a corresponding pixel of the first content.
18. The method of claim 13 , further comprising:
modifying the semi-transparent overlay to cancel out the first content presented in the overlay zone, using operations comprising:
identifying an average color of a background of the overlay zone in the video input stream;
comparing a color of each pixel of a frame of the first content lacking the semi-transparent overlay to the average color of the background; and
for each of the pixels having a color within a predetermined range of the average color of the background, changing the color of the pixel to the average color of the background.
19. The method of claim 13 , further comprising:
identifying a text portion of the semi-transparent overlay; and
modifying the semi-transparent overlay to cancel out the first content while maintaining the text portion of the semi-transparent overlay without change.
20. A non-transitory machine-readable medium that stores instructions which, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
accessing, from an intermediate device upstream of the one or more processors, a video input stream that includes first content and a semi-transparent overlay overlaid at least partially on the first content in an overlay zone, wherein the first content is from a content provider and the semi-transparent overlay is overlaid on the first content by the intermediate device;
accessing a template that includes an indication of the overlay zone in the video input stream;
analyzing, using the template, the video input stream to determine that the video input stream includes the semi-transparent overlay in the overlay zone;
accessing replacement content for replacing the first content outside of the overlay zone;
while accessing the video input stream, generating a video output stream that includes a modified semi-transparent overlay overlaid on the replacement content in the overlay zone; and
causing the video output stream, including the replacement content outside of the overlay zone and the modified semi-transparent overlay in the overlay zone, to be presented on a display.