IP Library Granted Patent US 10,136,183
Granted Patent B2
US 10,136,183 · App. 15/381,556 · Granted Nov 20, 2018

Dynamic video overlays

Inventors: Markus K. Cremer (Orinda, CA); Peter Dunker (El Cerrito, CA); Steven D. Scherf (Fremont, CA); David Henry Pearce (Kentfield, CA); Sherman Ling Fung Li (San Francisco, CA); Wilson Harron (Berkeley, CA); Konstantinos Antonios Dimitriou (Bavaria, DE); Dewey Ho Lee (Seoul, KR)
Assignee: Gracenote, Inc.
H04N21/44016H04N21/23418H04N21/431H04N21/4312H04N21/4314H04N21/4318H04N21/4333H04N21/44008
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,136,183
App. No.
15/381,556
Granted
Nov 20, 2018
Kind
B2
Abstract

A client device accesses a video input stream from an intermediate device for display. The client device analyzes the video input stream to determine that the video input stream matches a template indicating a change of content source. The analysis is performed while the client device is generating a video output stream that replaces video content received from a content source via the intermediate device. Based on the video input stream matching the template, the client device ceases replacement of the video input stream. The client device continues to analyze the video input stream and determines that the video input stream matches a second template indicating a return to the original content source. Based on the video input stream matching the second template, the client device resumes replacement of the video input stream.

Claims (82)

1. A method for dynamically providing a video overlay, the method comprising:

accessing, by one or more processors, a video input stream that includes first video content from a first content provider;

based on the first video content of the video input stream, selecting replacement video content for a video output stream, wherein the replacement video content of the video output stream is different from the first video content of the video input stream, wherein the replacement video content is selected in response to a video identification system receiving a query fingerprint from a query fingerprint generator of a client device and the video identification system identifying a match between the query fingerprint and at least one reference fingerprint generated by a reference fingerprint generator, wherein the query fingerprint is generated based on a frame of the first video content, and wherein the video identification system and the reference fingerprint generator are remote from the client device;

causing the replacement video content of the video output stream to be presented on a display of the client device;

while the display is presenting the replacement video content of the video output stream, determining, by the one or more processors, that a first change in the video input stream has occurred, wherein the first change indicates a change from the first content provider to a second content provider of the video input stream has occurred, wherein the video input stream includes second video content from the second content provider after the first change has occurred;

in response to determining the change from the first content provider to the second content provider has occurred, causing the second video content of the video input stream to be presented on the display instead of the replacement video content;

after causing the second video content of the video input stream to be presented on the display:

accessing, from a database, a template corresponding to a screen portion of the display for third video content generated by an intermediate device;

analyzing, by the one or more processors, the video input stream to determine that at least a portion of the third video content of a frame of the video input stream matches the template when the video input stream includes the first video content from the first content provider and the third video content generated by the intermediate device, wherein the third video content is configured to be presented on the display to indicate an identity of the first content provider;

based on the at least the portion of the third video content of the frame of the video input stream matching the template, determining that a second change in the video input stream indicates a return to the first content provider of the video input stream; and

in response to the determination that the first content provider of the video input stream has been returned to:

modifying the video output stream to include the replacement video content and the third video content, and

causing a presentation of the replacement video content and the third video content of the modified video output stream on the display, wherein the third video content is presented as the video overlay on the replacement video content.

2. The method of claim 1 , wherein:

the selecting of the video output stream comprises sending to a replacement video content server a request, which includes an indication of the first video content to be replaced.

3. The method of claim 2 , wherein the sending of the request comprises:

generating, by the one or more processors, a fingerprint of the video input stream; and

comparing the fingerprint of the video input stream to a fingerprint of content to be replaced.

4. The method of claim 2 , wherein the video input stream is received via a first communication network and the replacement video content is received via a second communication network.

5. The method of claim 1 , wherein the determining that the change in the video input stream indicates the change from the first content provider to the second content provider of the video input stream comprises:

generating a sequence of fingerprints from the video input stream; and

comparing successive fingerprints of the sequence of fingerprints to successive fingerprints of content to be replaced.

6. The method of claim 1 , wherein the first content provider is a cable television station.

7. The method of claim 1 , wherein the analyzing of the video input stream to determine that at least the portion of the third video content of the frame of the video input stream matches the template comprises:

downsampling the frame of the video input stream;

determining a cross-correlation between the downsampled frame and the template; and

comparing the cross-correlation to a threshold.

8. The method of claim 1 , wherein the intermediate device is a set-top box,

wherein accessing the video input stream comprises receiving, by one or more processors, the video input stream from the set-top box, and

wherein the third video content is configured to be presented on the display to indicate the identity of the first content provider responsive to a change in the video input stream from the second content provider to the first content provider.

9. The method of claim 1 , wherein when the video input stream includes the first video content from the first content provider and the third video content generated by the intermediate device, the first video content is configured to be presented in a first screen portion of the display and the third video content is configured to be presented in a second screen portion of the display, and

wherein modifying the video output stream to include the replacement video content and the third video content comprising modifying the video output stream to include the replacement video content in first screen portion of the display and the third video content in the second screen portion of the display.

10. A system for dynamically providing a video overlay, the system comprising:

a memory that stores instructions; and

one or more processors configured by the instructions to perform operations comprising:

accessing a video input stream that includes first video content from a first content provider;

based on the first video content of the video input stream, selecting replacement video content for a video output stream, wherein the replacement video content of the video output stream is different from the first video content of the video input stream, wherein the replacement video content is selected in response to a video identification system receiving a query fingerprint from a query fingerprint generator of a client device and the video identification system identifying a match between the query fingerprint and at least one reference fingerprint generated by a reference fingerprint generator, wherein the query fingerprint is generated based on a frame of the first video content, and wherein the video identification system and the reference fingerprint generator are remote from the client device;

causing the replacement video content of the video output stream to be presented on a display of the client device;

while the display is presenting the replacement video content of the video output stream, determining, by the one or more processors, that a first change in the video input stream has occurred, wherein the first change indicates a change from the first content provider to a second content provider of the video input stream has occurred, wherein the video input stream includes second video content from the second content provider after the first change has occurred;

in response to determining the change from the first content provider to the second content provider has occurred, causing the second video content of the video input stream to be presented on the display instead of the replacement video content;

after causing the second video content of the video input stream to be presented on the display:

accessing, from a database, a template corresponding to a screen portion of the display for third video content generated by an intermediate device;

analyzing the video input stream to determine that at least a portion of the third video content of a frame of the video input stream matches the template when the video input stream includes the first video content from the first content provider and the third video content generated by the intermediate device, wherein the third video content is configured to be presented on the display to indicate an identity of the first content provider;

based on the at least the portion of the third video content of the frame of the video input stream matching the template, determining that a second change in the video input stream indicates a return to the first content provider of the video input stream; and

in response to the determination that the first content provider of the video input stream has been returned to:

modifying the video output stream to include the replacement video content and the third video content, and

causing a presentation of the replacement video content and the third video content of the modified video output stream on the display, wherein the third video content is presented as the video overlay on the replacement video content.

11. The system of claim 10 , wherein:

the selecting of the video output stream comprises sending to a replacement content server a request, which includes an indication of the first video content to be replaced.

12. The system of claim 11 , wherein the sending of the request comprises:

generating a fingerprint of the video input stream; and

comparing the fingerprint of the video input stream to a fingerprint of content to be replaced.

13. The system of claim 11 , wherein the video input stream is received via a first communication network and the replacement video content is received via a second communication network.

14. The system of claim 10 , wherein the determining that the change in the video input stream indicates the change from the first content provider to the second content provider of the video input stream comprises:

generating a sequence of fingerprints from the video input stream; and

comparing successive fingerprints of the sequence of fingerprints to successive fingerprints of content to be replaced.

15. The system of claim 10 , wherein the first content provider is a cable television station.

16. The system of claim 10 , wherein the analyzing of the video input stream to determine that at least the portion of the third video content of the frame of the video input stream matches the template comprises:

downsampling the frame of the video input stream;

determining a cross-correlation between the downsampled frame and the template; and

comparing the cross-correlation to a threshold.

17. A non-transitory machine-readable medium that stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

accessing a video input stream that includes first video content from a first content provider;

based on the first video content of the video input stream, selecting replacement video content for a video output stream, wherein the replacement video content of the video output stream is different from the first video content of the video input stream, wherein the replacement video content is selected in response to a video identification system receiving a query fingerprint from a query fingerprint generator of a client device and the video identification system identifying a match between the query fingerprint and at least one reference fingerprint generated by a reference fingerprint generator, wherein the query fingerprint is generated based on a frame of the first video content, and wherein the video identification system and the reference fingerprint generator are remote from the client device;

causing the replacement video content of the video output stream to be presented on a display of the client device;

while the display is presenting the replacement video content of the video output stream, determining, by the one or more processors, that a first change in the video input stream has occurred, wherein the first change indicates a change from the first content provider to a second content provider of the video input stream has occurred, wherein the video input stream includes second video content from the second content provider after the first change has occurred;

in response to determining the first change from the first content provider to the second content provider has occurred, causing the second video content of the video input stream to be presented on the display instead of the replacement video content;

after causing the second video content of the video input stream to be presented on the display:

accessing, from a database, a template corresponding to a screen portion of the display for third video content generated by an intermediate device;

analyzing the video input stream to determine that at least a portion of the third video content of a frame of the video input stream matches the template when the video input stream includes the first video content from the first content provider and the third video content generated by the intermediate device, wherein the third video content is configured to be presented on the display to indicate an identity of the first content provider;

based on the at least the portion of the third video content of the frame of the video input stream matching the template, determining that a second change in the video input stream indicates a return to the first content provider of the video input stream; and

in response to the determination that the first content provider of the video input stream has been returned to:

modifying the video output stream to include the replacement video content and the third video content, and

causing a presentation of the replacement video content and the third video content of the modified video output stream on the display, wherein the third video content is presented as a video overlay on the replacement video content.

18. The non-transitory machine-readable medium of claim 17 , wherein:

the selecting of the video output stream comprises sending to a replacement content server a request, which includes an indication of the first video content to be replaced.

19. The non-transitory machine-readable medium of claim 18 , wherein the sending of the request comprises:

generating a fingerprint of the video input stream; and

comparing the fingerprint of the video input stream to a fingerprint of content to be replaced.

20. The non-transitory machine-readable medium of claim 17 , wherein the determining that the change in the video input stream indicates the change from the first content provider to the second content provider of the video input stream comprises:

generating a sequence of fingerprints from the video input stream; and

comparing successive fingerprints of the sequence of fingerprints to successive fingerprints of content to be replaced.

Assignments (12)
SECURITY INTEREST Recorded Sep 18, 2024
From: ROKU, INC.
To: CITIBANK, N.A.
Reel/Frame 068982/0377 →
RELEASE (REEL 054066 / FRAME 0064) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063605/0001 →
RELEASE (REEL 053473 / FRAME 0001) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063603/0001 →
TERMINATION AND RELEASE OF INTELLECTUAL PROPERTY SECURITY AGREEMENT (REEL/FRAME 056982/0194) Recorded Feb 22, 2023
From: MORGAN STANLEY SENIOR FUNDING, INC.
To: ROKU, INC.; ROKU DX HOLDINGS, INC.
Reel/Frame 062826/0664 →
RELEASE (REEL 042262 / FRAME 0601) Recorded Oct 13, 2022
From: CITIBANK, N.A.
To: GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC
Reel/Frame 061748/0001 →
PATENT SECURITY AGREEMENT SUPPLEMENT Recorded Jun 29, 2021
From: ROKU, INC.
To: MORGAN STANLEY SENIOR FUNDING, INC.
Reel/Frame 056982/0194 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 30, 2021
From: GRACENOTE, INC.
To: ROKU, INC.
Reel/Frame 056103/0786 →
PARTIAL RELEASE OF SECURITY INTEREST Recorded Apr 20, 2021
From: CITIBANK, N.A.
To: THE NIELSEN COMPANY (US), LLC; GRACENOTE, INC.
Reel/Frame 056973/0280 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 21, 2020
From: CREMER, MARKUS K.; DUNKER, PETER; SCHERF, STEVEN D.; PEARCE, DAVID HENRY; LI, SHERMAN LING FUNG; HARRON, WILSON; DIMITRIOU, KONSTANTINOS ANTONIOS; LEE, DEWEY HO
To: GRACENOTE, INC.
Reel/Frame 054124/0540 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT. Recorded Oct 7, 2020
From: A.C. NIELSEN (ARGENTINA) S.A.; A.C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A
Reel/Frame 054066/0064 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Jun 9, 2020
From: A. C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NIELSEN UK FINANCE I, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A.
Reel/Frame 053473/0001 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Apr 13, 2017
From: GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE DIGITAL VENTURES, LLC
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 042262/0601 →
Continuity (2)
Provisional Application 62268410 · Dec 16, 2015
Related Publication 20170180794A1 · Jun 22, 2017