IP Library Granted Patent US 11,233,840
Granted Patent B2
US 11,233,840 · App. 15/929,210 · Granted Jan 25, 2022

Use of in-band metadata as basis to access reference fingerprints to facilitate content-related action

Inventors: Steven M. Cormie (Cambridge, GB); Brian Jentz (Buxton, GB)
Assignee: Roku, Inc.
H04L65/604H04L65/4069H04L67/42
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,233,840
App. No.
15/929,210
Granted
Jan 25, 2022
Kind
B2
Abstract

As a media client receives a media stream for presentation, the media client could receive, in-band with the media stream, metadata identifying the media stream, such as a media-stream identifier encoded as a watermark in the media stream or carried together with the media stream in a packet-based transport stream, among other possibilities. And while the media client is processing the media stream for presentation, the media client could then convey the received metadata out-of-band to a server to enable the server to readily obtain reference fingerprints representing the media stream and to thereby facilitate, based on the reference fingerprints, a content-related action such as dynamic content modification for instance.

Claims (43)

1. A method comprising:

receiving by a media client a media stream, and receiving by the media client, as in-band metadata with the media stream, an identifier of the media stream;

processing by the media client the media stream for presentation; and

while processing by the media client the media stream for presentation, transmitting by the media client to a server, in out-of-band signaling, the identifier of the media stream received by the media client as in-band metadata with the media stream, to enable the server to obtain, based on a lookup keyed to the identifier, reference fingerprints representing the media stream without a need for the server to search for reference fingerprints that match query fingerprints representing the media stream being processed by the media client, wherein obtaining the reference fingerprints based on the lookup keyed to the identifier comprises referring to predefined reference data that associates various identifiers with respective reference fingerprints, to determine, based on the reference data, the reference fingerprints with which the identifier is associated.

2. The method of claim 1 , further comprising:

receiving by the media client from the server a directive for the media client to perform a content-related action at a particular time in the media stream, wherein the particular time is established based on the obtained reference fingerprints.

3. The method of claim 2 , wherein the content-related action comprises dynamic ad insertion.

4. The method of claim 1 , wherein receiving by the media client the identifier as in-band metadata with the media stream comprises receiving by the media client the identifier steganographically encoded as a watermark in the media stream.

5. The method of claim 1 , wherein receiving by the media client the identifier as in-band metadata with the media stream comprises receiving by the media client the identifier carried in a transport stream that carries the media stream to the media client.

6. The method of claim 1 , wherein the out-of-band signaling comprises signaling over a broadband network connection.

7. The method of claim 1 , wherein the media client comprises a content presentation device, and wherein the processing of the media stream for presentation comprises the media client presenting content of the media stream on a user interface.

8. The method of claim 1 , wherein processing of the media stream for presentation comprises outputting the media stream for presentation by an associated device.

9. The method of claim 1 , wherein enabling the server to obtain the reference fingerprints representing the media stream enables establishment of synchronous lock that maps reference time to client time, wherein establishing of the synchronous lock enables translating a reference time of a content-related action to a client time of the content-related action.

10. The method of claim 1 , wherein enabling the server to obtain the reference fingerprints representing the media stream enables the server to perform fingerprint comparison between a replaceable content segment and the obtained reference fingerprints, to detect presence of the replaceable content segment in the media stream.

11. The method of claim 1 , wherein the media stream comprises a television channel.

12. A media client comprising:

a content communication interface;

a network communication interface;

a processing unit;

non-transitory data storage; and

program instructions stored in the non-transitory data storage and executable by the processing unit to carry out operations including:

receiving, through the content communication interface, a media stream,

receiving, through the content communication interface, as in-band metadata with the media stream, an identifier of the media stream,

processing the media stream for presentation,

while processing the media stream for presentation, transmitting to a server, in out-of-band signaling through the network communication interface, the identifier of the media stream received as in-band metadata with the media stream, to enable the server to obtain, based on a lookup keyed to the identifier, reference fingerprints representing the media stream without a need for the server to search for reference fingerprints that match query fingerprints representing the media stream being processed for presentation, wherein obtaining the reference fingerprints based on the lookup keyed to the identifier comprises referring to predefined reference data that associates various identifiers with respective reference fingerprints, to determine, based on the reference data, the reference fingerprints with which the identifier is associated, and

receiving from the server, through the network communication interface, a directive for the media client to perform a content-related action at a particular time in the media stream, wherein the particular time is established based on the obtained reference fingerprints.

13. The media client of claim 12 , wherein the content-related action comprises dynamic ad insertion.

14. The media client of claim 12 , wherein receiving the identifier as in-band metadata with the media stream comprises receiving the identifier steganographically encoded as a watermark in the media stream.

15. The media client of claim 12 , wherein receiving by the media client the identifier as in-band metadata with the media stream comprises receiving by the media client the identifier carried in a transport stream that carries the media stream to the media client.

16. The media client of claim 12 , wherein enabling the server to obtain the reference fingerprints representing the media stream enables establishment of synchronous lock that maps reference time to client time, wherein establishing of the synchronous lock enables translating a reference time of the content-related action to a client time of the content related action.

17. The media client of claim 12 , wherein enabling the server to obtain the reference fingerprints representing the media stream enables the server to perform fingerprint comparison between a replaceable content segment and the obtained reference fingerprints, to detect presence of the replaceable content segment in the media stream.

18. A method comprising:

while a media client is processing a media stream for presentation, receiving by a server from the media client an out-of-band signal providing the server with an identifier of the media stream that the media client extracted from in-band metadata received by the media client with the media stream;

using by the server the received identifier of the media stream as a basis to obtain, based on a lookup keyed to the identifier, reference fingerprints representing the media stream, without a need for the server to search for reference fingerprints that match query fingerprints representing the media stream being processed by the media client, wherein using the received identifier as a basis to obtain the reference fingerprints based on the lookup keyed to the identifier comprises referring to predefined reference data that associates various identifiers with respective reference fingerprints, to determine, based on the reference data, the reference fingerprints with which the identifier is associated;

using by the server the obtained reference fingerprints as a basis to determine a point in the media stream when the media client should perform a content-related action; and

directing by the server the media client to perform at the determined point the content-related action.

19. The method of claim 18 , wherein the content-related action comprises dynamic ad insertion.

20. The method of claim 18 , wherein the identifier of the media stream that the media client extracted from in-band metadata received by the media client with the media stream comprises an identifier that the media client extracted from a watermark in the media stream.

21. Non-transitory data storage embodying program instructions executable by one or more processors to carry out operations including:

while a media client is processing a media stream for presentation, receiving from the media client an out-of-band signal providing an identifier of the media stream that the media client extracted from in-band metadata received by the media client with the media stream;

using the received identifier of the media stream as a basis to obtain, based on a lookup keyed to the identifier, reference fingerprints representing the media stream, without a need to search for reference fingerprints that match query fingerprints representing the media stream being processed by the media client, wherein using the received identifier as a basis to obtain the reference fingerprints based on the lookup keyed to the identifier comprises referring to predefined reference data that associates various identifiers with respective reference fingerprints, to determine, based on the reference data, the reference fingerprints with which the identifier is associated;

using the obtained reference fingerprints as a basis to determine a point in the media stream when the media client should perform a content-related action; and

directing the media client to perform, at the determined point, the content-related action.

Assignments (10)
SECURITY INTEREST Recorded Sep 18, 2024
From: ROKU, INC.
To: CITIBANK, N.A.
Reel/Frame 068982/0377 →
RELEASE (REEL 053473 / FRAME 0001) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063603/0001 →
RELEASE (REEL 054066 / FRAME 0064) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063605/0001 →
TERMINATION AND RELEASE OF INTELLECTUAL PROPERTY SECURITY AGREEMENT (REEL/FRAME 056982/0194) Recorded Feb 22, 2023
From: MORGAN STANLEY SENIOR FUNDING, INC.
To: ROKU, INC.; ROKU DX HOLDINGS, INC.
Reel/Frame 062826/0664 →
PATENT SECURITY AGREEMENT SUPPLEMENT Recorded Jun 29, 2021
From: ROKU, INC.
To: MORGAN STANLEY SENIOR FUNDING, INC.
Reel/Frame 056982/0194 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 30, 2021
From: THE NIELSEN COMPANY (US), LLC
To: ROKU, INC.
Reel/Frame 056106/0376 →
PARTIAL RELEASE OF SECURITY INTEREST Recorded Apr 20, 2021
From: CITIBANK, N.A.
To: THE NIELSEN COMPANY (US), LLC; GRACENOTE, INC.
Reel/Frame 056973/0280 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT. Recorded Oct 7, 2020
From: A.C. NIELSEN (ARGENTINA) S.A.; A.C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A
Reel/Frame 054066/0064 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Jun 9, 2020
From: A. C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NIELSEN UK FINANCE I, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A.
Reel/Frame 053473/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 29, 2020
From: CORMIE, STEVEN M.; JENTZ, BRIAN
To: THE NIELSEN COMPANY (US), LLC
Reel/Frame 051735/0307 →