IP Library Granted Patent US 10,055,010
Granted Patent B2
US 10,055,010 · App. 15/003,658 · Granted Aug 21, 2018

Detecting an event within interactive media including spatialized multi-channel audio content

Inventors: Jeff Benson (San Francisco, CA); Michael Gubman (San Francisco, CA); Craig Kawahara (San Jose, CA); Robert Coover (Orinda, CA); Markus K. Cremer (Orinda, CA); Andy Mai (Oakland, CA)
Assignee: Gracenote, Inc.
G06F3/01A63F13/69A63F13/73A63F13/75G06F3/0484G06F3/04842
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,055,010
App. No.
15/003,658
Granted
Aug 21, 2018
Kind
B2
Abstract

As a user is being presented with interactive media by a presenting device, a separate monitoring device may be used to monitor the presentation of the interactive media and detect an event that occurs therein. Such a monitoring device may be configured and positioned to access media content from the presentation of the interactive media. For example, the monitoring device may be configured and positioned to record video content with a camera and record audio content with a microphone. Having accessed this media content, the monitoring device may generate an identifier, such as a fingerprint or watermark, of the media content and compare the generated identifier with a reference identifier that is generated from the source of the media content. Based on the generated identifier matching the reference identifier, the monitoring device may detect that an event has occurred within the interactive media presentation and present a corresponding notification.

Claims (82)

1. A method comprising:

generating, by one or more processors of a machine, a query fingerprint from a playback of spatialized media content that is spatialized in three dimensions and that indicates a virtual orientation at a virtual location in a virtual world;

accessing, by one or more processors of the machine, a database that stores reference fingerprints, a reference fingerprint among the reference fingerprints being generated from the spatialized media content, the reference fingerprint identifying information that corresponds to the virtual orientation at the virtual location in the virtual world;

querying, by one or more processors of the machine, the database with the query fingerprint to determine whether the query fingerprint matches any of the reference fingerprints;

selecting, by one or more processors of the machine, the information identified by the reference fingerprint based on the reference fingerprint matching the query fingerprint; and

causing, by one or more processors of the machine, presentation of the information that corresponds to the virtual orientation at the virtual location within the virtual world in response to the playback of the spatialized media content.

2. The method of claim 1 , wherein:

the spatialized media content includes spatialized multi-channel audio content that indicates the virtual orientation at the virtual location in the virtual world.

3. The method of claim 1 , further comprising:

monitoring an interactive media presentation being presented by a device, the interactive media presentation including the playback of the spatialized media content that indicates the virtual orientation at the virtual location within the virtual world; and wherein

the generating of the query fingerprint from the playback of the spatialized media content occurs during the monitoring of the interactive media presentation.

4. The method of claim 1 , wherein:

the virtual world is included in an interactive media presentation that enables navigation within the virtual world in response to user input; and

the generating of the query fingerprint from the playback of the spatialized media content is in response to the playback of the spatialized media content occurring during the navigation within the virtual world in response to the user input.

5. The method of claim 1 , wherein:

the virtual world is included in an interactive media presentation that enables an avatar of a user to selectively move among multiple virtual locations in the virtual world and selectively orient toward any of multiple virtual orientations within the virtual world; and

the generating of the query fingerprint from the playback of the spatialized media content is in response to the playback of the spatialized media content being triggered by the avatar moving to the virtual location and orienting toward the virtual orientation within the virtual world.

6. The method of claim 1 , wherein:

the virtual location is a first virtual location in the virtual world;

the spatialized media content indicates the first virtual location and a second virtual location in the virtual world;

the query fingerprint is a first query fingerprint;

the reference fingerprint is a first reference fingerprint; and

the selecting of the information is based on a second reference fingerprint matching a second query fingerprint in conjunction with the first reference fingerprint matching the first query fingerprint.

7. The method of claim 6 , wherein:

the spatialized media content is first media content that indicates the first and second virtual locations in the virtual world;

the second reference fingerprint is generated from second media content that indicates the first virtual location but not the second virtual location in the virtual world; and

the method further comprises

generating the second query fingerprint from a playback of the second media content that indicates the first virtual location but not the second virtual location.

8. The method of claim 6 , wherein:

the spatialized media content is first media content that indicates the first and second virtual locations in the virtual world;

the second reference fingerprint is generated from second media content that indicates at least the first virtual location in the virtual world; and

the method further comprises

generating the second query fingerprint from a playback of the second media content that indicates at least the first virtual location, wherein

the selecting of the information is based on a probability that the playback of the first media content is contemporaneous with the playback of the second media content.

9. The method of claim 6 , wherein:

the virtual world is included in an interactive media presentation that presents events within the virtual world in response to user input;

the spatialized media content is first media content that indicates the first and second virtual locations in the virtual world;

the second reference fingerprint is generated from second media content that indicates at least the first virtual location in the virtual world; and

the method further comprises

generating the second query fingerprint from a playback of the second media content that indicates at least the first virtual location, wherein

the selecting of the information is based on a history of events presented within the virtual world, the history including the playback of the first media content and the playback of the second media content.

10. The method of claim 1 , wherein:

the virtual world is included in an interactive media presentation;

the generating of the query fingerprint is in response to a request that a current portion of the interactive media presentation be identified; and

the causing of the presentation of the information includes causing presentation of an identifier of the current portion of the interactive media presentation.

11. The method of claim 1 , wherein:

the spatialized media content further indicates acquisition of a virtual item within the virtual world; and

the causing of the presentation of the information includes causing presentation of a description of the virtual item.

12. The method of claim 1 , wherein:

the spatialized media content further indicates a level of progress within a storyline of an interactive media presentation; and

the selecting of the information is based on the level of progress within the storyline of the interactive media presentation.

13. The method of claim 1 , wherein:

the spatialized media content further indicates an achievement within a game being played by a user; and

the causing of the presentation of the information includes causing presentation of a notification that corresponds to the achievement within the game.

14. The method of claim 1 , wherein:

the spatialized media content further indicates a failure to achieve a goal within a game being played by a user; and

the causing of the presentation of the information includes causing presentation of a notification that corresponds to the failure to achieve a goal within the game.

15. The method of claim 1 , wherein:

the virtual world is included in an interactive media presentation that presents events within the virtual world in response to user input submitted by a user; and

the causing of the presentation of the information includes communicating the information to a device of the user.

16. The method of claim 1 , wherein:

the virtual world is included in an interactive media presentation that presents events within the virtual world in response to user input submitted by a first user; and

the causing of the presentation of the information includes communicating the information to a device of a second user that is socially connected to the first user according to a social networking service.

17. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

generating a query fingerprint from a playback of spatialized media content that is spatialized in three dimensions and that indicates a virtual orientation at a virtual location in a virtual world;

accessing a database that stores reference fingerprints, a reference fingerprint among the reference fingerprints being generated from the spatialized media content, the reference fingerprint identifying information that corresponds to the virtual orientation at the virtual location in the virtual world;

querying the database with the query fingerprint to determine whether the query fingerprint matches any of the reference fingerprints;

selecting the information identified by the reference fingerprint based on the reference fingerprint matching the query fingerprint; and

causing presentation of the information that corresponds to the virtual orientation at the virtual location within the virtual world in response to the playback of the spatialized media content.

18. The non-transitory machine-readable storage medium of claim 17 , wherein the operations further comprise:

monitoring an interactive media presentation being presented by a device, the interactive media presentation including the playback of the spatialized media content that indicates the virtual orientation at the virtual location within the virtual world; and wherein

the generating of the query fingerprint from the playback of the spatialized media content occurs during the monitoring of the interactive media presentation.

19. A system comprising:

one or more processors; and

a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the system to perform operations comprising:

generating a query fingerprint from a playback of spatialized media content that is spatialized in three dimensions and that indicates a virtual orientation at a virtual location in a virtual world;

accessing a database that stores reference fingerprints, a reference fingerprint among the reference fingerprints being generated from the spatialized media content, the reference fingerprint identifying information that corresponds to the virtual orientation at the virtual location in the virtual world;

querying the database with the query fingerprint to determine whether the query fingerprint matches any of the reference fingerprints;

selecting the information identified by the reference fingerprint based on the reference fingerprint matching the query fingerprint; and

causing presentation of the information that corresponds to the virtual orientation at the virtual location within the virtual world in response to the playback of the spatialized media content.

20. The system of claim 19 , wherein:

the spatialized media content includes spatialized multi-channel audio content that indicates the virtual orientation at the virtual location in the virtual world.

Assignments (14)
SECURITY INTEREST Recorded Sep 18, 2024
From: ROKU, INC.
To: CITIBANK, N.A.
Reel/Frame 068982/0377 →
RELEASE (REEL 054066 / FRAME 0064) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063605/0001 →
RELEASE (REEL 053473 / FRAME 0001) Recorded May 11, 2023
From: CITIBANK, N.A.
To: A. C. NIELSEN COMPANY, LLC; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; THE NIELSEN COMPANY (US), LLC; NETRATINGS, LLC
Reel/Frame 063603/0001 →
TERMINATION AND RELEASE OF INTELLECTUAL PROPERTY SECURITY AGREEMENT (REEL/FRAME 056982/0194) Recorded Feb 22, 2023
From: MORGAN STANLEY SENIOR FUNDING, INC.
To: ROKU, INC.; ROKU DX HOLDINGS, INC.
Reel/Frame 062826/0664 →
RELEASE (REEL 042262 / FRAME 0601) Recorded Oct 13, 2022
From: CITIBANK, N.A.
To: GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC
Reel/Frame 061748/0001 →
PATENT SECURITY AGREEMENT SUPPLEMENT Recorded Jun 29, 2021
From: ROKU, INC.
To: MORGAN STANLEY SENIOR FUNDING, INC.
Reel/Frame 056982/0194 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 30, 2021
From: GRACENOTE, INC.
To: ROKU, INC.
Reel/Frame 056103/0786 →
PARTIAL RELEASE OF SECURITY INTEREST Recorded Apr 20, 2021
From: CITIBANK, N.A.
To: THE NIELSEN COMPANY (US), LLC; GRACENOTE, INC.
Reel/Frame 056973/0280 →
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT. Recorded Oct 7, 2020
From: A.C. NIELSEN (ARGENTINA) S.A.; A.C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A
Reel/Frame 054066/0064 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Jun 9, 2020
From: A. C. NIELSEN COMPANY, LLC; ACN HOLDINGS INC.; ACNIELSEN CORPORATION; ACNIELSEN ERATINGS.COM; AFFINNOVA, INC.; ART HOLDING, L.L.C.; ATHENIAN LEASING CORPORATION; CZT/ACN TRADEMARKS, L.L.C.; EXELATE, INC.; GRACENOTE, INC.; GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; NETRATINGS, LLC; NIELSEN AUDIO, INC.; NIELSEN CONSUMER INSIGHTS, INC.; NIELSEN CONSUMER NEUROSCIENCE, INC.; NIELSEN FINANCE CO.; NIELSEN FINANCE LLC; NIELSEN INTERNATIONAL HOLDINGS, INC.; NIELSEN MOBILE, LLC; NIELSEN UK FINANCE I, LLC; NMR INVESTING I, INC.; TCG DIVESTITURE INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC; VIZU CORPORATION; VNU MARKETING INFORMATION, INC.; NMR LICENSING ASSOCIATES, L.P.; NIELSEN HOLDING AND FINANCE B.V.; THE NIELSEN COMPANY B.V.; VNU INTERNATIONAL B.V.
To: CITIBANK, N.A.
Reel/Frame 053473/0001 →
SUPPLEMENTAL SECURITY AGREEMENT Recorded Apr 13, 2017
From: GRACENOTE, INC.; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE DIGITAL VENTURES, LLC
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 042262/0601 →
RELEASE OF SECURITY INTEREST IN PATENT RIGHTS Recorded Feb 8, 2017
From: JPMORGAN CHASE BANK, N.A.
To: GRACENOTE, INC.; CASTTV INC.; TRIBUNE MEDIA SERVICES, LLC; TRIBUNE DIGITAL VENTURES, LLC
Reel/Frame 041656/0804 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 3, 2016
From: BENSON, JEFF; GUBMAN, MICHAEL; KAWAHARA, CRAIG; COOVER, ROBERT; CREMER, MARKUS K.; MAI, ANDY
To: GRACENOTE, INC.
Reel/Frame 037884/0791 →
SECURITY AGREEMENT Recorded Mar 1, 2016
From: GRACENOTE, INC.; TRIBUNE BROADCASTING COMPANY, LLC
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 037960/0805 →
Continuity (2)
Continuation 13795877 · Mar 12, 2013
Related Publication 20160139756A1 · May 19, 2016