IP Library Granted Patent US 9,854,328
Granted Patent B2
US 9,854,328 · App. 13/542,696 · Granted Dec 26, 2017

Augmentation of multimedia consumption

Inventors: Narayanan Venkitaraman (Palatine, IL); Shirley A. Chaysinh (Grayslake, IL); Hiren M. Mandalia (Elk Grove Village, IL); Crysta J. Metcalf (Cary, IL); Ashley B. Novak (Chicago, IL); Isaac G. Kulka (Atlanta, GA)
Assignee: ARRIS Enterprises, Inc.
H04N21/8133H04N21/4126H04N21/4307H04N21/4722H04N21/84H04N21/8547G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,854,328
App. No.
13/542,696
Granted
Dec 26, 2017
Kind
B2
Abstract

Disclosed are methods and apparatus for augmenting a user's multimedia consumption experience. The methods comprise while the user is consuming the multimedia presentation using a first device, that device provides (to one or more remote processors) information that may be used to identify a relevant location. The one or more processors use this information to identify the location and acquire a virtual environment. This virtual environment may be a virtual representation the location. The virtual environment is presented to the user on a second (companion) device. Using the second device, the user may explore the virtual environment and interact with virtual objects therein.

Claims (69)

1. A method of augmenting a user's multimedia consumption experience, the user being a user of a first user device and of a second user device, the user using the first device to consume a multimedia presentation, the method comprising:

providing, for use by one or more processors remote from the first device, information that may be used by the one or more processors to identify a location, the location being relevant to at least part of the multimedia presentation;

identifying, by the one or more processors, the location;

acquiring, by the one or more processors, a virtual environment, the virtual environment being representative of the identified location;

acquiring, by the one or more processors, one or more virtual objects, each virtual object relating to at least part of the multimedia presentation;

determining, by the one or more processors, for each virtual object, a position for that virtual object within the virtual environment;

for each virtual object, including that virtual object in the virtual environment at the position within the virtual environment determined for that virtual object; and

presenting to the user, using the second device, the virtual environment with the one or more virtual objects included therein;

wherein the virtual environment is to some extent explorable by the user using the second user device; and

wherein the virtual objects within the virtual environment are capable of being interacted with by the user.

2. A method according to claim 1 :

wherein each virtual object corresponds to an object within the multimedia presentation;

the method further comprising changing one or more properties of a virtual object of the one or more virtual objects within the virtual environment in response to a change, in the multimedia presentation, to one or more properties of the object that corresponds to that virtual object.

3. A method according to claim 1 :

wherein each virtual object corresponds to an object within the multimedia presentation;

the method further comprising, in response to the user interacting with a virtual object of the one or more virtual objects within the virtual environment, changing, in the multimedia presentation, one or more properties of the object that corresponds to that virtual object.

4. A method according to claim 1 wherein the location is a location at which events that occur in the multimedia presentation take place or that is referred to at some point in the multimedia presentation.

5. A method according to claim 4 wherein the location is a location at which events occurring in a portion of the multimedia presentation that is currently being consumed by the user take place or a location that is currently being referred to in a portion of the multimedia presentation that is currently being consumed by the user.

6. A method according to claim 1 wherein including comprises overlaying the one or more objects onto the virtual environment using an augmented-reality process.

7. A method according to claim 1 wherein the virtual environment is a three-dimensional virtual environment.

8. A method according to claim 1 :

wherein the virtual environment is an environment in which the user may interact with a further user;

wherein the further user is a user of a further device; and

wherein the virtual environment is being presented or has been presented, using the further device, to the further user.

9. A method according to claim 1 wherein the information that may be used by the one or more processors to identify the location comprises first identification data, the first identification data identifying the multimedia presentation, and second identification data, the second identification data identifying a point or range in the multimedia presentation that is currently being consumed by the user.

10. A method according to claim 1 wherein a watermark that is present in a portion of the multimedia presentation comprises information that may be used by the one or more processors to identify the location.

11. A method according to claim 1 wherein acquiring the virtual environment comprises selecting one or more images of the identified location and forming the virtual environment using the selected images.

12. A method according to claim 1 wherein the virtual environment comprises a panoramic representation of the location.

13. A method according to claim 1 wherein the multimedia presentation is selected from a group of multimedia presentations consisting of: a video presentation, a television program, and a movie.

14. A method according to claim 1 wherein the first device is the same as the second device.

15. A method according to claim 14 :

wherein the second device may be moved between having a first orientation and a second orientation;

wherein when the second device is positioned in the first orientation, the second device displays the multimedia presentation; and

wherein when the second device is positioned in the second orientation, the second device displays the virtual environment.

16. A method of augmenting a user's multimedia consumption experience, the user being a user of a first user device and of a second user device, the user using the first device to consume a multimedia presentation, the method comprising:

providing, for use by one or more processors remote from the first device, information that may be used by the one or more processors to identify a location, the location being relevant to at least part of the multimedia presentation;

identifying, by the one or more processors, the location;

acquiring, by the one or more processors, a virtual environment; and

presenting to the user, using the second device, the virtual environment;

wherein providing information that may be used by the one or more processors to identify a location comprises either:

the first device providing the information that may be used by the one or more processors to identify a location; or

the one or more processors receiving the information that may be used by the one or more processors to identify a location from one or more servers, the one or more servers storing information that may be used to identify a location;

wherein the virtual environment is representative of the identified location; and

wherein the virtual environment is to some extent explorable by the user using the second user device.

17. A method according to claim 16 further comprising presenting to the user, using the second device, one or more virtual objects, the virtual objects being presented as objects within the virtual environment, the virtual objects being capable of being interacted with by the user.

18. A method according to claim 17 :

wherein each virtual object corresponds to an object within the multimedia presentation;

the method further comprising:

changing one or more properties of a virtual object within the virtual environment in response to a change, in the multimedia presentation, to one or more properties of the object that corresponds to that virtual object; or

in response to the user interacting with a virtual object within the virtual environment, changing, in the multimedia presentation, one or more properties of the object that corresponds to that virtual object.

19. Apparatus for augmenting a user's multimedia consumption experience, the user being a user of a first user device and of a second user device, the user using the first device to consume a multimedia presentation, the apparatus comprising one or more processors arranged to:

use information to identify a location relevant to at least part of the multimedia presentation;

acquire a virtual environment, the virtual environment being representative of the identified location;

acquire one or more virtual objects, the virtual objects relating to at least part of the multimedia presentation;

determine, for each virtual object, a position for that virtual object within the virtual environment;

for each virtual object, include, in the virtual environment at the position within the virtual environment determined for that virtual object, that virtual object; and

present to the user, using the second device, the virtual environment with the one or more virtual objects included therein;

wherein the virtual environment is to some extent explorable by the user using the second user device; and

wherein the virtual objects within the virtual environment are capable of being interacted with by the user.

20. Apparatus for augmenting a user's multimedia consumption experience, the user being a user of a first user device and of a second user device, the user using the first device to consume a multimedia presentation, the apparatus comprising one or more processors configured to:

receive information for identifying a location, the location being relevant to at least part of the multimedia presentation;

identify the location;

acquire a virtual environment; and

present to the user, using the second device, the virtual environment;

wherein the information that may be used by the one or more processors to identify a location is either:

received from the first device; or

received from one or more servers, the one or more servers storing information that may be used to identify a location;

wherein the virtual environment is representative of the identified location; and

wherein the virtual environment is to some extent explorable by the user using the second user device.

Assignments (16)
RELEASE OF SECURITY INTEREST AT REEL/FRAME 049905/0504 Recorded Dec 19, 2024
From: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
To: ARRIS ENTERPRISES LLC (F/K/A ARRIS ENTERPRISES, INC.); ARRIS TECHNOLOGY, INC.; ARRIS SOLUTIONS, INC.; COMMSCOPE, INC. OF NORTH CAROLINA; COMMSCOPE TECHNOLOGIES LLC; RUCKUS WIRELESS, LLC (F/K/A RUCKUS WIRELESS, INC.)
Reel/Frame 071477/0255 →
PARTIAL RELEASE OF PATENT SECURITY INTERESTS (ABL) Recorded Jan 2, 2024
From: JPMORGAN CHASE BANK, N.A.
To: ARRIS ENTERPRISES, LLC; COMMSCOPE TECHNOLOGIES LLC
Reel/Frame 066163/0727 →
PARTIAL RELEASE OF PATENT SECURITY INTERESTS (TL) Recorded Jan 2, 2024
From: JPMORGAN CHASE BANK, N.A.
To: ARRIS ENTERPRISES, LLC; COMMSCOPE TECHNOLOGIES LLC
Reel/Frame 066163/0739 →
PARTIAL TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS Recorded Dec 29, 2023
From: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT
To: COMMSCOPE TECHNOLOGIES LLC; ARRIS ENTERPRISES LLC
Reel/Frame 066140/0541 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 23, 2022
From: ARRIS TECHNOLOGY, INC.
To: ARRIS ENTERPRISES, INC.
Reel/Frame 060791/0583 →
SECURITY INTEREST Recorded Nov 19, 2021
From: ARRIS SOLUTIONS, INC.; ARRIS ENTERPRISES LLC; COMMSCOPE TECHNOLOGIES LLC; COMMSCOPE, INC. OF NORTH CAROLINA; RUCKUS WIRELESS, INC.
To: WILMINGTON TRUST
Reel/Frame 060752/0001 →
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL: 049820 FRAME: 0495. ASSIGNOR(S) HEREBY CONFIRMS THE CHANGE OF NAME. Recorded Jul 25, 2019
From: ARRIS ENTERPRISES, INC.
To: ARRIS ENTERPRISES LLC
Reel/Frame 049858/0161 →
TERM LOAN SECURITY AGREEMENT Recorded Jul 3, 2019
From: COMMSCOPE, INC. OF NORTH CAROLINA; COMMSCOPE TECHNOLOGIES LLC; ARRIS ENTERPRISES LLC; ARRIS TECHNOLOGY, INC.; RUCKUS WIRELESS, INC.; ARRIS SOLUTIONS, INC.
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 049905/0504 →
PATENT SECURITY AGREEMENT Recorded Jul 3, 2019
From: ARRIS ENTERPRISES LLC
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT
Reel/Frame 049820/0495 →
ABL SECURITY AGREEMENT Recorded Jul 3, 2019
From: COMMSCOPE, INC. OF NORTH CAROLINA; COMMSCOPE TECHNOLOGIES LLC; ARRIS ENTERPRISES LLC; ARRIS TECHNOLOGY, INC.; RUCKUS WIRELESS, INC.; ARRIS SOLUTIONS, INC.
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 049892/0396 →
CHANGE OF NAME Recorded Jul 2, 2019
From: ARRIS ENTERPRISES. INC
To: ARRIS
Reel/Frame 049669/0652 →
TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS Recorded Apr 8, 2019
From: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
To: ARRIS GROUP, INC.; ARRIS ENTERPRISES, INC.; ARRIS SOLUTIONS, INC.; ARRIS KOREA, INC.; ARRIS HOLDINGS CORP. OF ILLINOIS, INC.; BIG BAND NETWORKS, INC.; TEXSCAN CORPORATION; POWER GUARD, INC.; 4HOME, INC.; ACADIA AIC, INC.; AEROCAST, INC.; BROADBUS TECHNOLOGIES, INC.; GENERAL INSTRUMENT CORPORATION; GENERAL INSTRUMENT AUTHORIZATION SERVICES, INC.; GENERAL INSTRUMENT INTERNATIONAL HOLDINGS, INC.; IMEDIA CORPORATION; JERROLD DC RADIO, INC.; LEAPSTONE SYSTEMS, INC.; MODULUS VIDEO, INC.; MOTOROLA WIRELINE NETWORKS, INC.; NETOPIA, INC.; NEXTLEVEL SYSTEMS (PUERTO RICO), INC.; QUANTUM BRIDGE COMMUNICATIONS, INC.; SETJAM, INC.; SUNUP DESIGN SYSTEMS, INC.; UCENTRIC SYSTEMS, INC.; GIC INTERNATIONAL HOLDCO LLC; GIC INTERNATIONAL CAPITAL LLC; CCE SOFTWARE LLC; THE GI REALTY TRUST 1996
Reel/Frame 048825/0294 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 18, 2015
From: ARRIS TECHNOLOGY, INC
To: ARRIS ENTERPRISES, INC.
Reel/Frame 037328/0341 →
MERGER AND CHANGE OF NAME Recorded Mar 10, 2015
From: GENERAL INSTRUMENT CORPORATION; GENERAL INSTRUMENT CORPORATION
To: ARRIS TECHNOLOGY, INC.
Reel/Frame 035176/0620 →
SECURITY AGREEMENT Recorded May 28, 2013
From: ARRIS GROUP, INC.; ARRIS ENTERPRISES, INC.; ARRIS SOLUTIONS, INC.; ARRIS KOREA, INC.; ARRIS HOLDINGS CORP. OF ILLINOIS; BIGBAND NETWORKS, INC.; TEXSCAN CORPORATION; POWER GUARD, INC.; 4HOME, INC.; ACADIA AIC, INC.; AEROCAST, INC.; BROADBUS TECHNOLOGIES, INC.; GENERAL INSTRUMENT CORPORATION; GENERAL INSTRUMENT AUTHORIZATION SERVICES, INC.; GENERAL INSTRUMENT INTERNATIONAL HOLDINGS, INC.; IMEDIA CORPORATION; JERROLD DC RADIO, INC.; LEAPSTONE SYSTEMS, INC.; MODULUS VIDEO, INC.; MOTOROLA WIRELINE NETWORKS, INC.; NETOPIA, INC.; NEXTLEVEL SYSTEMS (PUERTO RICO), INC.; QUANTUM BRIDGE COMMUNICATIONS, INC.; SETJAM, INC.; SUNUP DESIGN SYSTEMS, INC.; UCENTRIC SYSTEMS, INC.; GIC INTERNATIONAL HOLDCO LLC; GIC INTERNATIONAL CAPITAL LLC; CCE SOFTWARE LLC; THE GI REALTY TRUST 1996
To: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 030498/0023 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 6, 2012
From: VENKITARAMAN, NARAYANAN; CHAYSINH, SHIRLEY A.; MANDALIA, HIREN M.; METCALF, CRYSTA J.; NOVAK, ASHLEY B.; KULKA, ISAAC G.
To: GENERAL INSTRUMENT CORPORATION
Reel/Frame 028497/0277 →
Continuity (1)
Related Publication 20140009476A1 · Jan 9, 2014