IP Library Granted Patent US 11,194,439
Granted Patent B2
US 11,194,439 · App. 16/642,449 · Granted Dec 7, 2021

Methods, apparatus, systems, computer programs for enabling mediated reality

Inventors: Lasse Laaksonen (Tampere, FI); Arto Lehtiniemi (Lempaala, FI); Sujeet Shyamsundar Mate (Tampere, FI); Francesco Cricri (Tampere, FI)
Assignee: Nokia Technologies Oy
G06F3/04815G06F3/011G06T15/20H04S3/008H04S7/303H04S2400/01H04S2400/11
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,194,439
App. No.
16/642,449
Filed
Feb 27, 2020
Granted
Dec 7, 2021
Kind
B2
Art Unit
2619
USPC
345/419
Abstract

A method comprising: using a tracked real point of view of a user in a real space and a first mapping between the real space and a virtual space to determine a point of view of a virtual user within the virtual space; causing rendering to the user at least part of a virtual scene determined by the point of view of the virtual user within the virtual space; and using a selected one of a plurality of different mappings to map tracked user actions in the real space to actions of the virtual user in the virtual space, wherein, when a first mode is selected, the method comprises mapping tracked user actions in the real space, using the first mapping, to spatially-equivalent actions of the virtual user in the virtual space, and wherein, when a second mode is selected, the method comprises mapping tracked user actions in the real space, using a second mapping different to the first mapping, to non-spatially-equivalent actions of the virtual user in the virtual space, wherein the second mapping makes available user interactions within a zone of the virtual space unavailable using the first mapping.

Claims (58)

1. An apparatus comprising:

at least one processor; and

at least one non-transitory memory including computer program code,

the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least:

use a tracked real point of view of a user in a real space and a first mapping between the real space and a virtual space to determine a point of view of a virtual user within the virtual space;

cause rendering to the user at least part of a virtual scene determined with the point of view of the virtual user within the virtual space; and

use a selected one of a plurality of different mappings to map tracked user actions in the real space to actions of the virtual user in the virtual space;

wherein the apparatus is configured so that, when a first mode is selected, the apparatus maps tracked user actions in the real space, using the first mapping, to spatially-equivalent actions of the virtual user in the virtual space, and

wherein the apparatus is configured so that, when a second mode is selected, the apparatus maps tracked user actions in the real space, using a second mapping different from the first mapping, to non-spatially-equivalent actions of the virtual user in the virtual space, wherein the second mapping makes available user interactions within a zone of the virtual space, the zone of the virtual space being unavailable using the first mapping, wherein a virtual location of the virtual user remains outside the zone of the virtual space while the virtual user reaches into the zone of the virtual space that is unavailable using the first mapping;

wherein a nearest interactive object is within the zone of the virtual space that is unavailable using the first mapping and beyond a boundary defining a first virtual zone of the virtual space that is available using the first mapping and the zone of the virtual space that is unavailable using the first mapping relative to the virtual location of the virtual user.

2. The apparatus of claim 1 , wherein the second mapping provides an extended reach to the virtual user to at least the nearest interactive object in a direction of virtual reach and wherein the first mapping does not provide the extended reach to the virtual user to at least the nearest interactive object in the direction of virtual reach.

3. The apparatus of claim 1 , wherein the apparatus is configured to select the second mapping from the plurality of mappings in dependence upon actual or predicted changes in a location of the user in the real space.

4. The apparatus of claim 1 , wherein the apparatus is configured to switch from the first mode to the second mode in response to at least one of:

an extended reach with the user in the real space that extends beyond a threshold value from the user; or

a reach with the user in the real space into a designated zone.

5. The apparatus of claim 1 , wherein the apparatus is configured so that a change in the selected one of the plurality of mappings is highlighted to the user within the virtual scene.

6. The apparatus of claim 1 , wherein the real space comprises a first zone and a second, different, zone, wherein

the first zone of the real space is associated with the first mapping, wherein the apparatus is configured so that when the user is located within the first zone but does not reach into the second zone, the first mapping is used to map tracked user actions in the real space to actions of the virtual user in the first virtual zone of the virtual space; and

the second zone of the real space is associated with the second mapping, wherein the apparatus is configured so that when the user reaches into the second zone, the second mapping is used to map tracked user actions in the real space to actions of the virtual user in a second virtual zone of the virtual space.

7. The apparatus of claim 6 , wherein the first mapping and the second mapping have an equivalent mapping of changes in orientation of the user in the real space to changes of orientation of the virtual user within the virtual space,

wherein the first mapping equivalently maps changes in displacement of the user in the real space to changes in displacement of the virtual user within the virtual space, providing for reach within the virtual space equivalent to reach within the real space,

wherein the second mapping non-equivalently maps changes in displacement of the user in the real space to changes in displacement of the virtual user within the virtual space, providing for reach within the virtual space in excess of reach within the real space.

8. The apparatus of claim 1 ,

wherein the tracked real point of view of the user in the real space depends upon a tracked orientation of the user; or

wherein the tracked real point of view of the user in the real space depends upon a tracked orientation of the user and a tracked location of the user, wherein the apparatus is configured so that a change in tracked location of the user within a first zone of the real space is mapped using the first mapping to a change in virtual location of the virtual user within the virtual space.

9. The apparatus of claim 1 , wherein the apparatus is configured to use the second mapping to map tracked user actions in the real space to actions of the virtual user in the virtual space to move an object to, within or from the zone of the virtual space that is unavailable using the first mapping.

10. The apparatus of claim 1 , wherein the apparatus is further caused to cause rendering to the user of a representation of the actions of the virtual user in at least the virtual space.

11. The apparatus of claim 1 , wherein the apparatus is configured to provide user interactive-mediated reality so that user actions at least partially determine what happens within the virtual space.

12. The apparatus of claim 1 , configured as a head mounted apparatus.

13. A method comprising:

using a tracked real point of view of a user in a real space and a first mapping between the real space and a virtual space to determine a point of view of a virtual user within the virtual space;

causing rendering to the user at least part of a virtual scene determined with the point of view of the virtual user within the virtual space; and

using a selected one of a plurality of different mappings to map tracked user actions in the real space to actions of the virtual user in the virtual space;

wherein, when a first mode is selected, the method comprises mapping tracked user actions in the real space, using the first mapping, to spatially-equivalent actions of the virtual user in the virtual space, and

wherein, when a second mode is selected, the method comprises mapping tracked user actions in the real space, using a second mapping different from the first mapping, to non-spatially-equivalent actions of the virtual user in the virtual space, wherein the second mapping makes available user interactions within a zone of the virtual space, the zone of the virtual space being unavailable using the first mapping, wherein a virtual location of the virtual user remains outside the zone of the virtual space while the virtual user reaches into the zone of the virtual space that is unavailable using the first mapping;

wherein a nearest interactive object is within the zone of the virtual space that is unavailable using the first mapping and beyond a boundary defining a first virtual zone of the virtual space that is available using the first mapping and the zone of the virtual space that is unavailable using the first mapping relative to the virtual location of the virtual user.

14. The method as claimed in claim 13 , wherein the second mapping provides an extended reach to the virtual user to at least the nearest interactive object in a direction of virtual reach and wherein the first mapping does not provide the extended reach to the virtual user to at least the nearest interactive object in the direction of virtual reach.

15. The method of claim 13 , wherein the method comprises selecting the second mapping from the plurality of mappings in dependence upon actual or predicted changes in a location of the user in the real space.

16. The method of claim 13 , wherein the method comprises switching from the first mode to the second mode in response to at least one of:

an extended reach with the user in the real space that extends beyond a threshold value from the user; or

a reach with the user in the real space into a designated zone.

17. The method of claim 13 , wherein a change in the selected one of the plurality of mappings is highlighted to the user within the virtual scene.

18. The method of claim 13 , wherein the real space comprises a first zone and a second, different, zone, wherein

the first zone of the real space is associated with the first mapping, wherein when the user is located within the first zone but does not reach into the second zone, the first mapping is used to map tracked user actions in the real space to actions of the virtual user in the first virtual zone of the virtual space; and

the second zone of the real space is associated with the second mapping, wherein when the user reaches into the second zone, the second mapping is used to map tracked user actions in the real space to actions of the virtual user in a second virtual zone of the virtual space.

19. The method of claim 18 , wherein the first mapping and the second mapping have an equivalent mapping of changes in orientation of the user in the real space to changes of orientation of the virtual user within the virtual space,

wherein the first mapping equivalently maps changes in displacement of the user in the real space to changes in displacement of the virtual user within the virtual space, providing for reach within the virtual space equivalent to reach within the real space,

wherein the second mapping non-equivalently maps changes in displacement of the user in the real space to changes in displacement of the virtual user within the virtual space, providing for reach within the virtual space in excess of displacement of reach within the real space.

20. A non-transitory computer readable medium comprising program instructions stored thereon for performing at least:

use a tracked real point of view of a user in a real space and a first mapping between the real space and a virtual space to determine a point of view of a virtual user within the virtual space;

cause rendering to the user at least part of a virtual scene determined with the point of view of the virtual user within the virtual space; and

use a selected one of a plurality of different mappings to map tracked user actions in the real space to actions of the virtual user in the virtual space;

wherein, when a first mode is selected, tracked user actions in the real space are mapped, using the first mapping, to spatially-equivalent actions of the virtual user in the virtual space, and

wherein, when a second mode is selected, tracked user actions in the real space are mapped, using a second mapping different from the first mapping, to non-spatially-equivalent actions of the virtual user in the virtual space, wherein the second mapping makes available user interactions within a zone of the virtual space, the zone of the virtual space being unavailable using the first mapping, wherein a virtual location of the virtual user remains outside the zone of the virtual space while the virtual user reaches into the zone of the virtual space that is unavailable using the first mapping;

wherein a nearest interactive object is within the zone of the virtual space that is unavailable using the first mapping and beyond a boundary defining a first virtual zone of the virtual space that is available using the first mapping and the zone of the virtual space that is unavailable using the first mapping relative to the virtual location of the virtual user.

21. The apparatus of claim 5 , wherein the change in the selected one of the plurality of mappings is highlighted to the user within the virtual scene as a virtual visualization of the reaching of the virtual user into the zone of the virtual space that is unavailable using the first mapping.

22. The apparatus of claim 5 , wherein the change in the selected one of the plurality of mappings is highlighted to the user within the virtual scene as a virtual visualization of the zone of the virtual space that is unavailable using the first mapping.

23. The apparatus of claim 1 , wherein the virtual space comprises a sound space and a virtual visual space with audio-visual elements, such that a virtual point of view of the virtual user within the sound space corresponds to a virtual point of view of the virtual user within the virtual visual space, and the apparatus comprises a display to render at least parts of the virtual visual space and audio to the user in the real space.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 8, 2020
From: LAAKSONEN, LASSE; LEHTINIEMI, ARTO; MATE, SUJEET SHYAMSUNDAR; CRICRI, FRANCESCO
To: NOKIA TECHNOLOGIES OY
Reel/Frame 053150/0101 →
Priority Claims (1)
EP 17190112 · Sep 8, 2017 · regional
Continuity (1)
Related Publication 20200192550A1 · Jun 18, 2020
Cited By (8)
US 12,267,482 US 12,282,604 US 12,301,930 US 12,322,052 US 12,417,593 US 12,449,891 US 12,519,924 US 12,718,494