IP Library › Granted Patent US 10,510,190
Granted Patent B2
US 10,510,190 · App. 15/694,476 · Granted Dec 17, 2019

Mixed reality interactions

Inventors: Michael Scavezze (Bellevue, WA); Jonathan Steed (Redmond, WA); Stephen Latta (Seattle, WA); Kevin Geisner (Mercer Island, WA); Daniel McCulloch (Kirkland, WA); Brian Mount (Seattle, WA); Ryan Hastings (Seattle, WA); Phillip Charles Heckinger (Woodinville, WA)
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
G06T19/006G02B27/0093G02B27/0172G06F3/011G06K9/6267G06T19/20G02B2027/014G02B2027/0138
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,510,190
App. No.
15/694,476
Granted
Dec 17, 2019
Kind
B2
Abstract

Embodiments that relate to interacting with a physical object in a mixed reality environment via a head-mounted display are disclosed. In one embodiment a mixed reality interaction program identifies an object based on an image from captured by the display. An interaction context for the object is determined based on an aspect of the mixed reality environment. A profile for the physical object is queried to determine interaction modes for the object. A selected interaction mode is programmatically selected based on the interaction context. A user input directed at the object is received via the display and interpreted to correspond to a virtual action based on the selected interaction mode. The virtual action is executed with respect to a virtual object associated with the physical object to modify an appearance of the virtual object. The modified virtual object is then displayed via the display.

Claims (49)

1. A mixed reality interaction system comprising:

a head-mounted display device including a display system, and a camera; and

a processor configured to:

identify a physical object in a mixed reality environment based on an image captured by the camera;

determine an interaction context for the identified physical object based on one or more aspects of the mixed reality environment;

programmatically select an interaction mode from a plurality of interaction modes for the identified physical object based on the interaction context and a stored profile for the physical object, wherein the plurality of interaction modes are respectively associated with a plurality of different virtual actions;

interpret a user input directed at the physical object to correspond to a virtual action that is selected from the plurality of different virtual actions based on the selected interaction mode;

execute the virtual action to modify an appearance of a virtual object associated with the physical object; and

display the virtual object via the head-mounted display device with the modified appearance.

2. The mixed reality interaction system of claim 1 , wherein the one or more aspects of the mixed reality environment include temporal data.

3. The mixed reality interaction system of claim 1 , wherein the one or more aspects of the mixed reality environment include social networking data.

4. The mixed reality interaction system of claim 1 , wherein the one or more aspects of the mixed reality environment include a detected presence of one or more other persons.

5. The mixed reality interaction system of claim 4 , wherein the processor is further configured to:

identify the one or more other persons; and

determine the interaction context based on the identified one or more other persons.

6. The mixed reality interaction system of claim 1 , wherein the one or more aspects of the mixed reality environment include user biometric parameters.

7. The mixed reality interaction system of claim 1 , wherein the processor is further configured to:

determine a change in the interaction context;

based on the change in the interaction context, programmatically select a second selected interaction mode from the plurality of interaction modes; and

interpret the user input directed at the physical object to correspond to a different virtual action based on the second selected interaction mode.

8. The mixed reality interaction system of claim 1 , wherein the processor is further configured to:

interpret a second user input to correspond to a request to modify the selected interaction mode;

based on the second user input, change from the selected interaction mode to a second selected interaction mode;

interpret a third user input to correspond to a second virtual action based on the second selected interaction mode;

execute the second virtual action to modify an appearance of the virtual object associated with the physical object; and

display the virtual object via the head-mounted display device with the modified appearance.

9. The mixed reality interaction system of claim 1 , wherein the processor is further configured to create a new profile for the physical object that associates one or more interaction modes with the physical object.

10. The mixed reality interaction system of claim 1 , wherein the processor is further configured to:

identify an updated physical state of the physical object; and

based on the updated physical state, further modify the appearance of the virtual object associated with the physical object.

11. A method for interacting with a physical object in a mixed reality environment, comprising:

providing a head-mounted display device including a display system and a camera;

identifying the physical object based on an image captured by the camera;

determining an interaction context for the identified physical object based on one or more aspects of the mixed reality environment;

programmatically selecting an interaction mode from a plurality of interaction modes for the identified physical object based on the interaction context and a stored profile for the physical object, wherein the plurality of interaction modes are respectively associated with a plurality of different virtual actions;

interpreting a user input directed at the physical object to correspond to a virtual action that is selected from the plurality of different virtual actions based on the selected interaction mode;

executing the virtual action to modify an appearance of a virtual object associated with the physical object; and

displaying the virtual object via the head-mounted display device with the modified appearance.

12. The method of claim 11 , further comprising:

determining a change in the interaction context;

based on the change in the interaction context, programmatically selecting a second selected interaction mode from the plurality of interaction modes; and

interpreting the user input directed at the physical object to correspond to a different virtual action based on the second selected interaction mode.

13. The method of claim 11 , further comprising:

interpreting a second user input to correspond to a request to modify the selected interaction mode;

based on the second user input, changing from the selected interaction mode to a second selected interaction mode;

interpreting a third user input to correspond to a second virtual action based on the second selected interaction mode;

executing the second virtual action to modify an appearance of the virtual object associated with the physical object; and

displaying the virtual object via the head-mounted display device with the modified appearance.

14. The method of claim 11 , further comprising using a second user input to associate the physical object with the selected interaction mode.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2017
From: SCAVEZZE, MICHAEL; STEED, JONATHAN; LATTA, STEPHEN; GEISNER, KEVIN; MCCULLOCH, DANIEL; MOUNT, BRIANT; HASTINGS, RYAN; HECKINGER, PHILLIP CHARLES
To: MICROSOFT CORPORATION
Reel/Frame 043474/0532 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2017
From: MICROSOFT CORPORATION
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 043474/0570 →
Continuity (3)
Continuation 15262973 · Sep 12, 2016
Continuation 13872861 · Apr 29, 2013
Related Publication 20180012412A1 · Jan 11, 2018
Cited By (1)
US 12,602,864