IP Library Granted Patent US 9,824,499
Granted Patent B2
US 9,824,499 · App. 14/747,997 · Granted Nov 21, 2017

Mixed-reality image capture

Inventors: Shawn Crispin Wright (Sammamish, WA); Dan Osborn (Woodinville, WA); Alberto E. Cerriteno (Bothell, WA); Bofan Hsu (Bothell, WA); Evan Jones (Kirkland, WA); Joe Thompson (Seattle, WA); Forest Woodcroft Gouin (Seattle, WA); Megan Saunders (Kirkland, WA)
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
G06T19/006G02B27/0093G02B27/0172G02B2027/0174G02B2027/0198
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,824,499
App. No.
14/747,997
Filed
Jun 23, 2015
Granted
Nov 21, 2017
Kind
B2
Art Unit
2674
USPC
345/633
Abstract

A head-mounted display includes a visible-light camera configured to collect a visible-light image of a physical space, a surface sensor configured to measure one or more surface parameters of the physical space, a see-through display configured to visually present an augmentation image while light from the physical space passes through the see-through display to a user eye, and an augmented-reality engine. The augmented-reality engine may be configured to identify a surface of the physical space from the one or more measured surface parameters, compose a mixed-reality image that includes the augmentation image overlaid on the visible-light image, and visually present, via the see-through display, the mixed-reality image in alignment with the identified surface.

Claims (48)

1. A head-mounted display device comprising:

a visible-light camera configured to collect a visible-light image of a physical space;

a surface sensor configured to measure one or more surface parameters of the physical space;

a see-through display configured to visually present an augmentation image while light from the physical space passes through the see-through display to a user eye; and

an augmented-reality engine configured to:

identify a surface of the physical space from the one or more measured surface parameters,

determine an angle of the identified surface relative to a perspective of the see-through display;

compose a mixed-reality image that includes the augmentation image overlaid on the visible-light image, and

visually present, via the see-through display, the mixed-reality image in alignment with the identified surface at a non-parallel angle relative to the identified surface if the angle is less than a threshold angle.

2. The head-mounted display device of claim 1 , further comprising:

a gaze sensor configured to measure one or more gaze parameters; and

wherein the augmented-reality engine is configured to recognize a gaze vector from the one or more gaze parameters, and wherein the identified surface is a surface nearest the gaze vector.

3. The head-mounted display device of claim 1 , wherein the surface sensor is a depth camera, and wherein the one or more surface parameters include a relative surface depth measured by the depth camera.

4. The head-mounted display device of claim 1 , wherein the mixed-reality image is aligned at an angle that is parallel with the identified surface.

5. The head-mounted display device of claim 1 , wherein the mixed-reality image is automatically visually presented via the see-through display responsive to a user request during use of the head-mounted display device.

6. The head-mounted display device of claim 1 , wherein the mixed-reality image is automatically visually presented via the see-through display responsive to the visible-light camera collecting the visible light image.

7. The head-mounted display device of claim 1 , wherein the see-through display is configured to visually present an animation in which the mixed-reality image appears to move away from the user eye into alignment with the identified surface of the physical space.

8. The head-mounted display device of claim 1 , wherein the augmented-reality engine is configured to visually present, via the see-through display, a virtual adjustment control for adjusting a size or position of the mixed-reality image in the physical space.

9. A method for controlling an augmented-reality device, comprising:

collecting, with a visible-light camera of the augmented-reality device, a visible-light image of a physical space;

measuring, with a surface sensor of the augmented-reality device, one or more surface parameters of the physical space;

identifying a surface of the physical space from the one or more measured surface parameters;

determining an angle of the identified surface relative to a perspective of a user of the augmented-reality device; and

visually presenting, via a display of the augmented-reality device, the visible-light image in alignment with the identified surface at a non-parallel angle relative to the identified surface if the angle is less than a threshold angle from a perspective of a user of the augmented-reality device.

10. The method of claim 9 , further comprising

visually presenting, via the display of the augmented-reality device, one or more virtual objects overlaid on the physical space;

composing a mixed-reality image that includes the one or more virtual objects overlaid on the visible-light image of the physical space; and

visually presenting, via the display of the augmented-reality device, the mixed-reality image in alignment with the identified surface from the perspective of the user of the augmented-reality device.

11. The method of claim 9 , further comprising:

measuring, with a gaze sensor of the augmented-reality device, one or more gaze parameters of a user eye; and

recognizing a gaze vector of the user eye from the one or more gaze parameters, and wherein the identified surface is a surface nearest the gaze vector.

12. The method of claim 9 , wherein the visible-light image is aligned at an angle that is parallel with the identified surface.

13. The method of claim 9 , wherein the visible-light image is automatically visually presented via the display responsive to the visible-light camera collecting the visible light image.

14. The method of claim 9 , further comprising:

visually presenting, via the display of the augmented-reality device, an animation in which the visible-light image appears to move away from a perspective of the user of the augmented-reality device into alignment with the identified surface of the physical space.

15. A method for controlling an augmented-reality device, comprising:

measuring, with a surface sensor of the augmented-reality device, one or more surface parameters of a physical space;

identifying one or more surfaces of the physical space from the one or more measured surface parameters;

visually presenting, via a see-through display of the augmented-reality device, an augmentation image while light from the physical space passes through the see-through display to a user eye;

collecting, with a visible-light camera of the augmented-reality device, a visible-light image of the physical space;

composing a mixed-reality image that includes the augmentation image overlaid on the visible-light image;

measuring, with a gaze sensor of the augmented-reality device, one or more gaze parameters of a user eye;

recognizing a gaze vector from the one or more gaze parameters;

determining an angle of the identified surface relative to a perspective of the user eye; and

automatically visually presenting, via the see-through display of the augmented-reality device, the mixed-reality image in alignment with an identified surface nearest the gaze vector at a non-parallel angle relative to the identified surface if the angle is less than a threshold angle.

16. The method of claim 15 , wherein the mixed-reality image is automatically visually presented via the see-through display responsive to the visible-light camera collecting the visible light image.

17. The method of claim 15 , further comprising:

visually presenting, via the see-through display of the augmented-reality device, an animation in which the mixed-reality image appears to move away from a perspective of the user of the augmented-reality device into alignment with the identified surface of the physical space.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 6, 2017
From: WRIGHT, SHAWN CRISPIN; OSBORN, DAN; CERRITENO, ALBERTO E.; HSU, BOFAN; JONES, EVAN; THOMPSON, JOE; GOUIN, FOREST WOODCROFT; SAUNDERS, MEGAN
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 043806/0307 →
Continuity (1)
Related Publication 20160379408A1 · Dec 29, 2016