IP Library › Granted Patent US 9,799,145
Granted Patent B2
US 9,799,145 · App. 14/805,727 · Granted Oct 24, 2017

Augmented reality display of scene behind surface

Inventors: Mike Scavezze (Bellevue, WA); Jason Scott (Kirkland, WA); Jonathan Steed (Redmond, WA); Ian McIntyre (Redmond, WA); Aaron Krauss (Snoqualmie, WA); Daniel McCulloch (Kirkland, WA); Stephen Latta (Seattle, WA)
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
G06T19/006G02B27/0172G02B2027/0178
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,799,145
App. No.
14/805,727
Granted
Oct 24, 2017
Kind
B2
Abstract

Embodiments are disclosed that relate to augmenting an appearance of a surface via a see-through display device. For example, one disclosed embodiment provides, on a computing device comprising a see-through display device, a method of augmenting an appearance of a surface. The method includes acquiring, via an outward-facing image sensor, image data of a first scene viewable through the display. The method further includes recognizing a surface viewable through the display based on the image data and, in response to recognizing the surface, acquiring a representation of a second scene comprising one or more of a scene located physically behind the surface viewable through the display and a scene located behind a surface contextually related to the surface viewable through the display. The method further includes displaying the representation via the see-through display.

Claims (42)

1. On a computing device comprising an outward-facing image sensor, a method comprising:

acquiring, via the outward-facing image sensor, image data of a first scene;

recognizing a surface based on the image data;

in response to recognizing the surface, acquiring a representation of a second scene comprising a scene located behind a surface contextually related to the surface; and

displaying the representation via a display device.

2. The method of claim 1 , wherein recognizing the surface comprises identifying a location of the computing device based on one or more of location data from a location sensor and the image data from the outward-facing image sensor, and recognizing the surface based upon the location of the computing device.

3. The method of claim 1 , wherein recognizing the surface comprises recognizing whether the surface is a movable surface or an unmovable surface, and displaying the representation only if the surface is a movable surface.

4. The method of claim 1 , wherein recognizing the surface comprises determining a context of the surface by identifying one or more of an object incorporating the surface and an object located physically behind the surface.

5. The method of claim 4 , wherein acquiring the representation of the second scene comprises selecting the representation from a plurality of representations of scenes comprising surfaces related by context to the surface.

6. The method of claim 1 , further comprising:

receiving an input to acquire a representation of a third scene physically located behind a surface in the second scene;

in response to the input, acquiring the representation of the third scene; and

displaying the representation of the third scene via the display device.

7. The method of claim 1 , wherein the display device is a see-through display-device, and wherein displaying the representation comprises displaying an image comprising a representation of the second scene in spatial registration with the surface to augment an appearance of the surface.

8. The method of claim 1 , further comprising detecting a trigger to display the representation, the trigger comprising one or more of a direct voice command, a contextual trigger, a programmatically-generated trigger, and a gesture.

9. The method of claim 1 , wherein acquiring the representation of the second scene comprises determining a most recent representation of the second scene and acquiring the most recent representation of the second scene.

10. A computing device, comprising:

a see-through display device;

an outward-facing image sensor configured to acquire image data of a scene viewable through the see-through display device;

a logic subsystem configured to execute instructions; and

a data-holding subsystem comprising instructions stored thereon that are executable by a logic subsystem to:

recognize a surface viewable through the display based on the image data,

in response to recognizing the surface, acquire a representation of a scene located behind a surface contextually related to the surface viewable through the display, and

display the representation via the see-through display.

11. The computing device of claim 10 , wherein the image sensor comprises one or more two-dimensional cameras.

12. The computing device of claim 10 , wherein the image sensor comprises one or more depth cameras.

13. The computing device of claim 10 , wherein the instructions are executable to retrieve the representation from a remote device over a computer network.

14. The computing device of claim 13 , wherein the instructions executable to acquire the representation comprise instructions executable to acquire image data previously collected by a device other than the computing device.

15. The computing device of claim 13 , wherein the instructions executable to acquire the representation comprise instructions executable to acquire real-time image data collected by a device other than the computing device.

16. A display device comprising

an outward-facing image sensor;

a display;

a logic device; and

a data-holding device comprising instructions stored thereon that are executable by the logic device to:

acquire, via the outward-facing image sensor, image data of a first scene;

recognize a surface based on the image data;

in response to recognizing the surface, acquire a representation of a second scene comprising a scene located behind a surface contextually related to the surface viewable through the display; and

display the representation via the display of the display device.

17. The display device of claim 16 , further comprising a location sensor, and wherein the instructions executable to recognize the surface comprise instructions executable to identify a location of the logic device based on one or more of location data from the location sensor and the image data from the outward-facing image sensor, and recognize the surface based upon the location of the logic device.

18. The display device of claim 16 , wherein the instructions executable to recognize the surface comprise instructions executable to recognize whether the surface is a movable surface or an unmovable surface, and display the representation only if the surface is a movable surface.

19. The display device of claim 16 , wherein the instructions executable to recognize the surface comprise instructions executable to determine a context of the surface by identifying one or more of an object incorporating the surface and an object located physically behind the surface.

20. The display device of claim 16 , wherein the instructions are executable to acquire the representation of the second scene by selecting the representation from a plurality of representations of scenes comprising surfaces related by context to the surface.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 25, 2017
From: KRAUSS, AARON; MCCULLOCH, DANIEL; LATTA, STEPHEN
To: MICROSOFT CORPORATION
Reel/Frame 043406/0537 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 22, 2015
From: SCAVEZZE, MIKE; SCOTT, JASON; STEED, JONATHAN; MCINTYRE, IAN
To: MICROSOFT CORPORATION
Reel/Frame 036152/0407 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 22, 2015
From: MICROSOFT CORPORATION
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 036152/0527 →
Continuity (2)
Continuation 13569013 · Aug 7, 2012
Related Publication 20150332513A1 · Nov 19, 2015