IP Library Granted Patent US 9,626,802
Granted Patent B2
US 9,626,802 · App. 14/267,755 · Granted Apr 18, 2017

Determining coordinate frames in a dynamic environment

Inventors: Adam G. Poulos (Sammamish, WA); Arthur Tomlin (Kirkland, WA); Tony Ambrus (Seattle, WA); Jeffrey Cole (Seattle, WA); Ian Douglas McIntyre (Redmond, WA); Drew Steedly (Redmond, WA); Frederik Schaffalitzky (Bellevue, WA); Georg Klein (Seattle, WA); Kathleen P. Mulcahy (Seattle, WA)
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
G06T19/006G02B27/017G06F3/012G06F3/017G06F3/04815G06K9/00624G06K9/00791G06K9/00832G06K9/4604G06K9/6215G06T7/20G02B2027/0178G06K9/00208G06K9/00671G06K9/2018
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,626,802
App. No.
14/267,755
Granted
Apr 18, 2017
Kind
B2
Abstract

Embodiments are disclosed for methods and systems of distinguishing movements of features in a physical environment. For example, on a head-mounted display device, one embodiment of a method includes obtaining a representation of real-world features in two or more coordinate frames and obtaining motion data from one or more sensors external to the head-mounted display device. The method further includes distinguishing features in one coordinate frame from features in another coordinate frame based upon the motion data.

Claims (38)

1. On a head-mounted display device, a method comprising:

obtaining via image data from a camera a representation of real-world features in two or more coordinate frames;

obtaining motion data from one or more sensors external to the head-mounted display device, each sensor being associated with a corresponding coordinate frame;

displaying, on a see-through display of the head-mounted display device, a first virtual object at location relative to a first real-world feature in a first coordinate frame in a field of view of the display device and a second virtual object at location relative to a second real-world feature in a second coordinate frame in the field of view of the display device;

determining relative motion characteristics of the sensors based upon the motion data;

distinguishing features in the first coordinate frame from features in the second coordinate frame based upon the relative motion characteristics; and

updating the location of the first virtual object based upon a relative motion of the first coordinate frame and updating the location of the second virtual object based upon a relative motion of the second coordinate frame.

2. The method of claim 1 , further comprising tracking locations of the first real-world feature and the second real-world feature relative to the camera over time via the image data.

3. The method of claim 2 , further comprising determining a motion characteristic for the first real-world feature and a motion characteristic for the second real-world feature based on tracking changes in the locations of the first real-world feature and the second real-world feature.

4. The method of claim 3 , wherein distinguishing the features in the first coordinate frame from the features in the second coordinate frame comprises comparing the motion characteristics for each of the first real-world feature and the second real-world feature to the motion data from the one or more sensors.

5. The method of claim 4 , wherein a first sensor is associated with the first coordinate frame, and the method further comprising associating the first real-world feature to the first coordinate frame by comparing the motion characteristic determined for the first real-world feature with motion data from the first sensor.

6. The method of claim 5 , wherein the one or more sensors includes a second sensor associated with the second coordinate frame.

7. The method of claim 1 , wherein obtaining the representation of the real-world features comprises detecting one or more objects, corners of objects, and/or edges of objects in a physical environment.

8. The method of claim 1 , wherein displaying the first virtual object and displaying the second virtual object comprise displaying the first virtual object and the second virtual object simultaneously.

9. The method of claim 1 , wherein the one or more sensors includes an inertial measurement unit external to the head-mounted display device.

10. The method of claim 9 , further comprising isolating movement of the head-mounted display device in the first coordinate frame relative to another moving element in the second coordinate frame by subtracting a motion signal from the inertial measurement unit external to the head-mounted display device from a motion signal from an inertial measurement unit on the head-mounted display device.

11. A head-mounted display device, comprising:

a plurality of sensors;

a see-through display;

a logic machine; and

a memory holding instructions executable by the logic machine to

obtain a representation of real-world features in two or more coordinate frames, the representation obtained from image data from a camera;

receive sensor data from a first sensor corresponding to a first coordinate frame and a second sensor corresponding to a second coordinate frame, and one or more of the first sensor and the second sensor being located external to the head-mounted display device;

determine relative motion characteristics of the first sensor and the second sensor based upon motion data;

distinguish a first real-world feature in the first coordinate frame from a second real-world feature in the second coordinate frame based upon the relative motion characteristics; and

display a first virtual object at location relative to the first real-world feature in the first coordinate frame based upon the relative motion characteristics and display a second virtual object at location relative to the second real-world feature in the second coordinate frame based upon the relative motion characteristics.

12. The head-mounted display device of claim 11 , wherein obtaining the representation of the real-world features further comprises tracking locations of the first real-world feature and the second real-world feature relative to the camera over time based on the image data from the camera.

13. The head-mounted display device of claim 11 , wherein the first sensor and the second sensor are external to the head-mounted display device.

14. The head-mounted display device of claim 11 , wherein the first sensor and the second sensor each includes one or more of an external camera and an external inertial measurement unit.

15. The head-mounted display device of claim 11 , wherein the plurality of sensors includes a global positioning sensor unit.

16. The head-mounted display device of claim 11 , wherein the instructions are further executable to update a location at which the first virtual object is displayed based upon a detected change in the location of the first real-world feature relative to a coordinate frame of the head-mounted display device.

17. On a head-mounted display device, a method comprising:

obtaining a representation of real-world features in a coordinate frame using image data from a camera;

obtaining motion data from two or more inertial measurement units, one or more of the inertial measurement units being external to the head-mounted display device;

detecting movement of a first real-world feature relative to a second real-world feature based on information from the image data and the motion data;

separating the coordinate frame into different coordinate frames including a first coordinate frame and a second coordinate frame, the first real-world feature being associated with the first coordinate frame based upon data from a first inertial measurement unit and the second real-world feature being associated with the second coordinate frame based upon data from a second inertial measurement unit; and

displaying a first virtual object associated with the first real-world feature in the first coordinate frame and a second virtual object associated with the second real-world feature in the second coordinate frame.

18. The method of claim 17 , further comprising merging the first and second coordinate frames into a single coordinate frame based at least on the first and second coordinate frames not having relative motion.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 16, 2016
From: POULOS, ADAM G.; TOMLIN, ARTHUR; AMBRUS, TONY; COLE, JEFFREY; MCINTYRE, IAN DOUGLAS; STEEDLY, DREW; SCHAFFALITZY, FREDERIK; KLEIN, GEORG; MULCAHY, KATHLEEN P.
To: MICROSOFT CORPORATION
Reel/Frame 039453/0775 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 9, 2015
From: MICROSOFT CORPORATION
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 039025/0454 →
Continuity (1)
Related Publication 20150317834A1 · Nov 5, 2015