IP Library › Granted Patent US 9,583,032
Granted Patent B2
US 9,583,032 · App. 13/488,966 · Granted Feb 28, 2017

Navigating content using a physical object

Inventors: Mathew J. Lamb (Mercer Island, WA); Ben J. Sugden (Woodinville, WA); Robert L. Crocco, Jr. (Seattle, WA); Brian E. Keane (Bellevue, WA); Christopher E. Miles (Seattle, WA); Kathryn Stone Perez (Kirkland, WA); Laura K. Massey (Redmond, WA); Alex Aben-Athar Kipman (Redmond, WA); Sheridan Martin Small (Seattle, WA); Stephen G. Latta (Seattle, WA)
Assignee: Microsoft Technology Licensing, LLC
G09G3/003
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,583,032
App. No.
13/488,966
Granted
Feb 28, 2017
Kind
B2
Abstract

Technology is disclosed herein to help a user navigate through large amounts of content while wearing a see-through, near-eye, mixed reality display device such as a head mounted display (HMD). The user can use a physical object such as a book to navigate through content being presented in the HMD. In one embodiment, a book has markers on the pages that allow the system to organize the content. The book could have real content, but it could be blank other than the markers. As the user flips through the book, the system recognizes the markers and presents content associated with the respective marker in the HMD.

Claims (84)

1. A method for navigating content, comprising:

receiving, by an electronic device coupled to a camera and a see-through, near-eye, mixed reality display, input that specifies what content is to be navigated by a user wearing the see-through, near-eye, mixed reality display, the content comprising an ordered data set from a calendar program that is organized based on a unit of time;

dividing, by the electronic device, the ordered data set into divisions based on the unit of time;

associating, by the electronic device, each of the divisions with a unique marker in a physical object that includes an ordered sequence of pages, the unique markers being on different ones of the pages and the divisions being assigned to different ones of the pages in accordance with an order of the ordered data set, the unit of time for each division being assigned to a unique marker;

identifying the unique markers in the physical object using the camera as the user manipulates the physical object, including identifying the unique markers in the ordered sequence of pages;

determining, by the electronic device, what division of the content is associated with each of the uniquely identified markers; and

presenting images representing the divisions of the content associated with the uniquely identified markers in the see-through, near-eye, mixed reality display device, including presenting images representing the unit of time for the respective divisions in response to identifying the unique markers.

2. The method of claim 1 , further comprising:

presenting a table of contents in the see-through, near-eye, mixed reality display device, the table of contents defines at which page of the ordered sequence of pages various divisions of the content can be accessed.

3. The method of claim 1 , further comprising:

presenting chapter headings in the see-through, near-eye, mixed reality display device, the chapter headings pertaining to the content.

4. The method of claim 1 , wherein the pages have content that is completely unrelated to the content presented in the see-through, near-eye, mixed reality display device for the corresponding page, the content on the pages serves as the markers.

5. The method of claim 1 , wherein the content is a media file, the presenting images includes skipping over a pre-determined time in the media file for each page in the ordered sequence of pages that is identified.

6. The method of claim 1 , wherein the page that the division is associated with corresponds to the location of the division in the content.

7. The method of claim 1 , wherein the dividing the content from the calendar program into divisions that each comprise the same unit of time comprises dividing the content from the calendar program based on days; wherein the associating each of the divisions with a unique marker in a physical object that includes an ordered sequence of pages comprises associating days to the pages.

8. The method of claim 1 , wherein the content is from a file system browser; wherein the dividing the ordered data set into divisions comprises dividing directories associated with the file system browser; wherein the associating each of the divisions with a unique marker in a physical object that includes an ordered sequence of pages comprises binding the directories to the pages.

9. The method of claim 1 , further comprising:

predicting, by the electronic device, which of the unique markers will be identified next by the camera in response to the user manipulating the physical object;

determining, by the electronic device, what division of the content is associated with the predicted marker; and

performing, by the electronic device, an anticipatory operation with respect to the division of the content that is associated with the predicted marker.

10. The method of claim 1 , wherein the physical object comprises a book having the ordered sequence of pages.

11. The method for navigating content of claim 10 , further comprising:

accessing a thickness of the book, the pages having edges; and

presenting a navigation aid to the user in the see-through, near-eye, mixed reality display as the user manipulates the book, comprising rendering tabs on the edges of the pages based on the thickness of the book.

12. A system comprising:

a see-through, near-eye display device;

an image sensor;

logic in communication with the display device and the image sensor, the logic is configured to:

receive input that specifies what digital content is to be navigated by a user that is manipulating a physical object that has unique markers, the physical object includes an ordered sequence of pages having edges, the unique markers being on different ones of the pages;

access a thickness of the physical object;

access the digital content to be navigated, the digital content comprises a set of ordered records;

divide the digital content into divisions;

associate each of the divisions with one of the unique markers in accordance with an order of the set of ordered records, the divisions being assigned to different ones of the pages, each division being assigned to a unique marker;

identify the unique markers in the physical object using the image sensor as the user manipulates the physical object, including identifying the unique markers in the ordered sequence of pages;

identify divisions of the digital content that are associated with the identified unique markers;

present images representing the identified divisions of the digital content in the see-through, near-eye, display device, including presenting images representing respective divisions in response to identifying the unique markers; and

present a navigation aid to the user in the see-through, near-eye, mixed reality display as the user manipulates the physical object, comprising the logic configured to render tabs on the edges of the pages of the physical object based on the thickness of the physical object.

13. The see-through, near-eye, display device system of claim 12 , wherein the logic is configured to:

present a table of contents in the see-through, near-eye display device, the table of contents defines which page of the ordered sequence of pages various portions of the digital content can be accessed at.

14. The see-through, near-eye, display device system of claim 12 , wherein the logic is configured to:

present chapter headings in the see-through, near-eye, mixed reality display device, the chapter headings pertaining to the digital content.

15. The see-through, near-eye, display device system of claim 12 , wherein the logic is further configured to scrub through the digital content in response to the user turning the pages, the digital content further comprises a media file including audio and/or video signals, the logic configured to skip over a pre-determined time of the media file for each page in the ordered sequence of pages that is identified using the image sensor.

16. The see-through, near-eye, display device system of claim 12 , wherein the logic being configured to present images representing the identified divisions of the digital content in the see-through, near-eye, display device includes the logic being configured to:

present one element of the digital content for each page of the ordered sequence of pages.

17. The see-through, near-eye, display device system of claim 12 , wherein the digital content comprises an ordered data set from a calendar program, wherein the logic is configured to:

divide the digital content from the calendar program into divisions that each comprise a day, the days being assigned to different ones of the pages of the physical object; and

in response to identifying the unique markers, present images representing the days in the see-through, near-eye, display device.

18. The see-through, near-eye, display device system of claim 12 , wherein the physical object comprises a book having the ordered sequence of pages.

19. A system comprising:

a see-through, near-eye display device;

an image sensor;

logic in communication with the display device and the image sensor, the logic is configured to:

receive input that specifies that directories associated with a file browser are to be navigated using a physical object that includes an ordered sequence of pages;

associate ones of the directories with unique markers in the physical object, the unique markers being on different ones of the pages and the ones of the directories being assigned to different ones of the pages in accordance with an order of the directories;

identify the unique markers in the ordered sequence of pages using the image sensor as a user manipulates the physical object;

determine which of the directories are associated with the identified unique markers; and

in response to identifying the unique markers, present images representing the directories in the see-through, near-eye, display device.

20. The system of claim 19 , wherein the logic is configured to bind the ones of the directories with unique markers in the physical object in response to the file browser being opened.

21. A system comprising:

a see-through, near-eye display;

an image sensor;

logic in communication with the see-through, near-eye display and the image sensor, the logic configured to:

receive input that specifies what content is to be navigated by a user wearing the see-through, near-eye display, the content comprising an ordered data set from a calendar program that is organized based on a unit of time;

divide the ordered data set into divisions based on the unit of time;

associate each of the divisions with a unique marker in a physical object that includes an ordered sequence of pages, the unique markers being on different ones of the pages and the divisions being assigned to different ones of the pages in accordance with an order of the ordered data set, the unit of time for each division being assigned to a unique marker;

identify the unique markers in the physical object using the image sensor as the user manipulates the physical object, including the logic configured to identify the unique markers in the ordered sequence of pages;

determine what division of the content is associated with each of the uniquely identified markers; and

present images representing the divisions of the content associated with the uniquely identified markers in the see-through, near-eye display, including the logic configured to present images that represent the unit of time for the respective divisions in response to identifying the unique markers.

22. A method, comprising:

receiving, at an electronic device, input that specifies what digital content is to be navigated by a user that is manipulating a physical object that has unique markers, the physical object includes an ordered sequence of pages having edges, the unique markers being on different ones of the pages, the electronic device comprising an image sensor and a see-through, near-eye display;

accessing, by the electronic device, a thickness of the physical object;

accessing, by the electronic device, the digital content to be navigated, the digital content comprises a set of ordered records;

dividing the digital content into divisions by the electronic device;

associating, by the electronic device, each of the divisions with one of the unique markers in accordance with an order of the set of ordered records, the divisions being assigned to different ones of the pages, each division being assigned to a unique marker;

identifying the unique markers in the physical object using the image sensor as the user manipulates the physical object, including identifying the unique markers in the ordered sequence of pages;

identifying, by the electronic device, divisions of the digital content that are associated with the identified unique markers;

presenting images representing the identified divisions of the digital content in the see-through, near-eye display, including presenting images that represent respective divisions in response to identifying the unique markers; and

presenting a navigation aid to the user in the see-through, near-eye display as the user manipulates the physical object, comprising rendering tabs on the edges of the pages of the physical object based on the thickness of the physical object.

23. A method, comprising:

receiving, at an electronic system, input that specifies that directories associated with a file browser are to be navigated using a physical object that includes an ordered sequence of pages, the electronic system comprising a see-through, near-eye display and an image sensor;

associating, by the electronic system, ones of the directories with unique markers in the physical object, the unique markers being on different ones of the pages and the ones of the directories being assigned to different ones of the pages in accordance with an order of the directories;

identifying the unique markers in the ordered sequence of pages using the image sensor as a user manipulates the physical object;

determining, by the electronic system, which of the directories are associated with the identified unique markers; and

in response to identifying the unique markers, presenting images representing the directories in the see-through, near-eye display.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 9, 2014
From: MICROSOFT CORPORATION
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 034544/0541 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 7, 2013
From: LAMB, MATHEW J.; SUGDEN, BEN J.; CROCCO, ROBERT L., JR.; KEANE, BRIAN E.; MILES, CHRISTOPHER E.; PEREZ, KATHRYN STONE; MASSEY, LAURA K.; KIPMAN, ALEX ABEN-ATHAR; SMALL, SHERIDAN MARTIN; LATTA, STEPHEN G.
To: MICROSOFT CORPORATION
Reel/Frame 030964/0371 →
Continuity (1)
Related Publication 20130321255A1 · Dec 5, 2013