Augmented books in a mixed reality environment
A system and method are disclosed for augmenting a reading experience in a mixed reality environment. In response to predefined verbal or physical gestures, the mixed reality system is able to answer a user's questions or provide additional information relating to what the user is reading. Responses may be displayed to the user on virtual display slates in a border or around the reading material without obscuring text or interfering with the user's reading experience.
1. A method of presenting a mixed reality experience to a user viewing a reading material including at least one of text and an image, the method comprising:
(a) identifying a position of the reading material in three-dimensional space;
(b) tracking a position of a user's eyes with respect to text or images in the reading material;
(c) receiving a request for information regarding a portion of the reading material;
(d) searching for a response to the request for information received in said step (c);
(e) displaying the response to a user, if found in said step (d), in a first virtual display slate at a position in the user's field of view that does not overlap with text and/or images from the reading material;
(f) pinning the position of the virtual display slate in a fixed position with respect to the reading material in the user's field of view; and
(g) displaying a second virtual display slate to the user, the second virtual display slate included by an author of a software application relating to the reading material, the software application executing on a computing system associated with the mixed reality environment, the second virtual display slate displaying additional information about a portion of the reading material when the user reaches the portion of the reading material.
2. The method of claim 1 , wherein the response displayed in said step (e) is one of text, an image and a video.
3. The method of claim 1 , further comprising adding a third virtual image in relation to reading material including an annotation with user-defined content.
4. The method of claim 3 , wherein the annotation is one of text, an image, a video, a data file, and audio file and an executable software application file.
5. The method of claim 1 , wherein the reading material or image is one of a tangible reading material or image, an electronic reading material or image, or a virtual reading material or image.
6. The method recited in claim 1 , wherein said step (c) of receiving a request for information comprises the step of receiving one of a plurality of predefined gestures associated with specific requests for information.
7. The method recited in claim 6 , wherein the plurality of predefined gestures are at least one of physical gestures and verbal gestures.
8. The method recited in claim 6 , wherein the plurality of predefined gestures are gestures for requesting at least one of: i) a definition of selected text, ii) accumulated knowledge on a person, character, place or thing selected in the reading material, iii) a map of a place mentioned in the reading material, iv) a map of relations between people or characters from the reading material, and v) a graphical representation of a person, character, place or thing recited in the text.
9. The method recited in claim 1 , wherein said step (c) of receiving a request for information regarding a portion of the reading material comprises the step of receiving one of a plurality of predefined gestures for identifying the portion of reading material on which information is sought.
10. The method recited in claim 9 , wherein the plurality of predefined gestures for identifying the portion of reading material including at least one of: i) pointing at the portion of reading material, ii) gazing at the portion of reading material, iii) speaking the portion of reading material, and iv) touching a display screen to highlight the portion of reading material.
11. The method recited in claim 1 , wherein said step (d) of searching for a response to the request for information includes the step of searching for a response in a memory of a computing device assisting in the presentation of the mixed reality experience.
12. The method recited in claim 11 , further comprising the step of executing a software application on the computing device, the software application including queriable information relating to the reading material.
13. The method recited in claim 1 , wherein said step (d) of searching for a response to the request for information includes the step of searching for a response at a designated website accessible via a network connection.
14. The method recited in claim 1 , wherein response displayed in said step (e) includes one of: i) a definition of selected text, ii) accumulated knowledge on a person, character, place or thing selected in the reading material, iii) a map of a place mentioned in the reading material, iv) a map of relations between people or characters from the reading material, v) a graphical representation of a person, character, place or thing recited in the text, vi) a synopsis of a story set forth in the reading material, the synopsis not including subject matter from the story set forth in the story after a point of the selected text.