Granted Patent B2

US 9,773,349 · App. 14/184,523 · Granted Sep 26, 2017

Inventor: Brian Mullins (Sierra Madre, CA)

Assignee: DAQRI, LLC

G06T19/006G02B27/0093G02B27/0101G02B27/0172G06F3/013G06K9/00597G02B2027/014G02B2027/0138

Abstract

Claims (55)

1. A computer-implemented method comprising:

receiving image data of real-world visual content, the image data having been captured by an image capture device;

determining virtual content based on the received image data;

determining a first gaze direction of at least one eye of a user;

making, by a machine having a memory and at least one processor, a determination about the virtual content based on the first gaze direction, the making the determination comprising determining a first location on a display surface at which to display the virtual content, the first location being determined using a first distance between the at least one eye of the user and a point at which the first gaze direction intersects with the display surface and a second distance between the real-world visual content and the point at which the first gaze direction intersects with the display surface as parameters;

causing the virtual content to be presented to the user based on the determination, the causing the virtual content to be presented comprising causing the virtual content to be displayed on the display surface at the first location, the virtual content displayed at the first location comprising a visual object;

determining a second gaze direction of the at least one eye of the user, the second gaze direction being different from the first gaze direction;

determining a second location on the display surface at which to display the virtual content, the second location being different from the first location and determined using a third distance between the at least one eye of the user and a point at which the second gaze direction intersects with the display surface and a fourth distance between the real-world content and the point at which the second gaze direction intersects with the display surface as parameters; and

causing the virtual content to be displayed on the display surface at the second location, the virtual content displayed at the second location comprising the same visual object as the virtual content displayed at the first location, the causing of the virtual content to be displayed on the display surface at the second location comprising an adjustment of a display location of the virtual content from the first location to the second location based on a change in gaze direction of the user from the first gaze direction to the second gaze direction.

2. The method of claim 1 , wherein determining the first location comprises:

emitting at least one light ray from at least one light emitting diode towards the at least one eye;

observing at least one glint reflected off of the at least one eye using at least one camera;

determining a corneal center of the at least one eye based on the observed at least one glint;

determining a path of incident light from the real-world visual content to the corneal center of the at least one eye; and

adjusting positioning of at least one virtual content light ray forming the virtual content on the display surface to make at least one reflected light ray reflecting off of the display surface as a result of the at least one virtual content light ray coincident with the incident light ray from the real-world visual content.

3. The method of claim 1 , wherein the display surface is transparent.

4. The method of claim 1 , wherein the display surface and the image capture device are coupled to a heads-up display device.

5. The method of claim 4 , wherein the heads-up display device comprises an optical head-mounted display device.

6. The method of claim 1 , further comprising:

storing a plurality of audio files in a database, each one of the plurality of audio files being stored in association with a corresponding gaze direction;

selecting a first one of the audio files to be presented to the user based on the first gaze direction; and

causing the selected first one of the audio files to be presented to the user.

7. The method of claim 1 , wherein:

a first angle between the first gaze direction and a direction from which the second distance between the real-world content and the point at which the first gaze direction intersects with the display surface is measured is also used as a parameter in determining the first location; and

a second angle between the second gaze direction and a direction from which the fourth distance between the real-world content and the point at which the second gaze direction intersects with the display surface is measured is also used as a parameter in determining the second location.

8. A system comprising:

an image capture device configured to capture image data of real-world visual content;

at least one processor; and

a computer-readable medium storing executable instructions that, when executed, cause the at least one processor to perform operations comprising:

determining virtual content based on the captured image data;

determining a first gaze direction of at least one eye of a user;

making a determination about the virtual content based on the first gaze direction, the making the determination comprising determining a first location on a display surface at which to display the virtual content, the first location being determined using a first distance between the at least one eye of the user and a point at which the first gaze direction intersects with the display surface and a second distance between the real-world visual content and the point at which the first gaze direction intersects with the display surface as parameters;

causing the virtual content to be presented to the user based on the determination, the causing the virtual content to be presented comprising causing the virtual content to be displayed on the display surface at the first location, the virtual content displayed at the first location comprising a visual object;

determining a second gaze direction of the at least one eye of the user, the second gaze direction being different from the first gaze direction;

determining a second location on the display surface at which to display the virtual content, the second location being different from the first location and determined using a third distance between the at least one eye of the user and a point at which the second gaze direction intersects with the display surface and a fourth distance between the real-world content and the point at which the second gaze direction intersects with the display surface as parameters; and

causing the virtual content to be displayed on the display surface at the second location, the virtual content displayed at the second location comprising the same visual object as the virtual content displayed at the first location, the causing of the virtual content to be displayed on the display surface at the second location comprising an adjustment of a display location of the virtual content from the first location to the second location based on a change in gaze direction of the user from the first gaze direction to the second gaze direction.

9. The system of claim 8 , further comprising:

at least one light emitting diode configured to emit at least one light ray towards the at least one eye; and

at least one camera configured to observe at least one glint reflected off of the at least one eye,

wherein the operations further comprise:

determining a corneal center of the at least one eye based on the observed at least one glint,

determining a path of incident light from the real-world visual content to the corneal center of the at least one eye, and

adjusting a positioning of at least one virtual content light ray forming the virtual content on the display surface to make at least one reflected light ray reflecting off of the display surface as a result of the at least one virtual content light ray coincident with the incident light ray from the real-world visual content.

10. The system of claim 8 , wherein the display surface is transparent.

11. The system of claim 8 , wherein the display surface and the image capture device are coupled to a heads-up display device.

12. The system of claim 11 , wherein the heads-up display device comprises an optical head-mounted display device.

13. A non-transitory machine-readable storage medium, tangibly embodying a set of instructions that, when executed by at least one processor, causes the at least one processor to perform a set of operations comprising:

receiving image data of real-world visual content, the image data having been captured by an image capture device;

determining virtual content based on the received image data;

determining a first gaze direction of at least one eye of a user;

making a determination about the virtual content based on the first gaze direction, the making the determination comprising determining a first location on a display surface at which to display the virtual content, the first location being determined using a first distance between the at least one eye of the user and a point at which the first gaze direction intersects with the display surface and a second distance between the real-world visual content and the point at which the first gaze direction intersects with the display surface as parameters;

causing the virtual content to be presented to the user based on the determination, the causing the virtual content to be presented comprising causing the virtual content to be displayed on the display surface at the first location, the virtual content displayed at the first location comprising a visual object;

determining a second gaze direction of the at least one eye of the user, the second gaze direction being different from the first gaze direction;

determining a second location on the display surface at which to display the virtual content, the second location being different from the first location and determined using a third distance between the at least one eye of the user and a point at which the second gaze direction intersects with the display surface and a fourth distance between the real-world content and the point at which the second gaze direction intersects with the display surface as parameters; and

causing the virtual content to be displayed on the display surface at the second location, the virtual content displayed at the second location comprising the same visual object as the virtual content displayed at the first location, the causing of the virtual content to be displayed on the display surface at the second location comprising an adjustment of a display location of the virtual content from the first location to the second location based on a change in gaze direction of the user from the first gaze direction to the second gaze direction.

Assignments (12)

Continuity (1)