Granted Patent B2

US 9,898,844 · App. 14/145,458 · Granted Feb 20, 2018

Inventor: Brian Mullins (Sierra Madre, CA)

Assignee: DAQRI, LLC

G06T11/60G06T19/006G06T2210/61

Abstract

Claims (75)

1. A method comprising:

capturing, by a head mounted viewing device, an image of a local environment;

identifying, based on an analysis of the image of the local environment, a first physical object and a second physical object located in the local environment;

determining, based on the analysis of the image, a location and physical size of the first physical object and a location and a physical size of the second physical object;

generating, based on the location and the physical size of the first physical object in the local environment and the location and the physical size of the second physical object in the local environment, a physical layout of the local environment, the physical layout of the local environment defining a relative position of the first physical object and the second physical object within the local environment and the physical size of the first physical object and the physical size of the second physical object;

comparing the physical layout of the local environment to a set of saved physical layouts stored in a memory of the viewing device, yielding a comparison, each saved physical layout defining relative positions and physical sizes of physical objects located in a respective local environment corresponding to the saved physical layout;

determining, based on the comparison, that the physical layout of the local environment matches a first saved physical layout;

retrieving a virtual object corresponding to the first saved physical layout;

displaying the virtual object on a display of the head mounted viewing device;

detecting that a new physical object has been added to the local environment;

adjusting a position of the virtual object on the display in response to detecting the new physical object.

2. The method of claim 1 , wherein the local environment comprises a scenery captured with an optical sensor.

3. The method of claim 1 , wherein identifying the first physical object and the second physical object comprises:

identifying feature points of the first physical object and the second physical object;

identifying the first physical object and the second physical object based on the feature points of the first physical object and the second physical object.

4. The method of claim 1 , further comprising:

identifying a physical change in the local environment; and

modifying the virtual object in response to the physical change.

5. The method of claim 4 , wherein the physical change comprises at least one of a change in a location of a physical object within the local environment or a removal of a physical object from the local environment.

6. The method of claim 5 , wherein modifying the content of the virtual object further comprises:

modifying a display and a behavior of the virtual object in response to the physical change.

7. The method of claim 6 , wherein modifying the display and the behavior of the virtual object comprises:

modifying a size or shape of the virtual object on the display in relation to the physical change in the local environment; and

modifying a behavior of the virtual object in relation to the physical change in the local environment.

8. The method of claim 4 , further comprising:

dynamically generating a new behavior for the virtual object in response to the physical change in the local environment using a library of virtual object alternative behaviors.

9. The method of claim 4 , further comprising:

generating a physical change identifier in response to the physical change in the local environment;

communicating the physical change identifier to a server;

receiving from the server a new behavior model for the virtual object corresponding to the physical change identifier; and

modifying a behavior of the virtual object based on the new behavior model.

10. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a head mounted viewing device, cause the head mounted viewing device to perform operations comprising:

capturing an image of a local environment;

identifying, based on an analysis of the image of the local environment, at least a first physical object and a second physical object located in the local environment;

determining, based on the analysis of the image, a location and physical size of the first physical object and a location and a physical size of the second physical object;

generating, based on the location and the physical size of the first physical object in the local environment and the location and the physical size of the second physical object in the local environment, a physical layout of the local environment, the physical layout of the local environment defining a relative position of the first physical object and the second physical object within the local environment and the physical size of the first physical object and the physical size of the second physical object;

comparing the physical layout of the local environment to a set of saved physical layouts stored in a memory of the viewing device, yielding a comparison, each saved physical layout defining relative positions and physical sizes of physical objects located in a respective local environment corresponding to the saved physical layout;

determining, based on the comparison, that the physical layout of the local environment matches a first saved physical layout;

retrieving a virtual object corresponding to the first saved physical layout;

displaying the virtual object on a display of the head mounted viewing device;

detecting that a new physical object has been added to the local environment;

adjusting a position of the virtual object on the display in response to detecting the new physical object.

11. A head mounted viewing device comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the head mounted viewing device to perform operations comprising:

capturing an image of a local environment;

identifying, based on an analysis of the image of the local environment, at least a first physical object and a second physical object located in the local environment;

determining, based on the analysis of the image, a location and physical size of the first physical object and a location and a physical size of the second physical object;

generating, based on the location and the physical size of the first physical object in the local environment and the location and the physical size of the second physical object in the local environment, a physical layout of the local environment, the physical layout of the local environment defining a relative position of the first physical object and the second physical object within the local environment and the physical size of the first physical object and the physical size of the second physical object;

comparing the physical layout of the local environment to a set of saved physical layouts stored in a memory of the viewing device, yielding a comparison, each saved physical layout defining relative positions and physical sizes of physical objects located in a respective local environment corresponding to the saved physical layout;

determining, based on the comparison, that the physical layout of the local environment matches a first saved physical layout;

retrieving a virtual object corresponding to the first saved physical layout;

displaying the virtual object on a display of the head mounted viewing device;

detecting that a new physical object has been added to the local environment;

adjusting a position of the virtual object on the display in response to detecting the new physical object.

12. The head mounted viewing device of claim 11 , wherein the local environment comprises a scenery captured with an optical sensor.

13. The head mounted viewing device of claim 11 , wherein identifying the first physical object and the second physical object comprises:

identifying feature points of the first physical object and the second physical object;

identifying the first physical object and the second physical object based on the feature points of the first physical object and the second physical object.

14. The head mounted viewing device of claim 11 , the operations further comprising:

identifying a physical change in the local environment; and

modifying the virtual object in response to the physical change.

15. The head mounted viewing device of claim 14 , wherein the physical change comprises at least one of a change in a location of a physical object within the local environment or a removal of a physical object from the local environment.

16. The head mounted viewing device of claim 15 , wherein modifying the content of the virtual object further comprises:

modifying a display and a behavior of the virtual object in response to the physical change.

17. The head mounted viewing device of claim 16 , wherein modifying the display and the behavior of the virtual object comprises:

modifying a size or shape of the virtual object on the display in relation to the physical change in the local environment; and

modifying a behavior of the virtual object in relation to the physical change in the local environment.

18. The head mounted viewing device of claim 14 , the operations further comprising:

dynamically generating a new behavior for the virtual object in response to the physical change in the local environment using a library of virtual object alternative behaviors.

19. The head mounted viewing device of claim 14 , the operations further comprising:

generating a physical change identifier in response to the physical change in the local environment;

communicating the physical change identifier to a server;

receiving from the server a new behavior model for the virtual object corresponding to the physical change identifier; and

modifying a behavior of the virtual object based on the new behavior model.

Assignments (12)

Continuity (1)