Granted Patent B1

US 9,767,613 · App. 14/626,683 · Granted Sep 19, 2017

Abstract

Claims (63)

1. A method of interacting with a virtual object in an augmented reality environment using a head mounted device and at least one control object for gesture control, the method including:

referencing the virtual object for display by the head mounted device in an augmented reality space;

identifying a physical location of a device in at least one image of the augmented reality space;

generating for display, in the augmented reality space and by the head mounted device, a virtual control coincident with and superimposed upon a surface of the device, such that the virtual control includes a partial perspective of at least a portion of the virtual object;

sensing interactions between at least one control object and the virtual control coincident with and superimposed upon the surface of the device;

generating data signaling manipulations of the virtual object for display by the head mounted device in the augmented reality space, the data being generated based upon the sensed interactions between the at least one control object and the virtual control coincident with and superimposed upon the surface of the device; and

emphasizing, in the augmented reality space and by the head mounted device, points of interaction between extremities of the device and the virtual control generated for display by the head mounted device.

2. The method of claim 1 , wherein the control object is a hand.

3. The method of claim 2 , wherein the control object is a portion of the hand.

4. The method of claim 1 , wherein the control object is a tool.

5. The method of claim 1 , wherein the control object is a combination of a hand and a tool.

6. The method of claim 1 , further including:

generating data representing the virtual control across an interface of the device.

7. The method of claim 1 , wherein the device is a smartphone.

8. The method of claim 1 , further including:

sensing interactions between the control object and the virtual control coincident with and superimposed upon the surface of the device when the control object touches the virtual control.

9. The method of claim 1 , further including:

sensing interactions between the control object and the virtual control generated for display by the head mounted device the interactions being sensed when the control object approaches the virtual control generated for display by the head mounted device within an initial hover proximity threshold.

10. The method of claim 1 , wherein the virtual object generated for display by the head mounted device in the augmented reality space is responsive to the manipulations of the virtual control generated for display by the head mounted device in the augmented reality space.

11. The method of claim 1 , further including:

sensing interactions between the control object and the virtual control generated for display by the head mounted device, the interactions being sensed using at least one phase conjugate component that creates a contrast between portions of the control object and a rest of the augmented reality space.

12. The method of claim 11 , wherein the at least one phase conjugate component is a retro-reflector.

13. The method of claim 1 , further including:

sensing gestures in a three-dimensional (3D) sensory space away from the surface and proximate to the virtual object using a motion sensory control device; and

generating data signaling manipulations of the virtual object in the augmented reality space responsive to the gestures.

14. The method of claim 1 , further including:

generating data virtually emphasizing at least part of the control object in the augmented reality space.

15. The method of claim 1 , further including:

generating data virtually representing the control object in the augmented reality space.

16. The method of claim 1 , further comprising:

identifying, by the head mounted device, a real world item other than the device;

determining, based on the identified real world item, a specific virtual control that corresponds to the identified real world item; and

implementing the specific virtual control as the virtual control coincident with and superimposed upon the surface of the device.

17. The method of claim 1 , further comprising:

identifying, by the head mounted device, a physical environment in which the head mounted device resides;

determining, based on the identified physical environment, a specific virtual control that corresponds to the identified physical environment; and

implementing the specific virtual control as the virtual control coincident with and superimposed upon the surface of the device.

18. A system of interacting with a virtual object in an augmented reality environment using a head mounted device and at least one control object for gesture control, the system including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to:

reference the virtual object for display by the head mounted device in an augmented reality space;

identify a physical location of a device in at least one image of the augmented reality space;

generate for display, in the augmented reality space and by the head mounted device, a virtual control coincident with and superimposed upon a surface of the device, such that the virtual control includes a partial perspective of at least a portion of the virtual object;

sense interactions between at least one control object and the virtual control coincident with and superimposed upon the surface of the device;

generate data signaling manipulations of the virtual object for display by the head mounted device in the augmented reality space, the data being generated based upon the sensed interactions between the at least one control object and the virtual control coincident with and superimposed upon the surface of the device; and

emphasize, in the augmented reality space and by the head mounted device, points of interaction between extremities of the device and the virtual control generated for display by the head mounted device.

19. A non-transitory machine readable medium, storing one or more instructions for interacting with a virtual object in an augmented reality environment using a head mounted device and at least one control object for gesture control, the instructions when executed by one or more processors cause the one or more processors to perform the following:

reference a virtual object for display by the head mounted device in an augmented reality space;

identify a physical location of a device in at least one image of the augmented reality space;

generate for display, in the augmented reality space and by the head mounted device, a virtual control coincident with and superimposed upon a surface of the device, such that the virtual control includes a partial perspective of at least a portion of the virtual object;

sense interactions between at least one control object and the virtual control coincident with and superimposed upon the surface of the device;

generate data signaling manipulations of the virtual object for display by the head mounted device in the augmented reality space, the data being generated based upon the sensed interactions between the at least one control object and the virtual control coincident with and superimposed upon the surface of the device; and

emphasize, in the augmented reality space and by the head mounted device, points of interaction between extremities of the device and the virtual control generated for display by the head mounted device.

20. A method of interacting with a virtual object in an augmented reality environment using a head mounted device and at least one control object for gesture control, the method comprising:

referencing the virtual object for display by the head mounted device in an augmented reality space;

identifying a physical location of a device in at least one image of the augmented reality space;

generating for display, in the augmented reality space and by the head mounted device, a virtual control coincident with and superimposed upon a surface of the device, such that the virtual control includes a partial perspective of at least a portion of the virtual object;

sensing interactions between at least one control object and the virtual control coincident with and superimposed upon the surface of the device;

generating data signaling manipulations of the virtual object for display by the head mounted device in the augmented reality space, the data being generated based upon the sensed interactions between the at least one control object and the virtual control coincident with and superimposed upon the surface of the device;

identifying, by the head mounted device, an item displayed by an embedded display of the device;

determining, based on the identified item, a specific virtual control that corresponds to the identified item; and

implementing the specific virtual control as the virtual control coincident with and superimposed upon the surface of the device.

21. A system including one or more processors coupled to a memory, the memory loaded with computer instructions to interact with a virtual object in an augmented reality environment using a head mounted device and at least one control object for gesture control, the instructions, when executed on the processors, implement the method of claim 20 .

22. A non-transitory computer-readable storage medium impressed with computer program instructions to interact with a virtual object in an augmented reality environment using a head mounted device and at least one control object for gesture control, the instructions, when executed on a processor, implement the method of claim 20 .

Assignments (14)