Systems and methods of creating a realistic grab experience in virtual reality/augmented reality environments
View Patent ↗The technology disclosed relates to a method of realistic rotation of a virtual object for an interaction between a control object in a three-dimensional (3D) sensory space and the virtual object in a virtual space that the control object interacts with. In particular, it relates to detecting free-form gestures of a control object in a three-dimensional (3D) sensory space and generating for display a 3D solid control object model for the control object during the free-form gestures, including sub-components of the control object and in response to detecting a two e sub-component free-form gesture of the control object in the 3D sensory space in virtual contact with the virtual object, depicting, in the generated display, the virtual contact and resulting rotation of the virtual object by the 3D solid control object model.
1. A method of manipulating virtual objects using real motions of one or more hands in a three-dimensional (3D) sensory space, the method including:
capturing an image of the hands in the a three-dimensional (3D) sensory space and sensing a location of the hands;
incorporating the image of the hands into a virtual reality scene; and
outlining in a contrasting color a modeled position of the location of the hands and incorporating the outline into the virtual reality scene.
2. The method of claim 1 , further including changing an appearance of the outline of the hands in the virtual reality scene upon detection of a discrepancy between the image of the hands and the outline.
3. The method of claim 1 , further including changing an appearance of the image of the hands in the virtual reality scene upon detection of a discrepancy between the image of the hands and the outline.
4. A system implementing the method of claim 1 , the system including:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
capture an image of the hands in the a three-dimensional (3D) sensory space and sense a location of the hands;
incorporate the image of the hands into a virtual reality scene; and
outline in a contrasting color a modeled position of the location of the hands and incorporate the outline into the virtual reality scene.
5. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 1 , the method including:
capturing an image of the hands in the a three-dimensional (3D) sensory space and sensing a location of the hands;
incorporating the image of the hands into a virtual reality scene; and
outlining in a contrasting color a modeled position of the location of the hands and incorporating the outline into the virtual reality scene.
6. A method of manipulating virtual objects using real motions of at least one hand in a three-dimensional (3D) sensory space, the method including:
capturing an image of at least one hand in a three-dimensional (3D) sensory space and sensing a location of a first hand;
incorporating the image of the first hand into a virtual reality scene; and
sensing a pinch action between a thumb and first finger of the first hand and rendering a first virtual pinch force image positioned between the thumb and the first finger while the pinch action continues.
7. The method of claim 6 , further including:
sensing a pinch action between a thumb and first finger of a second hand and rendering a second virtual pinch force image positioned between the thumb and the first finger of the second hand while the pinch action continues;
sensing a movement of at least one of the first and second hands that increases a separation distance between the first and second hands, while sensing continuing pinching actions of both the first and second hands; and
rendering a new virtual object between the first and second pinch force images, responsive to the increased separation distance, wherein at least a size of the new virtual object is responsive to the separation distance.
8. The method of claim 7 , wherein an orientation of the new virtual object is responsive to positions of the first and second hands.
9. A system implementing the method of claim 6 , the system including:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
capture an image of at least one hand in a three-dimensional (3D) sensory space and sense a location of a first hand;
incorporate the image of the first hand into a virtual reality scene; and
sense a pinch action between a thumb and first finger of the first hand and render a first virtual pinch force image positioned between the thumb and the first finger while the pinch action continues.
10. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 6 , the method including:
capturing an image of at least one hand in a three-dimensional (3D) sensory space and sensing a location of a first hand;
incorporating the image of the first hand into a virtual reality scene; and
sensing a pinch action between a thumb and first finger of the first hand and rendering a first virtual pinch force image positioned between the thumb and the first finger while the pinch action continues.
11. A method of manipulating virtual objects using real motions of at least one hand in a three-dimensional (3D) sensory space, the method including:
capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of a first hand;
incorporating the image of the first hand into a virtual reality scene that includes a grabable virtual object;
sensing a gesture of the first hand, the gesture made by unconstrained movement of the first hand in the three-dimensional (3D) sensory space and determining whether the gesture is intended to interact with the grabable virtual object by grabbing the virtual object, the determining further including taking into account at least:
an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;
fingertip separations between the fingertips of the first hand and the proximate surface;
a gesture rate at which the first hand closes on the virtual object;
a hand posture, whether suitable for grasping the virtual object or incompatible with grasping; and
linear velocity of the palm of the first hand relative to the virtual object; and
responsive to determining that a gesture of the first hand is intended to grab the virtual object, linking motion of the first hand to manipulation of the virtual object.
12. The method of claim 11 , further including taking into account when determining whether the gesture is intended to interact with the grabable virtual object a maximum separation between any of the fingertips of the first hand and the proximate surface.
13. The method of claim 11 , further including taking into account when determining whether the gesture is intended to interact with the grabable virtual object a rotational velocity of the palm of the first hand in the 3D sensory space.
14. A system implementing the method of claim 11 , the system including:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
capture an image of the hands in a three-dimensional (3D) sensory space and sense a location of a first hand;
incorporate the image of the first hand into a virtual reality scene that includes a grab able virtual object;
sense a gesture of the first hand, the gesture made by unconstrained movement of the first hand in the three-dimensional (3D) sensory space and determine whether the gesture is intended to interact with the grabable virtual object by grabbing the virtual object, the determination further including taking into account at least:
an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;
fingertip separations between the fingertips of the first hand and the proximate surface;
a gesture rate at which the first hand closes on the virtual object;
a hand posture, whether suitable for grasping the virtual object or incompatible with grasping; and
linear velocity of the palm of the first hand relative to the virtual object; and
responsive to the determination that a gesture of the first hand is intended to grab the virtual object, link motion of the first hand to manipulation of the virtual object.
15. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 11 , the method including:
capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of a first hand;
incorporating the image of the first hand into a virtual reality scene that includes a grabable virtual object;
sensing a gesture of the first hand, the gesture made by unconstrained movement of the first hand in the three-dimensional (3D) sensory space and determining whether the gesture is intended to interact with the grabable virtual object by grabbing the virtual object, the determining further including taking into account at least:
an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;
fingertip separations between the fingertips of the first hand and the proximate surface;
a gesture rate at which the first hand closes on the virtual object;
a hand posture, whether suitable for grasping the virtual object or incompatible with grasping; and
linear velocity of the palm of the first hand relative to the virtual object; and
responsive to determining that a gesture of the first hand is intended to grab the virtual object, linking motion of the first hand to manipulation of the virtual object.
16. A method of manipulating virtual objects using real motions of at least one hand in a three-dimensional (3D) sensory space, the method including:
capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of a first hand;
incorporating the image of the first hand into a virtual reality scene that includes a pushable virtual object;
sensing a gesture of the first hand, the gesture made by unconstrained movement of the first hand in the three-dimensional (3D) sensory space and determining whether the gesture is intended to interact with the pushable virtual object by pushing the virtual object, the determining further including taking into account at least:
an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;
fingertip separations between the fingertips of the first hand and the proximate surface;
a hand posture, whether suitable for pushing the virtual object or incompatible with pushing; and
linear velocity of the palm of the first hand relative to the virtual object; and
responsive to determining that a gesture of the first hand is intended to push the virtual object, linking motion of the first hand to manipulation of the virtual object resulting in a displacement of the virtual object in the virtual space.
17. The method of claim 16 , further including taking into account when determining whether the gesture is intended to interact with the pushable virtual object a gesture rate at which the first hand closes on the virtual object.
18. The method of claim 16 , further including taking into account when determining whether the gesture is intended to interact with the pushable virtual object a maximum separation between any of the fingertips of the first hand and the proximate surface.
19. The method of claim 16 , further including taking into account when determining whether the gesture is intended to interact with the pushable virtual object a rotational velocity of the palm of the first hand in the 3D sensory space.
20. A system implementing the method of claim 16 , the system including:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
capture an image of the hands in a three-dimensional (3D) sensory space and sense a location of a first hand;
incorporate the image of the first hand into a virtual reality scene that includes a pushable virtual object;
sense a gesture of the first hand, the gesture made by unconstrained movement of the first hand in the three-dimensional (3D) sensory space and determine whether the gesture is intended to interact with the pushable virtual object by pushing the virtual object, the determination further including taking into account at least:
an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;
fingertip separations between the fingertips of the first hand and the proximate surface;
a hand posture, whether suitable for pushing the virtual object or incompatible with pushing; and
linear velocity of the palm of the first hand relative to the virtual object; and
responsive to the determination that a gesture of the first hand is intended to push the virtual object, link motion of the first hand to manipulation of the virtual object resulting in a displacement of the virtual object in the virtual space.
21. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 16 , the method including:
capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of a first hand;
incorporating the image of the first hand into a virtual reality scene that includes a pushable virtual object;
sensing a gesture of the first hand, the gesture made by unconstrained movement of the first hand in the three-dimensional (3D) sensory space and determining whether the gesture is intended to interact with the pushable virtual object by pushing the virtual object, the determining further including taking into account at least:
an angular relationship of a normal to a palm of the first hand to a proximate surface of the virtual object;
fingertip separations between the fingertips of the first hand and the proximate surface;
a hand posture, whether suitable for pushing the virtual object or incompatible with pushing; and
linear velocity of the palm of the first hand relative to the virtual object; and
responsive to determining that a gesture of the first hand is intended to push the virtual object, linking motion of the first hand to manipulation of the virtual object resulting in a displacement of the virtual object in the virtual space.
22. A method of manipulating virtual objects using real motions of one or more hands in a three-dimensional (3D) sensory space, the method including:
capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of the hands;
incorporating at least part of the image of the hands into a virtual reality scene;
outlining using a separate outline having an appearance contrasting the image of the hands in the virtual reality scene, a modeled position of the location of the hands and incorporating the outline into the virtual reality scene;
detecting that at least part of the hands is obscured by a virtual object in virtual reality scene; and
rendering one of the outline and the image but not both where the hands are obscured.
23. The method of claim 22 , further including changing an appearance of the outline as the part of the hands obscured changes.
24. The method of claim 22 , further including changing an appearance of the image of the hands as the part of the hands obscured changes.
25. A system implementing the method of claim 22 , the system including:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
capture an image of the hands in a three-dimensional (3D) sensory space and sense a location of the hands;
incorporate at least part of the image of the hands into a virtual reality scene;
outline in a contrasting color a modeled position of the location of the hands and incorporate the outline into the virtual reality scene;
detect that at least part of the hands is obscured by a virtual object in virtual reality scene; and
render one of the outline and the image but not both where the hands are obscured.
26. One or more non-transitory computer readable media having instructions stored thereon for performing the method of claim 22 , the method including:
capturing an image of the hands in a three-dimensional (3D) sensory space and sensing a location of the hands;
incorporating at least part of the image of the hands into a virtual reality scene;
outlining in a contrasting color a modeled position of the location of the hands and incorporating the outline into the virtual reality scene;
detecting that at least part of the hands is obscured by a virtual object in virtual reality scene; and
rendering one of the outline and the image but not both where the hands are obscured.