Systems and methods of rerendering image hands to create a realistic grab experience in virtual reality/augmented reality environments
The technology disclosed relates to a method of realistic rendering of a real object as a virtual object in a virtual space using an offset in the position of the hand in a three-dimensional (3D) sensory space. An offset between expected positions of the eye(s) of a wearer of a head mounted device and a sensor attached to the head mounted device for sensing a position of at least one hand in a three-dimensional (3D) sensory space is determined. A position of the hand in the three-dimensional (3D) sensory space can be sensed using a sensor. The sensed position of the hand can be transformed by the offset into a re-rendered position of the hand as would appear to the wearer of the head mounted device if the wearer were looking at the actual hand. The re-rendered hand can be depicted to the wearer of the head mounted device.
1. A system of realistic displacement of a virtual object to render a realistic representation of a hand in a three-dimensional (3D) sensory space as a virtual object in a virtual space, the system including:
a head mounted device depicting at least a visual presentation to a wearer;
a hand-gesture sensor attached to the head mounted device; and
a processor and a computer readable storage medium storing computer instructions configured for performing:
determining an offset between expected positions of one or more eyes of a wearer of a head mounted device and an actual position of a hand-gesture sensor attached to the head mounted device;
using the hand-gesture sensor, obtaining a sequence of images correlated to construct a 3-D solid model of the hand, including its position and shape, and using the 3-D solid model of the hand computing a sensed position of hand in a three-dimensional (3D) sensory space, wherein the computing includes determining positions of fingers, thumb and palm of the hand;
calculating a transformation matrix using the determined offset, such that the transformation matrix, when applied to the sensed position of the hand, decreases a rendered size of the hand in accordance with the determined offset;
applying the transformation matrix to the sensed position of the hand to transform the sensed position of the hand into a re-rendered position of the hand, the re-rendered position of the hand placing in the 3D sensory space the positions of the fingers, thumb and palm of the re-rendered position of the hand at the offset with respect to the sensed position of the hand such that the wearer of the head mounted device perceives the re-rendered position of the hand situated in the 3D sensory space as expected by the wearer when the wearer is looking at the actual hand in the 3D sensory space; and
depicting to the wearer of the head mounted device a re-rendered image hand using the re-rendered position of the hand.
2. The system of claim 1 , further configured to perform:
sensing one or more additional positions of the hand and determining therefrom motion or gesture; and
applying the offset to the additional positions of the hand to transform the sensed additional positions to re-rendered positions of the hand as seen by the wearer, thereby providing the re-rendered image hand having the motion or gesture of the hand.
3. The system of claim 2 , further configured to perform:
sensing an indication of quality of tracking for the hand-gesture sensor, the indication of quality of tracking indicating whether the hand-gesture sensor is tracking position or motion of the hand; and
adding to the re-rendered image hand a visual indication corresponding to the indication of quality of tracking; thereby providing the wearer with a visual cue of how well the hand-gesture sensor is tracking the hand.
4. The system of claim 3 , wherein the indication of quality of tracking includes an indication that the hand-gesture sensor is able to recognize the hand in captured images.
5. The system of claim 4 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is able to recognize the hand in captured images.
6. The system of claim 4 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is unable to recognize the hand in captured images.
7. The system of claim 1 , wherein the transformation matrix is an affine matrix.
8. A method of realistic displacement of a virtual object to render a realistic representation of a hand in a three-dimensional (3D) sensory space as a virtual object in a virtual space, the method including:
determining an offset between expected positions of one or more eyes of a wearer of a head mounted device and an actual position of a hand-gesture sensor attached to the head mounted device;
using the hand-gesture sensor, obtaining a sequence of images correlated to construct a 3-D solid model of the hand, including its position and shape, and using the 3-D solid model of the hand computing a sensed position of at least one hand in a three-dimensional (3D) sensory space, wherein the computing includes determining positions of fingers, thumb and palm of the hand;
calculating a transformation matrix using the determined offset, such that the transformation matrix, when applied to the sensed position of the hand, decreases a rendered size of the hand in accordance with the determined offset;
applying the transformation matrix to the sensed position of the hand to transform the sensed position of the hand into a re-rendered position of the hand, the re-rendered position of the hand placing in the 3D sensory space the positions of the fingers, thumb and palm of the re-rendered position of the hand at the offset with respect to the sensed position of the hand such that the wearer of the head mounted device perceives the re-rendered position of the hand situated in the 3D sensory space as expected by the wearer when the wearer is looking at the actual hand in the 3D sensory space; and
depicting to the wearer of the head mounted device a re-rendered image hand using the re-rendered position of the hand.
9. The method of claim 8 , further including:
sensing one or more additional positions of the hand and determining therefrom motion or gesture; and
applying the offset to the additional positions of the hand to transform the sensed additional positions to re-rendered positions of the hand as seen by the wearer, thereby providing the re-rendered image hand having the motion or gesture of the hand.
10. The method of claim 9 , further including:
sensing an indication of quality of tracking for the hand-gesture sensor, the indication of quality of tracking indicating whether the hand-gesture sensor is tracking position or motion of the hand; and
adding to the re-rendered image hand a visual indication corresponding to the indication of quality of tracking; thereby providing the wearer with a visual cue of how well the hand-gesture sensor is tracking the hand.
11. The method of claim 10 , wherein the indication of quality of tracking includes an indication that the hand-gesture sensor is able to recognize the hand in captured images.
12. The method of claim 11 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is able to recognize the hand in captured images.
13. The method of claim 11 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is unable to recognize the hand in captured images.
14. One or more non-transitory computer readable media having instructions stored thereon for performing realistic displacement of a virtual object to render a realistic representation of a hand in a three-dimensional (3D) sensory space as a virtual object in a virtual space, the performing including:
determining an offset between expected positions of one or more eyes of a wearer of a head mounted device and an actual position of a hand-gesture sensor attached to the head mounted device;
using the hand-gesture sensor, obtaining a sequence of images correlated to construct a 3-D solid model of the hand, including its position and shape, and using the 3-D solid model of the hand computing a sensed position of at least one hand in a three-dimensional (3D) sensory space, wherein the computing includes determining positions of fingers, thumb and palm of the hand;
calculating a transformation matrix using the determined offset, such that the transformation matrix, when applied to the sensed position of the hand, decreases a rendered size of the hand in accordance with the determined offset;
applying the transformation matrix to the sensed position of the hand to transform the sensed position of the hand into a re-rendered position of the hand, the re-rendered position of the hand placing in the 3D sensory space the positions of the fingers, thumb and palm of the re-rendered position of the hand at the offset with respect to the sensed position of the hand such that the wearer of the head mounted device perceives the re-rendered position of the hand situated in the 3D sensory space as expected by the wearer when the wearer is looking at the actual hand in the 3D sensory space; and
depicting to the wearer of the head mounted device a re-rendered image hand using the re-rendered position of the hand.
15. The non-transitory computer readable media of claim 14 , the instructions further implementing performing:
sensing one or more additional positions of the hand and determining therefrom motion or gesture; and
applying the offset to the additional positions of the hand to transform the sensed additional positions to re-rendered positions of the hand as seen by the wearer, thereby providing the re-rendered image hand having the motion or gesture of the hand.
16. The non-transitory computer readable media of claim 15 , the instructions further implementing performing:
sensing an indication of quality of tracking for the hand-gesture sensor, the indication of quality of tracking indicating whether the hand-gesture sensor is tracking position or motion of the hand; and
adding to the re-rendered image hand a visual indication corresponding to the indication of quality of tracking; thereby providing the wearer with a visual cue of how well the hand-gesture sensor is tracking the hand.
17. The non-transitory computer readable media of claim 16 , wherein the indication of quality of tracking includes an indication that the hand-gesture sensor is able to recognize the hand in captured images.
18. The non-transitory computer readable media of claim 17 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is able to recognize the hand in captured images.
19. The non-transitory computer readable media of claim 17 , wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the re-rendered image hand when the hand-gesture sensor is unable to recognize the hand in captured images.
20. The non-transitory computer readable media of claim 17 , wherein the visual indication corresponding to the indication of quality of tracking includes coloring the re-rendered image hand a different color when the hand-gesture sensor is able to recognize the hand in captured images than when the hand-gesture sensor is unable to recognize the hand in captured images.
21. The non-transitory computer readable media of claim 15 , the instructions further implementing performing:
sensing an indication of whether the hand has grabbed a virtual object; and
adding to the re-rendered image hand a visual indication corresponding to the indication of whether the hand has grabbed the virtual object; thereby providing the wearer with a visual cue of whether the virtual object is grabbed.
22. The non-transitory computer readable media of claim 21 , the instructions further implementing performing: depicting a stretched version of the virtual object grabbed.