Spatially consistent representation of hand motion
Examples are disclosed that relate to representing recorded hand motion. One example provides a computing device comprising a logic subsystem and a storage subsystem comprising instructions executable by the logic subsystem to receive a recorded representation of hand motion determined relative to a virtual model aligned to a first instance of an object, receive image data corresponding to an environment, and recognize a second instance of the object in the environment. The instructions are further executable to align the virtual model to the second instance of the object, and output a parametric representation of hand motion for display relative to the virtual model as aligned to the second instance of the object, such that the parametric representation is spatially consistent with the recorded representation of hand motion relative to the virtual model as aligned to the first instance of the object.
1. A computing device, comprising:
a logic subsystem; and
a storage subsystem comprising instructions executable by the logic subsystem to:
receive a recorded representation of hand motion with respect to a first object;
receive image data corresponding to an environment;
recognize a second object in the environment; and
based on the recorded representation of hand motion, output a parametric representation of hand motion for display with respect to the second object, such that the parametric representation of hand motion is spatially consistent with the recorded representation of hand motion relative to the first object, wherein the parametric representation includes a multi-dimensional vector, wherein the parametric representation is aligned with respect to the second object such that the display of the parametric representation is spatially consistent with the recorded representation of hand motion relative to the first object, wherein each dimension of the multi-dimensional vector encodes an articulation of at least one of a plurality of hand joints, and wherein the articulation of each hand joint of the plurality of hand joints is encoded in a coordinate system aligned to the second object.
2. The computing device of claim 1 , wherein the parametric representation of hand motion is output for display on a head-mounted display device.
3. The computing device of claim 1 , further comprising instructions executable to use the multi-dimensional vector to reproduce an overall pose of a hand.
4. The computing device of claim 1 , further comprising instructions executable to:
use the image data to obtain a virtual model corresponding to the second object; and
use the virtual model to confirm a presence of the second object in the environment.
5. The computing device of claim 1 , further comprising instructions executable to convert a non-parametric representation of hand motion to the parametric representation of hand motion.
6. The computing device of claim 5 , wherein the non-parametric representation of hand motion comprises a geometric representation of hand motion.
7. A computing device, comprising:
a logic subsystem; and
a storage subsystem comprising instructions executable by the logic subsystem to:
receive image data corresponding to an environment;
recognize a first object in the environment;
receive a recording of hand motion;
based on the recording, determine a parametric representation of hand motion with respect to the first object; and
configure the parametric representation of hand motion for display with respect to a second object, such that the parametric representation of hand motion is spatially consistent with the parametric representation of hand motion relative to the first object, wherein the parametric representation includes a multi-dimensional vector, wherein the parametric representation is aligned with respect to the second object such that the display of the parametric representation is spatially consistent with the recording of the hand motion relative to the first object, wherein each dimension of the multi-dimensional vector encodes an articulation of at least one of a plurality of hand joints, and wherein the articulation of each hand joint of the plurality of hand joints is encoded in a coordinate system aligned to the second object.
8. The computing device of claim 7 , further comprising instructions executable to output the parametric representation of hand motion for display at another computing device.
9. The computing device of claim 7 , further comprising instructions executable to use the multi-dimensional vector to reproduce an overall pose of a hand.
10. The computing device of claim 7 , further comprising instructions executable to:
use the image data to obtain a virtual model corresponding to the first object; and
use the virtual model to confirm a presence of the first object in the environment.
11. The computing device of claim 7 , further comprising instructions executable to convert a non-parametric representation of hand motion to the parametric representation of hand motion.
12. The computing device of claim 11 , wherein the non-parametric representation of hand motion comprises a geometric representation of hand motion.
13. At a robotic device, a method of controlling a robot manipulator, comprising:
receiving a parametric representation of hand motion determined with respect to a first object;
receiving image data corresponding to an environment;
recognizing a second object in the environment; and
based on the parametric representation of hand motion, determining a sequence of actions for performance, relative to the second object, by a manipulator of the robotic device, where the sequence of actions is spatially consistent with the parametric representation of hand motion relative to the first object, wherein the parametric representation includes a multi-dimensional vector, wherein the parametric representation is aligned with respect to the second object, and wherein each dimension of the multi-dimensional vector encodes an articulation of at least one of a plurality of hand joints, and wherein the articulation of each hand joint of the plurality of hand joints is encoded in a coordinate system aligned to the second object.
14. The method of claim 13 , further comprising, for each action of the sequence of actions, generating one or more corresponding commands configured to cause the manipulator to perform the action.
15. The method of claim 14 , further comprising updating the one or more commands based on the image data to thereby align the manipulator to the second object.
16. The method of claim 13 , wherein determining the sequence of actions comprises translating human hand motion into motion of the manipulator.
17. The method of claim 13 , wherein a number of joints of the manipulator differs from a number of the hand joints encoded in the parametric representation, and wherein determining the sequence of actions comprises transforming one or more articulations encoded in the parametric representation to reproduce an overall pose of a hand.
18. The method of claim 13 , further comprising using a predetermined vocabulary to translate the parametric representation into the sequence of actions.
19. The method of claim 13 , further comprising inputting the parametric representation into a neural network or a support vector machine trained on a predetermined vocabulary of actions to thereby classify the parametric representation based upon the predetermined vocabulary of actions; and
converting the classification of the parametric representation into the sequence of actions for the manipulator.
20. The method of claim 13 , further comprising generating feedback based upon the image data to thereby update one or more control commands issued to the manipulator.