Predictive information for free space gesture control and communication
Free space machine interface and control can be facilitated by predictive entities useful in interpreting a control object's position and/or motion (including objects having one or more articulating members, i.e., humans and/or animals and/or machines). Predictive entities can be driven using motion information captured using image information or the equivalents. Predictive information can be improved applying techniques for correlating with information from observations.
1. A method of accurately capturing gestural motion of a control object in a three-dimensional (3D) sensory space, the method including:
determining observation information characterizing gestural motion of a control object in a three-dimensional (3D) sensory space from at least one image captured at time t0;
constructing a 3D solid model to represent the control object by fitting one or more 3D solid subcomponents to a construct of the control object defined by the observation information based on the image captured at time t0;
responsive to modifications in the observation information based on another image captured at time t1, wherein the control object moved between t0 and t1, improving conformance of the 3D solid model to the modifications in the observation information by:
interpolating multiple intermediate 3D solid model poses across time, which are interpolated well with other interpolations based on expected continuity in motion and deformation of the control object, the interpolated multiple 3D solid model poses including at least one of joint angle, pitch and yaw; and
using the 3D solid model as adjusted to control a machine.
2. The method of claim 1 , wherein fitting the one or more 3D solid subcomponents further includes fitting a set of closed curves to at least a portion of a construct of the control object.
3. The method of claim 2 , wherein closed curves include at least one of radial solids, capsuloids, spheres, ellipsoids, and hyperboloids.
4. The method of claim 2 , wherein fitting the one or more 3D solid subcomponents further includes fitting a contour and surface defined by a set of points at a fixed distance from a closest corresponding point on the contour.
5. The method of claim 2 , wherein fitting the one or more 3D solid subcomponents further includes fitting a set of points normal to points on a contour and a fixed distance therefrom.
6. The method of claim 2 , wherein fitting the one or more 3D solid subcomponents further includes selecting a point on a contour and projecting outward a set of points at a radius length from the point selected.
7. The method of claim 1 , wherein fitting the one or more 3D solid subcomponents further includes fitting a set of capsuloids to at least a portion of construct of the control object.
8. The method of claim 1 , wherein the control object is a hand and fitting the one or more 3D solid subcomponents further includes at least one of:
fitting capsuloids to finger portions of a construct of the control object; and
fitting radial solids to palm and/or wrist portions of the construct.
9. The method of claim 1 , wherein adjusting the 3D solid model further includes altering the 3D solid subcomponents to conform to at least one of length, width, orientation, and arrangement of portions of a construct of the control object.
10. The method of claim 9 , wherein altering the 3D solid subcomponents further includes applying a transformation matrix to a plurality of points on the 3D solid subcomponents.
11. The method of claim 9 , wherein altering the 3D solid subcomponents further includes determining a rotation matrix that provides a reduced root mean squared deviation (RMSD) between paired point sets on a construct of the control object and points sets on the 3D solid subcomponents.
12. The method of claim 1 , wherein determining the observation information further includes fitting a family of closed curves and positions of a plurality of two-dimensional (2D) cross-sectional portions of the control object to the image based at least on a location of at least one image capturing device.
13. The method of claim 1 , wherein constructing the 3D solid model further includes selecting a pre-determined 3D solid model from an object library based on characteristic parameters of the control object.
14. The method of claim 1 , further including improving a 3D solid model's representation of the gestural motion by:
detecting fits of 3D solid subcomponents with colliding subcomponents; and
fitting, to the construct, 3D solid subcomponents with least colliding subcomponents.
15. The method of claim 14 , wherein detection of the colliding subcomponents is based at least on identifying a subcomponent attribute incompatible with an adjacent subcomponent attribute.
16. The method of claim 15 , wherein the subcomponent attribute is at least one of orientation, angle, length, shape, behavior, total energy, structure, compression, deformation, shear, and torsion.
17. The method of claim 1 , wherein constructing the 3D solid model further includes determining the 3D solid subcomponents from physical characteristics of a type of control object being observed.
18. The method of claim 17 , wherein the control object is a hand and the physical characteristics of the hand include at least one of:
four fingers and a thumb of the hand;
a palm to which the fingers and the thumb are connected; and
positions and angles of the fingers and the thumb relative to each other and to the palm.
19. The method of claim 17 , wherein the control object is a tool and the physical characteristics of the tool include at least one of:
length of the tool;
width of the tool; and
pointing direction vector of the tool.
20. The method of claim 1 , wherein the control object is a hand and identified construct portions include at least one fingers, carpals, knuckles, palm, and wrist.
21. The method of claim 1 , further including improving a 3D solid model's representation of the gestural motion by:
detecting conflicting attributes between adjacent 3D solid subcomponents; and
fitting, to the construct, 3D solid subcomponents with least conflicting attributes.
22. The method of claim 21 , further including:
ranking the 3D solid subcomponents with conflicting attributes based on a degree of conflict; and
presenting the ranked 3D solid subcomponents for selection.
23. The method of claim 21 , wherein conflicting attributes include at least one of minima and maxima of intersection angles between the 3D solid subcomponents.
24. The method of claim 1 , further including:
repeatedly applying the method of claim 1 over time; and
determining gestural motion of the control object based on differences between 3D solid model positions across time.
25. The method of claim 1 , wherein the improving of the conformance of the 3D solid model to the modifications in the observation information further includes:
interpolating multiple intermediate 3D solid model velocities across time that interpolate well with other interpolations based on expected continuity in motion.
26. The method of claim 1 , wherein the improving of the conformance of the 3D solid model to the modifications in the observation information further includes:
discarding an intermediate 3D solid model pose that does not interpolate well with other interpolations based on correspondence to the other interpolations.
27. A non-transitory computer readable storage medium impressed with computer program instructions to accurately capture gestural motion of a control object in a three-dimensional (3D) sensory space, which computer program instructions, when executed on a processor, implement a method including:
determining observation information characterizing gestural motion of a control object in a three-dimensional (3D) sensory space from at least one image captured at time t0;
constructing a 3D solid model to represent the control object by fitting one or more 3D solid subcomponents to a construct of the control object defined by the observation information based on the image captured at time t0;
responsive to modifications in the observation information based on another image captured at time t1, wherein the control object moved between t0 and t1, improving conformance of the 3D solid model to the modifications in the observation information by:
interpolating multiple intermediate 3D solid model poses across time, which are interpolated well with other interpolations based on expected continuity in motion and deformation of the control object, the interpolated multiple 3D solid model poses including at least one of joint angle, pitch and yaw; and
using the 3D solid model as adjusted to control a machine.
28. A system to accurately capture gestural motion of a control object in a three-dimensional (3D) sensory space, comprising:
a processor and a computer readable storage medium storing computer instructions configured to cause the processor to:
determine observation information characterizing gestural motion of a control object in a three-dimensional (3D) sensory space from at least one image captured at time t0;
construct a 3D solid model to represent the control object by fitting one or more 3D solid subcomponents to a construct of the control object defined by the observation information based on the image captured at time t0;
responsive to modifications in the observation information based on another image captured at time t1, wherein the control object moved between t0 and t1, improve conformance of the 3D solid model constructed to the modifications in the observation information by:
interpolating multiple intermediate 3D solid model poses across time, which are interpolated well with other interpolations based on expected continuity in motion and deformation of the control object, the interpolated multiple 3D solid model poses including at least one of joint angle, pitch and yaw; and
using the 3D solid model as adjusted to control a machine.