Transmodal input fusion for a wearable system
Examples of wearable systems and methods can use multiple inputs (e.g., gesture, head pose, eye gaze, voice, totem, and/or environmental factors (e.g., location)) to determine a command that should be executed and objects in the three-dimensional (3D) environment that should be operated on. The wearable system can detect when different inputs converge together, such as when a user seeks to select a virtual object using multiple inputs such as eye gaze, head pose, hand gesture, and totem input. Upon detecting an input convergence, the wearable system can perform a transmodal filtering scheme that leverages the converged inputs to assist in properly interpreting what command the user is providing or what object the user is targeting.
1 . A method comprising:
under control of a hardware processor of a wearable system:
identifying a current transmodal state, the current transmodal state comprising a transmodal vergence associated with an object, wherein identifying the current transmodal state comprises analyzing convergence among a plurality of input targeting vectors, and wherein analyzing convergence comprises determining an angular distance between pairs of the plurality of input target vectors;
identifying a region of intent (ROI) associated with the transmodal vergence;
identifying a corresponding interaction field based at least partly on the ROI;
selecting an input fusion method based at least partly on the current transmodal state;
selecting settings for a primary targeting vector;
applying conditioning to the primary targeting vector to provide us stabilized pose vector;
communicating the stabilized pose vector to an application;
determining a relative variance between each pair of the plurality of input target vectors;
determining that an angular distance of a pair of input target vectors is below a first threshold and the relative variance of the pair of input target vectors is below a second threshold; and
in response to the determining, identifying the current transmodal state as a bimodal state associated with the pair of input target vectors.
2 . The method of claim 1 , wherein the corresponding interaction field comprises one or more of: a near field, a midfield, or a far field.
3 . The method of claim 1 , wherein applying conditioning comprises reducing registration error, jitter, or drift of the primary targeting vector.
4 . The method of claim 1 , further comprising:
targeting the object.
5 . The method of claim 1 , wherein identifying the current transmodal state comprises determining a fixation or a dwell.
6 . The method of claim 5 , further comprising:
determining if the fixation or the dwell exceeds a user focus threshold; and
in response to a determination that the fixation or the dwell exceeds the user focus threshold, activating microgesture manipulations.
7 . The method of claim 1 , wherein identifying the corresponding interaction field comprises identifying a field transition event, the field transition event comprising a transition between a first interaction field and a second interaction field.
8 . A method comprising:
under control of a hardware processor of a wearable system:
identifying a current transmodal state, the current transmodal state comprising a transmodal vergence associated with an object, wherein identifying the current transmodal state comprises analyzing convergence among a plurality of input targeting vectors, and wherein analyzing convergence comprises determining that a triplet of input target vectors is associated with a transmodal triangle having an area and three sides;
identifying a region of intent (ROI) associated with the transmodal vergence;
identifying a corresponding interaction field based at least partly on the ROI;
selecting an input fusion method based at least partly on the current transmodal state;
selecting settings for a primary targeting vector;
applying conditioning to the primary targeting vector to provide us stabilized pose vector;
communicating the stabilized pose vector to an application;
determining that the area of the transmodal triangle is below a third threshold, a variance in the area is below a fourth threshold or variances in lengths of the sides of the transmodal triangle are below a fifth threshold; and
in response to the determining, identifying the current transmodal state as a trimodal state associated with the triplet of input target vectors.
9 . The method of claim 1 , wherein the current transmodal state comprises a bimodal state, a trimodal state, or a quadmodal state.
10 . A wearable system comprising:
a plurality of sensors of different modalities; and
a hardware processor programmed to:
identify a current transmodal state, the current transmodal state comprising a transmodal vergence associated with an object;
identify a region of intent (ROI) associated with the transmodal vergence;
identify a corresponding interaction field based at least partly on the ROI;
select an input fusion method based at least partly on the current transmodal state;
select settings for a primary targeting vector;
apply conditioning to the primary targeting vector to provide us stabilized pose vector;
communicate the stabilized pose vector to an application;
identify the current transmodal state by determining if a fixation or a dwell exceeds a user focus threshold; and
in response to a determination that the fixation or the dwell exceeds the user focus threshold, activate microgesture manipulations.
11 . The wearable system of claim 10 , wherein the hardware processor is programmed to:
identify the corresponding interaction field by identifying a field transition event, the field transition event comprising a transition between a first interaction field and a second interaction field.
12 . The wearable system of claim 10 , wherein the hardware processor is programmed to:
identify the current transmodal state by analyzing convergence among a plurality of input targeting vectors.
13 . The wearable system of claim 12 , wherein the hardware processor is programmed to:
analyze convergence by determining an angular distance between pairs of the plurality of input target vectors.
14 . The wearable system of claim 10 , wherein the hardware processor is programmed to:
analyze convergence by determining that a triplet of input target vectors is associated with a transmodal triangle having an area and three sides;
determine that the area of the transmodal triangle is below a third threshold, a variance in the area is below a fourth threshold or variances in lengths of the sides of the transmodal triangle are below a fifth threshold; and
in response to the area of the transmodal triangle being determined to be below a third threshold, identify the current transmodal state as a trimodal state associated with the triplet of input target vectors.