Transmodal input fusion for a wearable system
Examples of wearable systems and methods can use multiple inputs (e.g., gesture, head pose, eye gaze, voice, totem, and/or environmental factors (e.g., location)) to determine a command that should be executed and objects in the three-dimensional (3D) environment that should be operated on. The wearable system can detect when different inputs converge together, such as when a user seeks to select a virtual object using multiple inputs such as eye gaze, head pose, hand gesture, and totem input. Upon detecting an input convergence, the wearable system can perform a transmodal filtering scheme that leverages the converged inputs to assist in properly interpreting what command the user is providing or what object the user is targeting.
1. A wearable system comprising:
a head pose sensor configured to determine a head pose of a user of the wearable system;
an eye gaze sensor configured to determine an eye gaze direction of the user of the wearable system;
a gesture sensor configured to determine a hand gesture of the user of the wearable system;
a hardware processor in communication with the head pose sensor, the eye gaze sensor, and the gesture sensor, the hardware processor programmed to:
determine a first vergence between the eye gaze direction and the head pose of the user relative to an object, wherein to determine the first vergence, the hardware processor is programmed to determine that an angle between the eye gaze direction and a head pose direction associated with the head pose is less than a first threshold;
perform a first interaction command associated with the object based at least partly on inputs from the head pose sensor and the eye gaze sensor;
determine a second vergence of the hand gesture with the eye gaze direction and the head pose of the user relative to the object; and
perform a second interaction command associated with the object based at least partly on inputs from the hand gesture, the head pose sensor, and the eye gaze sensor.
2. The wearable system of claim 1 , wherein the head pose sensor comprises an inertial measurement unit (IMU), the eye gaze sensor comprise an eye-tracking camera, and the gesture sensor comprises an outward-facing camera.
3. The wearable system of claim 1 , wherein the first interaction command comprises targeting the object.
4. The wearable system of claim 1 , wherein the second interaction command comprises selecting the object.
5. The wearable system of claim 1 , wherein the hardware processor is further programmed to determine a divergence of at least one of the hand gesture, the eye gaze direction, or the head pose from the object.
6. The wearable system of claim 1 , wherein the first interaction command comprises applying a first filter or the second interaction command comprises applying a second filter.
7. The wearable system of claim 6 , wherein the first filter is different from the second filter.
8. The wearable system of claim 6 , wherein the first filter or the second filter comprises a low-pass filter having an adaptive cutoff frequency.
9. The wearable system of claim 8 , wherein the low-pass filter comprises a one euro filter.
10. The wearable system of claim 1 , wherein to determine the first vergence, the hardware processor is programmed to determine that a dwell time of the eye gaze direction and the head pose toward the object exceeds a first dwell time threshold.
11. The wearable system of claim 1 , wherein to determine the second vergence, the hardware processor is programmed to determine that a dwell time of the eye gaze direction, the head pose, and the hand gesture relative to the object exceeds a second dwell time threshold.
12. The wearable system of claim 1 , wherein the first interaction command or the second interaction command comprises providing a stabilized targeting vector associated with the object.
13. The wearable system of claim 12 , wherein the hardware processor provides the stabilized targeting vector to an application.
14. The wearable system of claim 1 , wherein the gesture sensor comprises a handheld user input device.
15. The wearable system of claim 14 , wherein the hardware processor is programmed to determine a third vergence between input from the user input device and at least one of the eye gaze direction, the head pose, or the hand gesture.
16. The wearable system of claim 1 , further comprising a voice sensor, and wherein the hardware processor is programmed to determine a fourth vergence between input from the voice sensor and at least one of the eye gaze direction, the head pose, or the hand gesture.
17. A wearable system comprising:
a head pose sensor configured to determine a head pose of a user of the wearable system;
an eye gaze sensor configured to determine an eye gaze direction of the user of the wearable system;
a gesture sensor configured to determine a hand gesture of the user of the wearable system;
a hardware processor in communication with the head pose sensor, the eye gaze sensor, and the gesture sensor, the hardware processor programmed to:
determine a first vergence between the eye gaze direction and the head pose of the user relative to an object;
perform a first interaction command associated with the object based at least partly on inputs from the head pose sensor and the eye gaze sensor;
determine a second vergence of the hand gesture with the eye gaze direction and the head pose of the user relative to the object, wherein to determine the second vergence, the hardware processor is programmed to determine that a transmodal triangle associated with the hand gesture, the eye gaze direction, and the head pose is less than a second threshold; and
perform a second interaction command associated with the object based at least partly on inputs from the hand gesture, the head pose sensor, and the eye gaze sensor.
18. The wearable system of claim 17 , wherein the head pose sensor comprises an inertial measurement unit (IMU), the eye gaze sensor comprise an eye-tracking camera, and the gesture sensor comprises an outward-facing camera.
19. The wearable system of claim 17 , wherein the first interaction command comprises targeting the object.
20. The wearable system of claim 17 , wherein the second interaction command comprises selecting the object.
21. The wearable system of claim 17 , wherein the hardware processor is further programmed to determine a divergence of at least one of the hand gesture, the eye gaze direction, or the head pose from the object.
22. The wearable system of claim 17 , wherein the first interaction command comprises applying a first filter or the second interaction command comprises applying a second filter.
23. The wearable system of claim 22 , wherein the first filter is different from the second filter.
24. The wearable system of claim 22 , wherein the first filter or the second filter comprises a low-pass filter having an adaptive cutoff frequency.
25. The wearable system of claim 24 , wherein the low-pass filter comprises a one euro filter.
26. The wearable system of claim 17 , wherein to determine the first vergence, the hardware processor is programmed to determine that a dwell time of the eye gaze direction and the head pose toward the object exceeds a first dwell time threshold.
27. The wearable system of claim 17 , wherein to determine the second vergence, the hardware processor is programmed to determine that a dwell time of the eye gaze direction, the head pose, and the hand gesture relative to the object exceeds a second dwell time threshold.
28. The wearable system of claim 17 , wherein the first interaction command or the second interaction command comprises providing a stabilized targeting vector associated with the object.
29. The wearable system of claim 28 , wherein the hardware processor provides the stabilized targeting vector to an application.
30. The wearable system of claim 17 , wherein the gesture sensor comprises a handheld user input device.
31. The wearable system of claim 30 , wherein the hardware processor is programmed to determine a third vergence between input from the user input device and at least one of the eye gaze direction, the head pose, or the hand gesture.
32. The wearable system of claim 17 , further comprising a voice sensor, and wherein the hardware processor is programmed to determine a fourth vergence between input from the voice sensor and at least one of the eye gaze direction, the head pose, or the hand gesture.