Multimodal kinematic template matching and regression modeling for ray pointing prediction in virtual reality
An electronic device tracks, for a user performing a target acquisition movement within a 3D space, movement parameters of a plurality of input devices of the user. The electronic device predicts, for the user, a region of interest within the 3D space, using a regression model, based on the movement parameters. The region of interest includes a plurality of targets in close proximity. The electronic device predicts an endpoint of the target acquisition movement, within the region of interest, using a pointer facilitation technique. In some embodiments, the plurality of input devices includes an eye tracking input device, each input device corresponds to a predefined input device type, and the movement parameters include gaze data from the eye tracking input device. In some embodiments, input devices includes an eye tracking input device, a head-mounted display, and a hand-held controller, and the user's eye, hand, and head movements are coordinated.
1. A method of predicting future positions and directions of one or more input devices in 3D spaces, comprising:
at an electronic device having a display, one or more processors, and memory storing one or more programs having instructions for:
tracking, for a user performing a target acquisition movement within a 3D space, movement parameters of a plurality of input devices of the user;
predicting, for the user, a region of interest within the 3D space, using a regression model, based on the movement parameters, wherein the region of interest includes a plurality of targets in close proximity; and
predicting an endpoint of the target acquisition movement, within the region of interest, using a pointer facilitation technique.
2. The method of claim 1 , wherein the plurality of input devices includes an eye tracking input device, each input device corresponds to a predefined input device type, and the movement parameters include gaze data from the eye tracking input device.
3. The method of claim 1 , wherein the regression model represents coordination patterns between input channels of the plurality of input devices.
4. The method of claim 3 , wherein the plurality of input devices includes an eye tracking input device, a head-mounted display (HMD), and a hand-held controller, and the coordination patterns describe coordination between eye, hand, and head movements of the user.
5. The method of claim 4 , wherein the movement parameters include velocity profiles for the HMD and the hand-held controller, and a saccade velocity profile for the eye tracking device.
6. The method of claim 1 , wherein predicting the endpoint of the target acquisition movement is performed by biasing the pointer facilitation technique to predict a goal target ray towards the region of interest.
7. The method of claim 6 , wherein the biasing is performed during an early ballistic phase of the pointer facilitation technique corresponding to a ballistic trajectory of eye, head, and hand movement of the user when moving from one target to the next.
8. The method of claim 1 , wherein the regression model is trained based on collecting a set of movement parameters for the plurality of input devices for a plurality of users performing one or more target acquisition movements.
9. The method of claim 1 , further comprising:
selecting a candidate target from the plurality of targets based on predefined probabilities for presence of the plurality of targets in the 3D space.
10. The method of claim 1 , further comprising:
dynamically adapting a control-display (C-D) ratio based on predicting the region of interest within the 3D space and/or the endpoint of the target acquisition movement.
11. The method of claim 1 , further comprising:
predicting likelihood of targets to snap to a closest target in the region of interest.
12. The method of claim 1 , wherein the plurality of input devices includes an eye tracking input device, and the movement parameters includes gaze data from the eye tracking input device, the method further comprising:
predicting object depth for one or more targets within the region of interest based on the gaze data.
13. An electronic device comprising:
a display;
one or more processors; and
memory storing one or more programs having instructions for:
tracking, for a user performing a target acquisition movement within a 3D space, movement parameters of a plurality of input devices of the user;
predicting, for the user, a region of interest within the 3D space, using a regression model, based on the movement parameters, wherein the region of interest includes a plurality of targets in close proximity; and
predicting an endpoint of the target acquisition movement, within the region of interest, using a pointer facilitation technique.
14. The electronic device of claim 13 , wherein the plurality of input devices includes an eye tracking input device, each input device corresponds to a predefined input device type, and the movement parameters include gaze data from the eye tracking input device.
15. The electronic device of claim 13 , wherein the regression model represents coordination patterns between input channels of the plurality of input devices.
16. The electronic device of claim 15 , wherein the plurality of input devices includes an eye tracking input device, a head-mounted display (HMD), and a hand-held controller, and the coordination patterns describe coordination between eye, hand, and head movements of the user.
17. The electronic device of claim 16 , wherein the movement parameters include velocity profiles for the HMD and the hand-held controller, and a saccade velocity profile for the eye tracking device.
18. The electronic device of claim 13 , wherein predicting the endpoint of the target acquisition movement is performed by biasing the pointer facilitation technique to predict a goal target ray towards the region of interest.
19. The electronic device of claim 18 , wherein the biasing is performed during an early ballistic phase of the pointer facilitation technique corresponding to a ballistic trajectory of eye, head, and hand movement of the user when moving from one target to the next.
20. The electronic device of claim 13 , further comprising:
selecting a candidate target from the plurality of targets based on predefined probabilities for presence of the plurality of targets in the 3D space.