Intelligent stylus beam and assisted probabilistic input to element mapping in 2D and 3D graphical user interfaces
Systems and methods for implementing methods for user selection of a virtual object in a virtual scene. A user input may be received via a user input device. The user input may be an attempt to select a virtual object from a plurality of virtual objects rendered in a virtual scene on a display of a display system. A position and orientation of the user input device may be determined in response to the first user input. A probability the user input may select each virtual object may be calculated via a probability model. Based on the position and orientation of the user input device, a ray-cast procedure and a sphere-cast procedure may be performed to determine the virtual object being selected. The probability of selection may also be considered in determining the virtual object. A virtual beam may be rendered from the user input device to the virtual object.
1. A non-transitory computer readable memory medium storing programming instructions executable by a processor of a display system to:
render a plurality of virtual objects in a virtual scene displayed on at least one display of the display system;
determine a plurality of inputs to a probability model associated with selection of each virtual object of the plurality of virtual objects, wherein the plurality of inputs comprise a virtual object state and one or more of a user head state, a user eye state, a user input device state, a previous output of the probability model, a user attribute, or an output from a device in communication with the display system, and wherein the virtual object state includes a position and velocity of each of the plurality of virtual objects within the virtual scene;
calculate, for each virtual object of the plurality of virtual objects, a probability that a user intends to select that virtual object using the probability model; and
determine, responsive to a user input via a user input device, an intended virtual object of the plurality of virtual objects based on the calculated probabilities.
2. The non-transitory computer readable memory medium of claim 1 ,
wherein the output from the device in communication with the display system includes at least one of:
a confidence value of a tracking system;
data from a personal electronic device of the user; or
data from a computer vision system.
3. The non-transitory computer readable memory medium of claim 1 ,
wherein the probability model is trained based, at least in part, on one or more user sessions.
4. The non-transitory computer readable memory medium of claim 3 ,
wherein training the probability model includes customizing the probability model to a plurality of particular users, and wherein the plurality of particular users are associated with one or more user attributes.
5. The non-transitory computer readable memory medium of claim 1 ,
wherein the probability model is based, at least in part, on one or more of:
a neural network computational model;
a recurrent neural network computational model;
an oscillating neural network computational model;
a Bayesian network computational model;
a dynamic Bayesian network computational model;
a decision tree computational model;
a decision forest computational model;
a hidden Markov model computational model;
a Gaussian mixture model computational model; or
a Markov random field computational model.
6. The non-transitory computer readable memory medium of claim 1 ,
wherein the user head state includes a position, an orientation, and a velocity of the user's head.
7. The non-transitory computer readable memory medium of claim 1 ,
wherein the user eye state includes a gaze direction and a velocity of the user's eye gaze.
8. The non-transitory computer readable memory medium of claim 1 ,
wherein the user input device state includes a position, an orientation, and a velocity of the user input device.
9. The non-transitory computer readable memory medium of claim 1 ,
wherein the previous output of the probability model includes a previous calculated probability for each of the plurality of virtual objects.
10. A three-dimensional (3D) stereoscopic display system, comprising:
at least one processor;
one or more displays, coupled to the at least one processor;
a tracking system comprising two or more cameras and in communication with the at least one processor; and
a memory in communication with the tracking system and the at least one processor, wherein the at least one processor is configured to:
render a plurality of virtual objects in a virtual scene displayed on the one or more displays;
determine a plurality of inputs to a probability model associated with selection of each virtual object of the plurality of virtual objects, wherein the plurality of inputs comprises a virtual object state, a user attribute, and one or more of a user head state, a user eye state, a user input device state, a previous output of the probability model, or an output from a device in communication with the 3D stereoscopic display system, and wherein the virtual object state includes a position and velocity of each of the plurality of virtual objects within the virtual scene;
calculate, for each virtual object of the plurality of virtual objects, a probability that a user intends to select that virtual object using the probability model; and
determine, responsive to a user input via a user input device, an intended virtual object of the plurality of virtual objects based on the calculated probabilities.
11. The 3D stereoscopic display system of claim 10 ,
wherein the output from the device in communication with the 3D stereoscopic display system includes at least one of:
a confidence value of the tracking system;
data from a personal electronic device of the user; or
data from a computer vision system.
12. The 3D stereoscopic display system of claim 10 ,
wherein the probability model is trained based, at least in part, on one or more user sessions, wherein training the probability model includes customizing the probability model to a plurality of particular users, and wherein the plurality of particular users are associated with one or more user attributes.
13. The 3D stereoscopic display system of claim 10 ,
wherein the probability model is based, at least in part, on one or more of:
a neural network computational model;
a recurrent neural network computational model;
an oscillating neural network computational model;
a Bayesian network computational model;
a dynamic Bayesian network computational model;
a decision tree computational model;
a decision forest computational model;
a hidden Markov model computational model;
a Gaussian mixture model computational model; or
a Markov random field computational model.
14. The 3D stereoscopic display system of claim 10 ,
wherein the user head state includes a position, an orientation, and a velocity of the user's head, wherein the user eye state includes a gaze direction and a velocity of the user's eye gaze, and wherein the user input device state includes a position, an orientation, and a velocity of the user input device.
15. The 3D stereoscopic display system of claim 10 ,
wherein the at least one processor is further configured to:
compare results of a ray-cast procedure and a sphere-cast procedure performed in response to the user input and based on a determined position and orientation of the user input device, wherein, when the results differ, ray-cast results are prioritized over sphere-cast results, and wherein, when the results do not differ, the sphere-cast results are prioritized over the ray-cast results; and
render a virtual beam from the user input device to the intended virtual object based on the comparison, wherein when ray-cast results are used, the virtual beam is straight and rigid, and wherein when sphere-cast results are used, the virtual beam is non-rigid and curves to the intended virtual object.
16. A wireless device, comprising:
at least one antenna;
at least one radio in communication with the at least one antenna and configured to perform wireless communication using at least one radio access technology (RAT);
one or more processors coupled to the at least one radio;
a tracking system comprising two or more cameras and in communication with the one or more processors; and
at least one display in communication with the one or more processors;
wherein the one or more processors are configured to cause the wireless device to:
render a plurality of virtual objects in a virtual scene displayed on the at least one display;
determine a plurality of inputs to a probability model associated with selection of each virtual object of the plurality of virtual objects, wherein the plurality of inputs comprise a virtual object state, an output from a device in communication with the wireless device, and one or more of a user head state, a user eye state, a user input device state, a previous output of the probability model, or a user attribute, and wherein the virtual object state includes a position and velocity of each of the plurality of virtual objects within the virtual scene;
calculate, for each virtual object of the plurality of virtual objects, a probability that a user intends to select that virtual object using the probability model; and
determine, responsive to a user input via a user input device, an intended virtual object of the plurality of virtual objects based on the calculated probabilities.
17. The wireless device of claim 16 ,
wherein the output from the device in communication with the wireless device includes at least one of:
a confidence value of the tracking system;
data from a personal electronic device of the user; or
data from a computer vision system.
18. The wireless device of claim 16 ,
wherein the probability model is trained based, at least in part, on one or more user sessions, wherein training the probability model includes customizing the probability model to a plurality of particular users, and wherein the plurality of particular users are associated with one or more user attributes.
19. The wireless device of claim 16 ,
wherein the probability model is based, at least in part, on one or more of:
a neural network computational model;
a recurrent neural network computational model;
an oscillating neural network computational model;
a Bayesian network computational model;
a dynamic Bayesian network computational model;
a decision tree computational model;
a decision forest computational model;
a hidden Markov model computational model;
a Gaussian mixture model computational model; or
a Markov random field computational model.
20. The wireless device of claim 16 ,
wherein the user head state includes a position, an orientation, and a velocity of the user's head, wherein the user eye state includes a gaze direction and a velocity of the user's eye gaze, wherein the user input device state includes a position, an orientation, and a velocity of the user input device, and wherein the previous output of the probability model includes a previous calculated probability for each of the plurality of virtual objects.