Intelligent stylus beam and assisted probabilistic input to element mapping in 2D and 3D graphical user interfaces

1. A non-transitory computer readable memory medium storing programming instructions executable by a processor of a display system to:

receive a first user input via a user input device, wherein the first user input is an attempt to select a virtual object rendered in a virtual scene on a display of the display system, wherein the virtual scene comprises a plurality of virtual objects;

in response to the first user input, determine a position and orientation of the user input device;

perform, based on the position and orientation of the user input device, a ray-cast procedure and a sphere-cast procedure to determine the virtual object being selected;

compare results of the ray-cast procedure and the sphere-cast procedure, wherein, when the results differ, ray-cast results are prioritized over sphere-cast results, and wherein, when the results do not differ, the sphere-cast results are prioritized over the ray-cast results; and

render a virtual beam from the user input device to the virtual object based on the comparison, wherein when the ray-cast results are used, the virtual beam is straight and rigid, and wherein when the sphere-cast results are used, the virtual beam is non-rigid and curved.

2. The non-transitory computer readable memory medium of claim 1 ,

wherein, to perform the ray-cast procedure, the programming instructions are further executable to:

determine a starting location and direction of a ray based on the position and orientation of the user input device;

determine one or more hit points, wherein each hit point of the one or more hit points includes an intersection between the ray and a respective virtual object of the plurality of virtual objects of the virtual scene; and

prioritize the one or more hit points based on distances from each intersection to the starting location of the ray.

3. The non-transitory computer readable memory medium of claim 1 ,

wherein, to perform the sphere-cast procedure, the programming instructions are further executable to:

determine a starting location and direction of a reference ray based on the position and orientation of the user input device and a radius of a sphere moving along the reference ray;

determine one or more hit points, wherein each hit point of the one or more hit points includes an intersection between the sphere and a respective virtual object of the plurality of virtual objects of the virtual scene; and

prioritize the one or more hit points based on distances from each intersection to the starting location of the reference ray.

4. The non-transitory computer readable memory medium of claim 3 ,

wherein to determine the one or more hit points, the programming instructions are further executable to:

upon detection of at least one intersection, temporarily increasing a hit radius by a specified percentage of an originally specified value of the radius; and

upon determining that the at least one intersection no longer exists, returning the hit radius to the originally specified value of the radius.

5. The non-transitory computer readable memory medium of claim 1 ,

wherein the programming instructions are further executable to:

in response to determining that the ray-cast procedure has failed in identifying an intersection, prioritizing the sphere-cast results; and

in response to determining that the sphere-cast procedure has failed in identifying an intersection, prioritizing the ray-cast results.

6. The non-transitory computer readable memory medium of claim 1 ,

wherein, to determine the position and orientation of the user input device, the programming instructions are further executable to:

track, via inputs received from a tracking system of the display system, the position and orientation of the user input device, wherein the user input device includes one or more visually indicated points.

7. The non-transitory computer readable memory medium of claim 1 ,

wherein the position and orientation of the user input device is relative to the display.

8. The non-transitory computer readable memory medium of claim 1 ,

wherein the programming instructions are further executable to:

determine a plurality of inputs to a probability model associated with selection of each virtual object of the plurality of virtual objects, wherein the plurality of inputs comprise one or more of a user head state, a user eye state, a user input device state, a previous output of the probability model, a virtual object state, a user attribute, and an output from a neighboring device in communication with the display system;

calculate, for each virtual object of the plurality of virtual objects, a probability that a user intends to select that virtual object using the probability model; and

determine, responsive to the first user input, an intended virtual object of the plurality of virtual objects based on the calculated probabilities.

9. The non-transitory computer readable memory medium of claim 1 ,

wherein the programming instructions are further executable to:

monitor an amount of time the virtual object is selected; and

in response to the amount of time exceeding a threshold, designate the virtual object as a priority object, wherein selection of the priority object is prioritized over selection of a non-prioritized object, thereby easing subsequent selection of the virtual object.

10. The non-transitory computer readable memory medium of claim 1 ,

wherein the programming instructions are further executable to:

determine that multiple objects within the virtual scene are moving;

determine, based on one of the ray-cast results or the sphere-cast results, a moving object a user intends to select; and

snap the virtual beam to the moving object.

11. A three-dimensional (3D) stereoscopic display system, comprising:

at least one processor;

one or more displays, coupled to the at least one processor;

a tracking system comprising two or more cameras and in communication with the at least one processor; and

a memory in communication with the tracking system and the at least one processor, wherein the at least one processor is configured to:

receive a first user input via a user input device, wherein the first user input is an attempt to select a virtual object rendered in a virtual scene on the one or more displays, wherein the virtual scene comprises a plurality of virtual objects;

in response to the first user input, determine a position and orientation of the user input device;

perform, based on the position and orientation of the user input device, a ray-cast procedure and a sphere-cast procedure to determine the virtual object being selected;

compare results of the ray-cast procedure and the sphere-cast procedure, wherein, when the results differ, ray-cast results are prioritized over sphere-cast results, and wherein, when the results do not differ, the sphere-cast results are prioritized over the ray-cast results; and

render a virtual beam from the user input device to the virtual object based on the comparison, wherein when the ray-cast results are used, the virtual beam is straight and rigid, and wherein when the sphere-cast results are used, the virtual beam is non-rigid and curved.

12. The 3D stereoscopic display system of claim 11 ,

wherein the ray-cast procedure and the sphere-cast procedure each include a configurable hit radius.

13. The 3D stereoscopic display system of claim 11 ,

wherein the at least one processor is further configured to:

in response to determining that the ray-cast procedure has failed in identifying an intersection, prioritizing the sphere-cast results; and

in response to determining that the sphere-cast procedure has failed in identifying an intersection, prioritizing the ray-cast results.

14. The 3D stereoscopic display system of claim 11 ,

wherein, to determine the position and orientation of the user input device, wherein the at least one processor is further configured to:

track, via the tracking system, the position and orientation of the user input device, wherein the user input device includes one or more visually indicated points; and

receive, from the user input device, information associated with the position and orientation of the user input device.

15. A wireless device, comprising:

at least one antenna;

at least one radio device in communication with the at least one antenna and configured to perform wireless communication using at least one radio access technology (RAT);

one or more processors coupled to the at least one radio device;

a tracking system comprising two or more cameras and in communication with the one or more processors; and

at least one display in communication with the one or more processors;

wherein the one or more processors are configured to cause the wireless device to:

receive a first user input via a user input device, wherein the first user input is an attempt to select a virtual object rendered in a virtual scene on the at least one display, wherein the virtual scene comprises a plurality of virtual objects;

in response to the first user input, determine a position and orientation of the user input device;

perform, based on the position and orientation of the user input device, a ray-cast procedure and a sphere-cast procedure to determine the virtual object being selected;

compare results of the ray-cast procedure and the sphere-cast procedure, wherein, when the results differ, ray-cast results are prioritized over sphere-cast results, and wherein, when the results do not differ, the sphere-cast results are prioritized over the ray-cast results; and

render a virtual beam from the user input device to the virtual object based on the comparison, wherein when the ray-cast results are used, the virtual beam is straight and rigid, and wherein when the sphere-cast results are used, the virtual beam is non-rigid and curved.

16. The wireless device of claim 15 , wherein the one or more processors are further configured to cause the wireless device to:

determine a plurality of inputs to a probability model associated with selection of each virtual object of the plurality of virtual objects;

calculate, for each virtual object of the plurality of virtual objects, a probability that a user intends to select that virtual object using the probability model; and

determine, responsive to the first user input, an intended virtual object of the plurality of virtual objects based on the calculated probabilities.

17. The wireless device of claim 16 , wherein the plurality of inputs include one or more of a user head state, a user eye state, a user input device state, a previous output of the probability model, a virtual object state, a user attribute, and an output from a neighboring device in communication with the wireless device.

18. The wireless device of claim 17 , wherein the output from the neighboring device in communication with the wireless device includes at least one of:

a confidence value of the tracking system;

data from a personal electronic device of the user; or

data from a computer vision system.

19. The wireless device of claim 16 , wherein the probability model is trained based, at least in part, on one or more user sessions, wherein training the probability model includes customizing the probability model to a plurality of particular users, wherein the plurality of particular users are associated with one or more user attributes; and wherein the probability model is based, at least in part, on one or more of:

a neural network computational model;

a recurrent neural network computational model;

an oscillating neural network computational model;

a Bayesian network computational model;

a dynamic Bayesian network computational model;

a decision tree computational model;

a decision forest computational model;

a hidden Markov model computational model;

a Gaussian mixture model computational model; and

a Markov random field computational model.

20. The wireless device of claim 17 , wherein the user head state includes a position, an orientation, and a velocity of the user's head, wherein the user eye state includes a gaze direction and a velocity of the user's eye gaze, wherein the user input device state includes the position, the orientation, and a velocity of the user input device, wherein the previous output of the probability model includes a previous calculated probability for each of the plurality of virtual objects, and wherein the virtual object state includes a position and velocity of each of the plurality of virtual objects within the virtual scene.