Devices, methods, systems, and media for selecting virtual objects for extended reality interaction
Devices, methods, systems, and media are described for selecting virtual objects for user interaction in an extended reality environment. Distant virtual objects are brought closer to the user within a virtual 3D space to situate the selected virtual object in virtual proximity to the user's hand for direct manipulation. A virtual object is selected by the user based on movements of the user's hand and/or head that are correlated or associated with an intent to select a specific virtual object within the virtual 3D space. As the user's hand moves in a way that is consistent with this intent, the virtual object is brought closer to the user's hand within the virtual 3D space. To predict the user's intent, hand and head trajectory data may be compared to a library of kinematic trajectory templates to identify a best-matched trajectory template.
1. A method for presenting a virtual object of an extended reality (XR) environment to a user, comprising:
generating initial display information configured to control an XR display to present the virtual object to the user at an initial virtual location;
obtaining, from one or more hand movement sensors, hand movement information indicating a velocity of the hand;
processing the hand movement information to generate trajectory data;
processing the trajectory data to determine that the user intends to select the virtual object; and
generating updated display information configured to control the XR display to present the virtual object to the user at an updated virtual location closer to the hand than the initial virtual location.
2. The method of claim 1 ,
further comprising obtaining, from one or more head movement sensors, head movement information indicating a velocity of the head of the user; and
wherein processing the hand movement information to generate the trajectory data comprises processing the hand movement information and the head movement information to generate the trajectory data.
3. The method of claim 2 , wherein:
generating the trajectory data comprises:
retrieving previous hand movement information from a memory;
storing the hand movement information obtained from the hand movement sensors in the memory;
retrieving previous head movement information from a memory;
storing the head movement information obtained from the head movement sensors in the memory; and
processing the previous hand movement information, the hand movement information obtained from the hand movement sensors, the previous head movement information, and the head movement information obtained from the head movement sensors to generate the trajectory data.
4. The method of claim 2 , wherein:
the hand movement information indicates a linear velocity and an angular velocity of the hand; and
the head movement information indicates a linear velocity and an angular velocity of the head.
5. The method of claim 4 , wherein:
the one or more hand movement sensors comprise an inertial measurement unit coupled to the hand; and
the one or more head movement sensors comprise an inertial measurement unit coupled to the head.
6. The method of claim 5 , wherein:
the inertial measurement unit coupled to the hand is an inertial measurement unit of a smart watch worn on the wrist of the hand; and
the inertial measurement unit coupled to the head is an inertial measurement unit of a head-mounted display device worn on the head, the head-mounted display device comprising the XR display.
7. The method of claim 2 , wherein processing the trajectory data to determine that the user intends to select the virtual object comprises comparing the trajectory data to a plurality of trajectory templates to identify a best-matched trajectory template of the plurality of trajectory templates corresponding to an intent to select the virtual object.
8. The method of claim 1 ,
further comprising processing the trajectory data to predict a future location of the hand; and
wherein the updated virtual location is closer to the future location of the hand than the initial virtual location is.
9. The method of claim 1 , wherein:
the initial display information is configured to control the XR display to present a plurality of virtual objects to the user on the XR display, each virtual object being presented at a respective initial virtual location;
the trajectory data is processed to determine that the user intends to select any one of two or more virtual objects of the plurality of virtual objects; and
the updated display information is configured to control the XR display to present the two or more virtual objects to the user on the XR display at a respective two or more updated virtual locations, each respective updated virtual location being closer to the hand than each respective initial virtual location.
10. A system for presenting a virtual object of an extended reality (XR) environment to a user, comprising:
a processor device; and
a memory storing machine-executable instructions thereon which, when executed by the processing device, cause the system to:
generate initial display information configured to control an XR display to present the virtual object to the user at an initial virtual location;
obtain, from one or more hand movement sensors, hand movement information indicating a velocity of the hand;
process the hand movement information to generate trajectory data;
process the trajectory data to determine that the user intends to select the virtual object; and
generate updated display information configured to control the XR display to present the virtual object to the user at an updated virtual location closer to the hand than the initial virtual location.
11. The system of claim 10 , wherein:
the machine-executable instructions, when executed by the processing device, further cause the system to obtain, from one or more head movement sensors, head movement information indicating a velocity of the head of the user; and
processing the hand movement information to generate the trajectory data comprises processing the hand movement information and the head movement information to generate the trajectory data.
12. The system of claim 11 , wherein:
generating the trajectory data comprises:
retrieving previous hand movement information from a memory;
storing the hand movement information obtained from the hand movement sensors in the memory;
retrieving previous head movement information from a memory;
storing the head movement information obtained from the head movement sensors in the memory; and
processing the previous hand movement information, the hand movement information obtained from the hand movement sensors, the previous head movement information, and the head movement information obtained from the head movement sensors to generate the trajectory data.
13. The system of claim 11 , wherein:
the hand movement information indicates a linear velocity and an angular velocity of the hand; and
the head movement information indicates a linear velocity and an angular velocity of the head.
14. The system of claim 13 , wherein:
the one or more hand movement sensors comprise an inertial measurement unit coupled to the hand; and
the one or more head movement sensors comprise an inertial measurement unit coupled to the head.
15. The system of claim 14 , further comprising a head-mounted display device configured to be worn on the head, comprising the XR display and the inertial measurement unit coupled to the head.
16. The system of claim 15 , further comprising a smart watch configured to be worn on the wrist of the hand, comprising the inertial measurement unit coupled to the hand.
17. The method of claim 11 , wherein:
the memory further stores a plurality of trajectory templates; and
processing the trajectory data to determine that the user intends to select the virtual object comprises:
retrieving from the memory the plurality of trajectory templates; and
comparing the trajectory data to the plurality of trajectory templates to identify a best-matched trajectory template of the plurality of trajectory templates corresponding to an intent to select the virtual object.
18. The system of claim 10 , wherein:
the machine-executable instructions, when executed by the processing device, further cause the system to process the trajectory data to predict a future location of the hand; and
the updated virtual location is closer to the future location of the hand than the initial virtual location is.
19. The system of claim 10 , wherein:
the initial display information is configured to control the XR display to present a plurality of virtual objects to the user on the XR display, each virtual object being presented at a respective initial virtual location;
the trajectory data is processed to determine that the user intends to select any one of two or more virtual objects of the plurality of virtual objects; and
the updated display information is configured to control the XR display to present the two or more virtual objects to the user on the XR display at a respective two or more updated virtual locations, each respective updated virtual location being closer to the hand than each respective initial virtual location.
20. A non-transitory processor-readable medium having machine-executable instructions stored thereon which, when executed by a processor device of a device, cause the device to:
generate initial display information configured to control an extended reality (XR) display to present a virtual object of an XR environment to a user at an initial virtual location;
obtain, from one or more hand movement sensors, hand movement information indicating a velocity of the hand;
process the hand movement information to generate trajectory data;
process the trajectory data to determine that the user intends to select the virtual object; and
generate updated display information configured to control the XR display to present the virtual object to the user at an updated virtual location closer to the hand than the initial virtual location.