Widget interaction for extended reality (XR) applications
A method includes rendering a virtual widget at a first position in an extended reality (XR) space using a head-mounted display. The method also includes detecting a hand gesture of a user at a second position in the XR space using at least one optical sensor. The second position is between the first position and the optical sensor. The method further includes determining a duration of the hand gesture and a distance between the first and second positions. The method also includes performing a first or second type of action based on the duration and/or the distance. The first type of action corresponds to a system-level control of the virtual widget, and the second type of action corresponds to a content-level control of the virtual widget. In addition, the method includes rendering a visual and/or audio feedback based on at least one of the duration or the distance.
1 . A method comprising:
rendering a virtual widget at a first position in an extended reality (XR) space using a head-mounted display (HMD);
detecting a hand gesture of a user at a second position in the XR space using at least one optical sensor of the HMD, wherein the second position is between the first position and the at least one optical sensor;
determining a duration of the hand gesture and a distance between the first position and the second position;
performing a first type of action or a second type of action based on at least one of the duration or the distance, wherein the first type of action corresponds to a system-level control of the virtual widget and the second type of action corresponds to a content-level control of the virtual widget, and wherein a content hit target is marked to disable widget system level hits when the second type of action is performed; and
rendering at least one of a visual feedback or an audio feedback based on at least one of the duration or the distance.
2 . The method of claim 1 , wherein the first type of action comprises at least one of: invoking a widget menu, moving the virtual widget, placing the virtual widget at a location, or invoking resizing handles.
3 . The method of claim 1 , wherein the second type of action comprises at least one of: selecting a button on the virtual widget, navigating or scrolling a content on the virtual widget, or invoking widget expansion for additional interactions with the virtual widget.
4 . The method of claim 1 , wherein rendering the visual feedback comprises rendering a raycast between the first position and the second position based on the distance.
5 . The method of claim 1 , wherein the hand gesture includes at least one of: a pinch gesture, a poke gesture, a pinch-and-drag gesture, a pinch-and-hold gesture, or a hover gesture.
6 . The method of claim 1 , further comprising:
detecting a second hand gesture of the user in the XR space, the second hand gesture associated with a physical object; and
performing a third type of action or rendering at least one of the visual feedback or the audio feedback based on the second hand gesture.
7 . The method of claim 1 , further comprising:
detecting a voice command from the user; and
performing the first type of action or the second type of action based on the voice command.
8 . An electronic device comprising:
a head-mounted display (HMD) configured to render a virtual widget at a first position in an extended reality (XR) space; and
at least one processing device configured to:
receive a hand gesture of a user at a second position in the XR space detected using at least one optical sensor of the HMD, wherein the second position is between the first position and the at least one optical sensor;
determine a duration of the hand gesture and a distance between the first position and the second position;
perform a first type of action or a second type of action based on at least one of the duration or the distance, wherein the first type of action corresponds to a system-level control of the virtual widget and the second type of action corresponds to a content-level control of the virtual widget, and wherein a content hit target is marked to disable widget system level hits when the second type of action is performed; and
render at least one of a visual feedback or an audio feedback based on at least one of the duration or the distance.
9 . The electronic device of claim 8 , wherein:
the first type of action comprises at least one of: invoking a widget menu, moving the virtual widget, placing the virtual widget at a location, or invoking resizing handles; and
the second type of action comprises at least one of: selecting a button on the virtual widget, navigating or scrolling a content on the virtual widget, or invoking widget expansion for additional interactions with the virtual widget.
10 . The electronic device of claim 8 , wherein, to render the visual feedback, the at least one processing device is configured to render a raycast between the first position and the second position based on the distance.
11 . The electronic device of claim 8 , wherein the hand gesture includes at least one of: a pinch gesture, a poke gesture, a pinch-and-drag gesture, a pinch-and-hold gesture, or a hover gesture.
12 . The electronic device of claim 8 , wherein the at least one processing device is further configured to:
receive a second hand gesture of the user detected in the XR space, the second hand gesture associated with a physical object; and
perform a third type of action or render at least one of the visual feedback or the audio feedback based on the second hand gesture.
13 . The electronic device of claim 8 , wherein the at least one processing device is further configured to:
receive a voice command from the user; and
perform the first type of action or the second type of action based on the voice command.
14 . A method comprising:
receiving a first video captured using an outward facing camera associated with a head-mounted display (HMD) worn by a user, the first video capturing a user environment;
receiving a second video captured using an eye tracking camera associated with the HMD;
identifying a real world object on which eyes of the user are focused based on the second video, wherein the real world object is associated with a set of control options;
calibrating the real world object and user contact with the real world object to map and interpret one or more movements of the real world object and the user contact to control one or more widgets based on one or more calibration thresholds;
determining a gaze of the user based on the second video;
registering a widget last gazed on by the user;
determining a type of interaction the user makes with the real world object during the user contact based on the first video, the type of interaction comprising one of a contact interaction, a rotation interaction, or a shift interaction;
selecting a control option among the set of control options based on the determined type of interaction;
selecting a virtual widget based on the gaze of the user and the registered widget; and
controlling the virtual widget in an extended reality (XR) space based on the selected control option and the one or more calibration thresholds.
15 . The method of claim 14 , wherein the contact interaction is associated with a Boolean control of the virtual widget, the Boolean control having two states.
16 . The method of claim 14 , wherein:
the rotation interaction is associated with a continuous control of the virtual widget between a minimum value and a maximum value; and
a change in angle caused by the rotation interaction is mapped to a value of the virtual widget between the minimum value and the maximum value.
17 . The method of claim 14 , wherein:
the shift interaction is associated with a continuous control of the virtual widget between a minimum value and a maximum value; and
a change in position caused by the shift interaction is mapped to a value of the virtual widget between the minimum value and the maximum value.
18 . The method of claim 17 , wherein:
the change in position represents a change along a first axis and a second axis;
the change along the first axis is used to control a first feature of the virtual widget; and
the change along the second axis is used to control a second feature of the virtual widget.
19 . The method of claim 14 , further comprising:
detecting one or more voice commands from the user; and
controlling the virtual widget based on the one or more voice commands.
20 . The method of claim 1 , wherein the second type of action comprises invoking widget expansion for additional interactions with the virtual widget and the expanded virtual widget does not invoke the system-level control.