Artificial reality multi-modal input switching model
Embodiments described herein disclose methods and systems directed to input mode selection in artificial reality. In some implementations, various input modes enable a user to perform precise interactions with a target object without occluding the target object. Some input modes can include rays that extend along a line that intersects an origin point, a control point, and an interaction point. An interaction model can specify when the system switches between input modes, such as modes based solely on gaze, using long or short ray input, or with direct interaction between the user's hand(s) and objects. These transitions can be performed by evaluating rules that take context factors such as whether a user's hands are in view of the user, what posture the hands are in, whether a target object is selected, and whether a target object is within a threshold distance from the user.
1. A method for input mode selection in artificial reality, the method comprising:
identifying that a user hand is in view of one or more posture capturing cameras of an artificial reality (XR) device;
monitoring hand positions of the user hand using one or more images captured by the one or more posture capturing cameras;
identifying a hand posture of a user based on the monitoring the hand positions of the user;
determining the hand posture is a ray posture, by determining that an identified gesture of the hand of the user is mapped to performing inputs with ray projections, wherein a target object has been selected through a previous user interaction performed prior to determining that the hand posture is in the ray posture;
measuring a distance between the user and the previously selected target object, wherein the measuring is used to determine, while the hand posture is a ray posture, that the previously selected target object is within a threshold distance from the user, wherein the threshold distance from the user to the previously selected target object is a distance defined to determine the input mode to cast a ray; and
in response to both the determining the hand posture is in the ray posture and the determining that the previously selected target object is within the threshold distance defined to determine the input mode to cast a ray, automatically enabling a short ray input mode by selecting the short ray input mode from among multiple available ray input modes available on the XR device,
wherein the short ray input mode provides a ray of a specified length for interacting with the previously selected target object; and
wherein the ray input mode selecting comprises selecting from among the multiple available ray input modes including the short ray input mode and a long ray input mode, wherein the long ray input mode provides a ray that is longer than the specified length and/or that extends until the ray intersects with an object.
2. The method of claim 1 , wherein the specified length is a length between three and twelve inches.
3. The method of claim 1 further comprising:
tracking an origin point on the user hand and a control point, different from the origin point, on the user hand; and
casting the ray, of the specified length from the control point to an interaction point, along a line connecting the origin point and the control point.
4. The method of claim 1 , wherein the selecting the short ray input mode from among multiple available ray input modes available on the XR device is performed by selecting the short ray input mode to transition to from a current input mode, wherein the selection of a short ray input mode includes each of:
the identifying that the user hand is in view,
the monitoring hand positions of the user,
the identifying the hand posture of a user,
the determining the hand posture is the ray posture, and
the measuring the distance between the user and the previously selected target object.
5. The method of claim 4 , wherein the current input mode is a default input mode.
6. The method of claim 1 , wherein the hand posture of the user is a first hand posture of the user, and wherein the method further comprises:
after enabling the short ray input mode:
identifying a second hand posture of the user based on the monitoring the hand positions of the user;
determining the second hand posture is not the ray posture;
in response to the determining the second hand posture is not the ray posture, determining a second target object is within the threshold distance from the user; and
in response to determining the second target object is within the threshold distance, enabling a direct hand input mode, wherein the direct hand input mode uses an intersection between the hand of the user and the second target object to move the second target object.
7. The method of claim 1 , wherein the hand posture of the user is a first hand posture of the user, and wherein the method further comprises:
after enabling the short ray input mode:
identifying a second hand posture of the user based on the monitoring the hand positions of the user;
determining the second hand posture is not the ray posture;
in response to the determining the second hand posture is not the ray posture, determining a second target object is not selected or is not within the threshold distance from the user; and
in response to determining the second target object is not selected or is not within the threshold distance, enabling a gaze input mode, wherein the gaze input mode uses eye gaze direction of the user to determine input direction.
8. The method of claim 1 , wherein the previously selected target object is a first target object, the ray is a first ray, and the specified length is a first specified length, and wherein the method further comprises:
determining that a second target object is not within the threshold distance from the user; and
in response to determining the second target object is not within the threshold distance, enabling a long ray input mode, wherein the long ray input mode provides a second ray of a second specified length, longer than the first specified length, for interacting with the second target object.
9. The method of claim 1 further comprising:
after enabling the short ray input mode:
determining that the user hand is no longer in view of the one or more posture capturing cameras; and
in response to determining the user hand is no longer in view of the one or more posture capturing cameras, enabling a gaze input mode, wherein the gaze input mode uses eye gaze direction of the user to determine input direction.
10. The method of claim 1 further comprising receiving the previous user interaction through a gaze selection by the user.
11. The method of claim 1 further comprising receiving the previous user interaction through a hand gesture selection by the user.
12. A computing system for input mode selection in artificial reality, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
identifying that a user hand is in view of one or more posture capturing cameras of an artificial reality (XR) device;
monitoring hand positions of the user hand using one or more images captured by the one or more posture capturing cameras;
identifying a hand posture of a user based on the monitoring the hand positions of the user;
determining the hand posture is a ray posture, by determining that an identified gesture of the hand of the user is mapped to performing inputs with ray projections, wherein a target object has been selected through a previous user interaction performed prior to determining that the hand posture in in the ray posture;
measuring a distance between the user and the previously selected target object, wherein the measuring is used to determine, while the hand posture is a ray posture, that the previously selected target object is within a threshold distance from the user, wherein the threshold distance from the user to the previously selected target object is a distance defined to determine the input mode to cast a ray; and
in response to both the determining the hand posture is in the ray posture and the determining that the previously selected target object is within the threshold distance defined to determine the input mode to cast a ray, automatically enabling a short ray input mode by selecting the short ray input mode from among multiple available ray input modes available on the XR device,
wherein the short ray input mode provides a ray of a specified length for interacting with the previously selected target object; and
wherein the ray input mode selecting comprises selecting from among the multiple available ray input modes including the short ray input mode and a long ray input mode, wherein the long ray input mode provides a ray that is longer than the specified length and/or that extends until the ray intersects with an object.
13. The computing system of claim 12 , wherein the process further comprises:
tracking an origin point on the user hand and a control point, different from the origin point, on the user hand; and
casting the ray, of the specified length from the control point to an interaction point, along a line connecting the origin point and the control point.
14. The computing system of claim 12 , wherein the hand posture of the user is a first hand posture of the user, and wherein the process further comprises:
after enabling the short ray input mode:
identifying a second hand posture of the user based on the monitoring the hand positions of the user;
determining the second hand posture is not the ray posture;
in response to the determining the second hand posture is not the ray posture, determining a second target object is within the threshold distance from the user; and
in response to determining the second target object is within the threshold distance, enabling a direct hand input mode, wherein the direct hand input mode uses an intersection between the hand of the user and the second target object to move the second target object.
15. The computing system of claim 12 , wherein the hand posture of the user is a first hand posture of the user, and wherein the process further comprises:
after enabling the short ray input mode:
identifying a second hand posture of the user based on the monitoring the hand positions of the user;
determining the second hand posture is not the ray posture;
in response to the determining the second hand posture is not the ray posture, determining a second target object is not selected or is not within the threshold distance from the user; and
in response to determining the second target object is not selected or is not within the threshold distance, enabling a gaze input mode, wherein the gaze input mode uses eye gaze direction of the user to determine input direction.
16. The computing system of claim 12 , wherein the previously selected target object is a first target object, the ray is a first ray, and the specified length is a first specified length, and wherein the process further comprises:
determining that a second target object is not within the threshold distance from the user; and
in response to determining the second target object is not within the threshold distance, enabling a long ray input mode, wherein the long ray input mode provides a second ray of a second specified length, longer than the first specified length, for interacting with the second target object.
17. A machine-readable storage medium having machine executable instructions stored thereon that, when executed by one or more processors, direct the one or more processors to perform a method for input mode selection in artificial reality, the method comprising:
identifying that a user hand is in view of one or more posture capturing cameras of an artificial reality (XR) device;
monitoring hand positions of the user hand using one or more images captured by the one or more posture capturing cameras;
identifying a hand posture of a user based on the monitoring the hand positions of the user;
determining the hand posture is a ray posture, by determining that an identified gesture of the hand of the user is mapped to performing inputs with ray projections, wherein a target object has been selected through a previous user interaction performed prior to determining that the hand posture is in the ray posture;
measuring a distance between the user and the previously selected target object, wherein the measuring is used to determine, while the hand posture is a ray posture, that the previously selected target object is within a threshold distance from the user, wherein the threshold distance from the user to the previously selected target object is a distance defined to determine the input mode to cast a ray; and
in response to both the determining the hand posture is in the ray posture and the determining that the previously selected target object is within the threshold distance defined to determine the input mode to cast a ray, automatically enabling a short ray input mode by selecting the short ray input mode from among multiple available ray input modes available on the XR device,
wherein the short ray input mode provides a ray of a specified length for interacting with the previously selected target object; and
wherein the ray input mode selecting comprises selecting from among the multiple available ray input modes including the short ray input mode and a long ray input mode, wherein the long ray input mode provides a ray that is longer than the specified length and/or that extends until the ray intersects with an object.
18. The machine-readable storage medium of claim 17 , wherein the method further comprises:
tracking an origin point on the user hand and a control point, different from the origin point, on the user hand; and
casting the ray, of the specified length from the control point to an interaction point, along a line connecting the origin point and the control point.
19. The machine-readable storage medium of claim 17 , wherein the hand posture of the user is a first hand posture of the user, and wherein the method further comprises:
after enabling the short ray input mode:
identifying a second hand posture of the user based on the monitoring the hand positions of the user;
determining the second hand posture is not the ray posture;
in response to the determining the second hand posture is not the ray posture, determining a second target object is within the threshold distance from the user; and
in response to determining the second target object is within the threshold distance, enabling a direct hand input mode, wherein the direct hand input mode uses an intersection between the hand of the user and the second target object to move the second target object.
20. The machine-readable storage medium of claim 17 , wherein the hand posture of the user is a first hand posture of the user, and wherein the method further comprises:
after enabling the short ray input mode:
identifying a second hand posture of the user based on the monitoring the hand positions of the user;
determining the second hand posture is not the ray posture;
in response to the determining the second hand posture is not the ray posture, determining a second target object is not selected or is not within the threshold distance from the user; and
in response to determining the second target object is not selected or is not within the threshold distance, enabling a gaze input mode, wherein the gaze input mode uses eye gaze direction of the user to determine input direction.