Non-user hand rejection for extended reality devices
Examples in the present disclosure relate to systems and methods for detecting and rejecting a non-user hand in the context of egocentric hand tracking performed by an extended reality (XR) device. While the XR device is worn by a user, the XR device captures at least one image of a hand and processes the at least one image to detect the hand. After detecting the hand, the XR device determines positioning of the hand relative to the XR device or another object in a field of view of the XR device. The XR device detects that the hand is a non-user hand. In response to detecting that the hand is a non-user hand, the XR device excludes the non-user hand from the egocentric hand tracking such that the non-user hand is not tracked for the user.
1 . An extended reality (XR) device comprising:
one or more optical sensors;
one or more processors; and
at least one memory storing instructions that, when executed by the one or more processors, cause the XR device, when worn by a user, to perform operations comprising:
capturing, via the one or more optical sensors, at least one image of a hand;
processing the at least one image to detect the hand, the processing comprising executing at least one object detection machine learning model that returns a confidence value;
after detecting the hand, determining positioning of the hand relative to at least one of the XR device or an additional object in a field of view of the XR device, the determining of the positioning of the hand relative to at least one of the XR device or the additional object being triggered based on determining that the confidence value meets or exceeds a threshold;
detecting, based on the positioning of the hand relative to at least one of the XR device or the additional object, that the hand is a non-user hand; and
in response to detecting that the hand is the non-user hand, excluding the hand from egocentric hand tracking performed by the XR device with respect to the user.
2 . The XR device of claim 1 , wherein the hand is detected to be the non-user hand based on both the positioning of the hand relative to the XR device and the positioning of the hand relative to the additional object in the field of view of the XR device.
3 . The XR device of claim 1 , wherein the detecting, based on the positioning of the hand relative to at least one of the XR device or the additional object, that the hand is the non-user hand, comprises executing a plurality of rejection filters in a predetermined sequence.
4 . The XR device of claim 1 , wherein the determining of the positioning of the hand relative to at least one of the XR device or the additional object comprises:
generating a three-dimensional (3D) position associated with the hand; and
determining, based on the 3D position associated with the hand, a distance between the hand and at least one of the user or the XR device, wherein the hand is detected to be the non-user hand based on the distance meeting or exceeding a threshold.
5 . The XR device of claim 1 , wherein the determining of the positioning of the hand relative to at least one of the XR device or the additional object comprises:
generating a zone associated with a location of the hand within the at least one image, wherein the hand is detected to be the non-user hand based on a size of the zone satisfying a predetermined condition.
6 . The XR device of claim 5 , wherein the generating of the zone comprises generating a bounding element that covers at least part of the hand, and the size of the zone comprises a two-dimensional (2D) area of the bounding element.
7 . The XR device of claim 1 , wherein the additional object is a user hand that is being tracked using the egocentric hand tracking performed by the XR device, and the determining of the positioning of the hand relative to at least one of the XR device or the additional object comprises:
comparing the positioning of the hand with positioning of the user hand, wherein the hand is detected to be the non-user hand based on the positioning of the hand relative to the user hand being invalid according to a predetermined condition.
8 . The XR device of claim 7 , wherein the predetermined condition indicates, based on a chirality of the user hand, on which side of the user hand the hand is to appear in the at least one image.
9 . The XR device of claim 1 , the operations further comprising:
identifying a chirality of the hand, wherein the hand is detected to be the non-user hand based on both the chirality of the hand and horizontal positioning of the hand within a scene captured by the at least one image.
10 . The XR device of claim 9 , wherein the horizontal positioning of the hand is provided as input to a decision function to generate a value indicative of a likelihood that the positioning of the hand is invalid.
11 . The XR device of claim 1 , wherein the determining of the positioning of the hand relative to at least one of the XR device or the additional object comprises:
determining an entry region of the hand within the field of view of the XR device, wherein the hand is detected to be the non-user hand based on the entry region being invalid according to a predetermined condition.
12 . The XR device of claim 1 , wherein the additional object comprises an arm that appears in the at least one image, and the determining of the positioning of the hand relative to at least one of the XR device or the additional object comprises:
detecting that the arm corresponds to the hand, wherein the hand is detected to be the non-user hand based on positioning of the arm relative to at least one of the hand or the XR device.
13 . The XR device of claim 1 , the operations further comprising:
causing presentation, to the user, of a gesture-driven user interface comprising virtual content; and
performing the egocentric hand tracking to obtain, from the user, user input for navigation of the gesture-driven user interface.
14 . The XR device of claim 1 , wherein, for a given hand detected by the XR device during a detection phase, the egocentric hand tracking is performed in a tracking phase that follows completion of the detection phase.
15 . The XR device of claim 14 , wherein the excluding of the hand from the egocentric hand tracking is performed after completion of the detection phase for the hand.
16 . The XR device of claim 14 , wherein the detecting, based on the positioning of the hand relative to at least one of the XR device or the additional object, that the hand is a non-user hand, comprises executing a plurality of rejection filters in a predetermined sequence that comprises at least one rejection filter that is executed before commencement of the tracking phase and at least one further rejection filter that is executed during the tracking phase.
17 . The XR device of claim 1 , wherein the XR device is a head-wearable XR device, and the operations are performed while the XR device is worn on a head of the user.
18 . A method performed by an extended reality (XR) device while the XR device is worn by a user, the method comprising:
capturing, via one or more optical sensors, at least one image of a hand;
processing the at least one image to detect the hand, the processing comprising executing at least one object detection machine learning model that returns a confidence value;
after detecting the hand, determining positioning of the hand relative to at least one of the XR device or an additional object in a field of view of the XR device, the determining of the positioning of the hand relative to at least one of the XR device or the additional object being triggered based on determining that the confidence value meets or exceeds a threshold;
detecting, based on the positioning of the hand relative to at least one of the XR device or the additional object, that the hand is a non-user hand; and
in response to detecting that the hand is the non-user hand, excluding the hand from egocentric hand tracking performed by the XR device with respect to the user.
19 . The method of claim 18 , wherein the hand is detected to be the non-user hand based on both the positioning of the hand relative to the XR device and the positioning of the hand relative to the additional object in the field of view of the XR device.
20 . One or more non-transitory computer-readable storage media, the one or more non-transitory computer-readable storage media including instructions that, when executed by at least one processor of an extended reality (XR) device worn by a user, cause the XR device to perform operations comprising:
capturing, via one or more optical sensors, at least one image of a hand;
processing the at least one image to detect the hand, the processing comprising executing at least one object detection machine learning model that returns a confidence value;
after detecting the hand, determining positioning of the hand relative to at least one of the XR device or an additional object in a field of view of the XR device, the determining of the positioning of the hand relative to at least one of the XR device or the additional object being triggered based on determining that the confidence value meets or exceeds a threshold;
detecting, based on the positioning of the hand relative to at least one of the XR device or the additional object, that the hand is a non-user hand; and
in response to detecting that the hand is the non-user hand, excluding the hand from egocentric hand tracking performed by the XR device with respect to the user.