Natural hand rendering in XR systems
An extended Reality (XR) system provides methodologies for clipping a virtual object displayed to a user. The XR system provides an XR user interface that includes a virtual object. The XR system captures tracking video frame data of a hand of the user and generates a clipping mask based on the tracking video frame data. The XR system generates a clipped virtual object by applying the clipping mask to the virtual object and displays the clipped virtual object in the XR user interface.
1 . A computer-implemented method comprising:
providing, by one or more processors, an extended Reality (XR) user interface of an XR system, the XR user interface comprising a virtual object displayed to a user, the virtual object including a 3D model;
capturing, by the one or more processors, tracking video frame data of a hand of the user;
generating, by the one or more processors, a 3D model of the hand of the user using the tracking video frame data;
determining, by the one or more processors, a hand distance based on the 3D model of the hand of the user;
determining, by the one or more processors, a near clipping plane based on the hand distance;
defining, by the one or more processors, a first surface of an irregular frustum by detecting an intersection of the near clipping plane with the 3D model of the hand of the user;
defining, by the one or more processors, a second surface of the irregular frustum by performing a projection of the first surface of the irregular frustum to a far clipping plane;
defining, by the one or more processors, a 3D clipping volume using the irregular frustum;
clipping, by the one or more processors, the 3D model of the virtual object using the 3D clipping volume;
generating, by the one or more processors, a clipped virtual object using the 3D model of the virtual object; and
displaying, by the one or more processors, the clipped virtual object in the XR user interface.
2 . The computer-implemented method of claim 1 , wherein the XR system comprises a head-wearable apparatus.
3 . The computer-implemented method of claim 1 , wherein determining the near clipping plane comprises:
determining a hand distance based using binocular video frame data of the tracking video frame data; and
determining the near clipping plane based on the hand distance.
4 . The computer-implemented method of claim 1 , wherein determining the near clipping plane comprises:
determining a hand distance based using monocular video frame data of the tracking video frame data; and
determining the near clipping plane based on the hand distance.
5 . The computer-implemented method of claim 4 , wherein the hand distance is determined based on the monocular video frame data and pose-tracking data.
6 . The computer-implemented method of claim 1 , wherein determining the near clipping plane comprises:
determining a hand distance based on a depth detection using Light Detection and Radar (LiDAR); and
determining the near clipping plane based on the hand distance.
7 . A machine comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
providing to a user an extended Reality (XR) user interface of an XR system, the XR user interface comprising a virtual object displayed to the user, the virtual object including a 3D model;
capturing tracking video frame data of a hand of the user;
generating a 3D model of the hand of the user using the tracking video frame data;
determining a hand distance based on the 3D model of the hand of the user;
determining a near clipping plane based on the hand distance;
defining a first surface of an irregular frustum by detecting an intersection of the near clipping plane with the 3D model of the hand of the user;
defining a second surface of the irregular frustum by performing a projection of the first surface of the irregular frustum to a far clipping plane;
defining a 3D clipping volume using the irregular frustum;
clipping the 3D model of the virtual object using the 3D clipping volume;
generating a clipped virtual object using the 3D model of the virtual object; and
displaying the clipped virtual object to the user in the XR user interface.
8 . The machine of claim 7 , wherein the XR system comprises a head-wearable apparatus.
9 . The machine of claim 7 , wherein determining the near clipping plane comprises:
determining a hand distance based using binocular video frame data of the tracking video frame data; and
determining the near clipping plane based on the hand distance.
10 . The machine of claim 7 , wherein determining the near clipping plane comprises:
determining a hand distance based using monocular video frame data of the tracking video frame data; and
determining the near clipping plane based on the hand distance.
11 . The machine of claim 10 , wherein the hand distance is determined based on the monocular video frame data and pose-tracking data.
12 . The machine of claim 7 , wherein determining the near clipping plane comprises:
determining a hand distance based on a depth detection using Light Detection and Radar (LiDAR); and
determining the near clipping plane based on the hand distance.
13 . A non-transitory machine-readable storage medium, the machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
providing, by one or more processors, to a user an extended Reality (XR) user interface of an XR system, the XR user interface comprising a virtual object displayed to the user, the virtual object including a 3D model;
capturing, by the one or more processors, tracking video frame data of a hand of the user;
generating, by the one or more processors, a 3D model of the hand of the user using the tracking video frame data;
determining, by the one or more processors, a hand distance based on the 3D model of the hand of the user;
determining, by the one or more processors, a near clipping plane based on the hand distance;
defining, by the one or more processors, a first surface of an irregular frustum by detecting an intersection of the near clipping plane with the 3D model of the hand of the user;
defining, by the one or more processors, a second surface of the irregular frustum by performing a projection of the first surface of the irregular frustum to a far clipping plane;
defining, by the one or more processors, a 3D clipping volume using the irregular frustum;
clipping, by the one or more processors, the 3D model of the virtual object using the 3D clipping volume;
generating, by the one or more processors, a clipped virtual object using the 3D model of the virtual object; and
displaying, by the one or more processors, the clipped virtual object to the user in the XR user interface.
14 . The non-transitory machine-readable storage medium of claim 13 , wherein the XR system comprises a head-wearable apparatus.
15 . The non-transitory machine-readable storage medium of claim 13 , wherein determining the near clipping plane comprises:
determining a hand distance based using binocular video frame data of the tracking video frame data; and
determining the near clipping plane based on the hand distance.
16 . The non-transitory machine-readable storage medium of claim 13 , wherein determining the near clipping plane comprises:
determining a hand distance based using monocular video frame data of the tracking video frame data; and
determining the near clipping plane based on the hand distance.
17 . The non-transitory machine-readable storage medium of claim 16 , wherein the hand distance is determined based on the monocular video frame data and pose-tracking data.