Hand scale factor estimation from mobile interactions
An XR system is provided that enhances user interaction within extended reality environments through precise hand scale estimation. The XR system is configured to capture tracking data of a user's hand as the user interacts with a mobile device. Concurrently, the XR system captures pose data of itself and uses the tracking data and the pose data to determine a reference line segment. This segment aids in calculating three-dimensional distances between node pairs of the user's hand. By employing these measurements, the XR system effectively calculates a hand scale factor that is used for accurately integrating the user's hands into an XR user interface.
1 . A method, comprising:
capturing, by one or more cameras of an extended Reality (XR) system, tracking data of a hand of a user as the user interacts with a touch surface of a mobile device operably connected to the XR system;
capturing, by the mobile device, a length of a reference line segment made by the user as the user interacts with the touch surface of the mobile device;
determining 3D distances between node pairs of the hand using the tracking data and the reference line segment; and
calculating a hand scale factor using the 3D distances between the node pairs.
2 . The method of claim 1 , further comprising using the calculated hand scale factor to determine a scale of a virtual object associated with the hand of the user.
3 . The method of claim 1 , wherein the determining of 3D distances between node pairs includes using a default set of values for node pair distances and adjusting the default set of values based on the reference line segment.
4 . The method of claim 1 , further comprising using the calculated hand scale factor to predict a 3D hand skeleton for subsequent semantic event detection.
5 . The method of claim 1 , wherein the capturing of the tracking data is synchronized with the capturing of the reference line segment.
6 . The method of claim 1 , wherein the determining of 3D distances between node pairs of the hand comprises an iterative operation of repeatedly adjusting a hand scale factor until a difference between consecutive hand scale factors meet or fall below a predefined threshold value.
7 . The method of claim 1 , wherein the XR system comprises a head-wearable apparatus.
8 . A machine comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the machine to perform operations comprising:
capturing, by one or more cameras of an extended Reality (XR) system, tracking data of a hand of a user as the user interacts with a touch surface of a mobile device operably connected to the XR system;
capturing, by the mobile device, a length of a reference line segment made by the user as the user interacts with the touch surface of the mobile device;
determining 3D distances between node pairs of the hand using the tracking data and the reference line segment; and
calculating a hand scale factor using the 3D distances between the node pairs.
9 . The machine of claim 8 , wherein the operations further comprise using the calculated hand scale factor to determine a scale of a virtual object associated with the hand of the user.
10 . The machine of claim 8 , wherein the determining of 3D distances between node pairs includes using a default set of values for node pair distances and adjusting the default set of values based on the reference line segment.
11 . The machine of claim 8 , wherein the operations further comprise using the calculated hand scale factor to predict a 3D hand skeleton for subsequent semantic event detection.
12 . The machine of claim 8 , wherein the capturing of the tracking data is synchronized with the capturing of the reference line segment.
13 . The machine of claim 8 , wherein the determining of 3D distances between node pairs of the hand comprises an iterative operation of repeatedly adjusting a hand scale factor until a difference between consecutive hand scale factors meet or fall below a predefined threshold value.
14 . The machine of claim 8 , wherein the XR system comprises a head-wearable apparatus.
15 . A machine-storage medium, the machine-storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
capturing, by one or more cameras of an extended Reality (XR) system, tracking data of a hand of a user as the user interacts with a touch surface of a mobile device operably connected to the XR system;
capturing, by the mobile device, a length of a reference line segment made by the user as the user interacts with the touch surface of the mobile device;
determining 3D distances between node pairs of the hand using the tracking data and the reference line segment; and
calculating a hand scale factor using the 3D distances between the node pairs.
16 . The machine-storage medium of claim 15 , wherein the operations further comprise using the calculated hand scale factor to determine a scale of a virtual object associated with the hand of the user.
17 . The machine-storage medium of claim 15 , wherein the determining of 3D distances between node pairs includes using a default set of values for node pair distances and adjusting the default set of values based on the reference line segment.
18 . The machine-storage medium of claim 15 , wherein the operations further comprise using the calculated hand scale factor to predict a 3D hand skeleton for subsequent semantic event detection.
19 . The machine-storage medium of claim 15 , wherein the capturing of the tracking data is synchronized with the capturing of the reference line segment.
20 . The machine-storage medium of claim 15 , wherein the XR system comprises a head-wearable apparatus.