Bending-assisted calibration for extended reality tracking

1 . A method comprising:

obtaining bending data indicative of bending of an extended reality (XR) device to accommodate a body part of a user wearing the XR device;

determining, based on the bending data, whether to use previously identified biometric data in a tracking operation of the XR device;

selecting a mode of the XR device, the selected mode being a first mode in which the previously identified biometric data is applied in the tracking operation or a second mode in which previous biometric data is not applied in the tracking operation; and

performing, by the XR device, the tracking operation based on the selected mode.

2 . The method of claim 1 , wherein the previously identified biometric data comprises a hand feature estimate generated during a previous user session.

3 . The method of claim 1 , wherein obtaining the bending data comprises:

analyzing images from a plurality of cameras of the XR device having overlapping fields of view to estimate the bending of the XR device.

4 . The method of claim 3 , wherein analyzing the images comprises:

generating a depth map based on matched features between views from the plurality of cameras; and

analyzing the depth map to estimate the bending of the XR device.

5 . The method of claim 1 , wherein performing the tracking operation comprises:

implementing a detection phase identifying an object of interest; and

implementing a tracking phase estimating a pose of the object over time.

6 . The method of claim 5 , wherein the detection phase comprises:

detecting the object of interest in a camera field of view;

detecting landmarks on the object of interest; and

finding the landmarks across a plurality of camera images.

7 . The method of claim 5 , wherein the tracking phase comprises:

estimating a location of the object of interest based on at least one of object appearance, object motion, or object landmarks; and

updating the estimated pose of the object of interest based on the estimated location.

8 . The method of claim 1 , wherein performing the tracking operation in the first mode comprises:

loading the previously identified biometric data, the previously identified biometric data comprising a hand feature estimate;

computing a three-dimensional model of a hand based on the hand feature estimate; and

constraining parameters for tracking of the hand based on the hand feature estimate.

9 . The method of claim 8 , wherein the hand feature estimate is based on one or more bones of the hand.

10 . The method of claim 1 , wherein determining, based on the bending data, whether to use the previously identified biometric data in the tracking operation of the XR device comprises:

comparing the bending data to previously stored bending data associated with the previously identified biometric data; and

determining that the bending matches the previously stored bending data when the bending data is within an acceptable range of the previously stored bending data.

11 . The method of claim 10 , wherein the acceptable range is defined by a threshold difference between the bending data and the previously stored bending data.

12 . The method of claim 1 , wherein determining, based on the bending data, whether to use the previously identified biometric data in the tracking operation of the XR device comprises:

processing the bending data to generate a prediction for biometric data; and

comparing the prediction with the previously identified biometric data.

13 . The method of claim 1 , wherein the selected mode is the first mode, and performing the tracking operation in the first mode comprises:

loading the previously identified biometric data; and

initializing the tracking operation using the loaded previously identified biometric data.

14 . The method of claim 1 , wherein the selected mode is the second mode, and performing the tracking operation in the second mode comprises:

performing a calibration operation to obtain new biometric data;

storing the new biometric data; and

performing the tracking operation using the new biometric data.

15 . The method of claim 1 , wherein the XR device comprises a head-wearable device including a flexible frame configured to bend to accommodate a head of the user.

16 . The method of claim 1 , wherein obtaining the bending data comprises using one or more strain gauge sensors to detect deformation.

17 . The method of claim 1 , wherein obtaining the bending data comprises:

measuring a strain of the XR device;

detecting changes in the strain when the XR device bends to accommodate a head of the user wearing the XR device; and

generating the bending data based on the changes in the strain.

18 . The method of claim 1 , wherein the tracking operation comprises tracking hand gestures for user input to the XR device.

19 . A system comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, configure the system to perform operations comprising:

obtaining bending data indicative of bending of an extended reality (XR) device to accommodate a body part of a user wearing the XR device;

determining, based on the bending data, whether to use previously identified biometric data in a tracking operation of the XR device;

selecting a mode of the XR device, the selected mode being a first mode in which the previously identified biometric data is applied in the tracking operation or a second mode in which previous biometric data is not applied in the tracking operation; and

performing, by the XR device, the tracking operation based on the selected mode.

20 . One or more non-transitory computer-readable storage media, the computer-readable storage media including instructions that when executed by at least one processor, cause the at least one processor to perform operations comprising:

obtaining bending data indicative of bending of an extended reality (XR) device to accommodate a body part of a user wearing the XR device;

determining, based on the bending data, whether to use previously identified biometric data in a tracking operation of the XR device;

selecting a mode of the XR device, the selected mode being a first mode in which the previously identified biometric data is applied in the tracking operation or a second mode in which previous biometric data is not applied in the tracking operation; and

performing, by the XR device, the tracking operation based on the selected mode.