Surface-based user input for extended reality

1 . A method performed by an extended reality (XR) device, the method comprising:

generating a three-dimensional reconstruction of a real-world environment;

tracking a hand of a user across multiple image frames captured by the XR device to obtain positions of one or more parts of the hand in the real-world environment;

identifying a plurality of potential input lanes in the real-world environment using the three-dimensional reconstruction;

calculating projection errors associated with each of the plurality of potential input planes identified in the real-world environment;

selecting an input plane, from the plurality of potential input planes, associated with a physical surface in the real-world environment, based on determining that the input plane minimizes one or more of the projection errors;

projecting the positions onto the input plane;

monitoring an input state with respect to the input plane to:

identify when the user is providing user input via the input plane, and

differentiate between ongoing user input and finalized user input;

recording, based on the monitoring of the input state, one or more of the projected positions as input data;

processing the input data to interpret the user input; and

executing an action based on the interpreted user input.

2 . The method of claim 1 , wherein the one or more of the projected positions comprise multiple projected positions, the recording of the one or more of the projected positions comprising:

while the input state indicates that the user input is ongoing, recording the multiple projected positions as the user input for a single user interaction;

detecting a transition in the input state indicating that the user input is finalized; and

in response to detecting the transition, consolidating the multiple projected positions recorded for the single user interaction for unified interpretation.

3 . The method of claim 2 , wherein the monitoring of the input state further comprises:

at a first point in time, detecting commencement of the single user interaction;

in response to detecting the commencement of the single user interaction, updating the input state to indicate that the user input is ongoing;

at a second point in time, detecting ending of the single user interaction; and

in response to detecting the ending of the single user interaction, updating the input state to indicate that the user input is finalized.

4 . The method of claim 1 , wherein the monitoring of the input state comprises:

processing at least a subset of the multiple image frames to detect engagement of the one or more parts of the hand with the input plane.

5 . The method of claim 4 , wherein the one or more parts of the hand comprise a fingertip of a finger of the hand, the XR device tracking the fingertip across the multiple image frames.

6 . The method of claim 1 , wherein the one or more parts of the hand comprise at least one finger, and the user provides the user input by using the at least one finger to write or draw on the physical surface in the real-world environment.

7 . The method of claim 6 , wherein the processing of the input data comprises detecting what the user has written or drawn by performing at least one of optical character recognition (OCR) or image classification.

8 . The method of claim 1 , wherein the tracking of the hand causes generation of three-dimensional position data, and the projecting of the positions onto the input plane comprises converting the three-dimensional position data to two-dimensional position data in an input plane space defined by the input plane.

9 . The method of claim 1 , wherein the hand is a first hand of the user, and the physical surface comprises a surface on a second hand of the user, the input plane being detected by fitting the input plane to the surface on the second hand.

10 . The method of claim 9 , further comprising:

tracking the second hand of the user to obtain a plurality of landmark positions for the second hand, the input plane being detected by fitting the input plane to the plurality of landmark positions.

11 . The method of claim 10 , further comprising:

tracking the plurality of landmark positions across the multiple image frames; and

applying temporal smoothing to dynamically update the input plane based on the tracking of the plurality of landmark positions across the multiple image frames.

12 . The method of claim 9 , wherein the tracking of the first hand comprises tracking at least one fingertip of the first hand used by the user to provide the user input.

13 . The method of claim 1 , wherein the input plane moves relative to the XR device while the user provides the user input, the method further comprising:

continuously tracking both a plane position of the input plane relative to the XR device and a hand position of the hand relative to the XR device.

14 . The method of claim 13 , further comprising, for each particular image frame of the multiple image frames:

updating the plane position and the hand position; and

using the updated plane position and the updated hand position to generate at least one of the one or more projected positions for a point in time associated with the particular image frame.

15 . The method of claim 1 , further comprising:

prior to interpreting the user input, performing interpolation to smooth the input data.

16 . The method of claim 1 , further comprising:

prior to interpreting the user input, updating the input data by performing rotation normalization.

17 . The method of claim 1 , wherein the monitoring the input state comprises using a machine learning model trained on at least one of:

interaction data indicating touch states and no-touch states; or

interaction data indicating ongoing user input states and finalized user input states.

18 . The method of claim 17 , wherein the monitoring the input state further comprises using the machine learning model in combination with at least one of:

one or more geometry-based cues;

one or more distance-based heuristics; or

temporal context data.

19 . An extended reality (XR) device comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, configure the XR device to perform operations comprising:

generating a three-dimensional reconstruction of a real-world environment;

tracking a hand of a user across multiple image frames captured by the XR device to obtain positions of one or more parts of the hand in the real-world environment;

identifying a plurality of potential input planes in the real-world environment using the three-dimensional reconstruction;

calculating projection errors associated with each of the plurality of potential input planes identified in the real-world environment;

selecting an input plane, from the plurality of potential input planes, associated with a physical surface in the real-world environment, based on determining that the input plane minimizes one or more of the projection errors;

projecting the positions onto the input plane;

monitoring an input state with respect to the input plane to:

identify when the user is providing user input via the input plane, and

differentiate between ongoing user input and finalized user input;

recording, based on the monitoring of the input state, one or more of the projected positions as input data;

processing the input data to interpret the user input; and

executing an action based on the interpreted user input.

20 . One or more non-transitory computer-readable storage media, the one or more non-transitory computer-readable storage media including instructions that when executed by at least one processor of an extended reality (XR) device cause the XR device to perform operations comprising:

generating a three-dimensional reconstruction of a real-world environment;

tracking a hand of a user across multiple image frames captured by the XR device to obtain positions of one or more parts of the hand in the real-world environment;

identifying a plurality of potential input planes in the real-world environment using the three-dimensional reconstruction;

calculating projection errors associated with each of the plurality of potential input planes identified in the real-world environment;

selecting an input plane, from the plurality of potential input planes, associated with a physical surface in the real-world environment, based on determining that the input plane minimizes one or more of the projection errors;

projecting the positions onto the input plane;

monitoring an input state with respect to the input plane to:

identify when the user is providing user input via the input plane, and

differentiate between ongoing user input and finalized user input;

recording, based on the monitoring of the input state, one or more of the projected positions as input data;

processing the input data to interpret the user input; and

executing an action based on the interpreted user input.