IP Library › Granted Patent US 12,710,807
Granted Patent B2
US 12,710,807 · App. 18/823,412 · Granted Aug 18, 2026

Dynamically orientated labels for XR user interfaces

Inventors: Viktoria Hwang (Monrovia, CA); James Powderly (Venice, CA); Karen Stolzenberg (Venice, CA); Mathieu Emmanuel Vignau (Los Angeles, CA)
Assignee: Snap Inc.
G06F3/011G06F3/017G06F3/04815
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,710,807
App. No.
18/823,412
Granted
Aug 18, 2026
Kind
B2
Abstract

An extended Reality (XR) system is provided that enhances user interaction within XR environments. The XR system captures tracking data using one or more sensors, including hand tracking data of a user's hand and pose data of the XR system itself. By continuously capturing this data, the XR system dynamically generates a hand-located user interface that includes interactive virtual objects associated with specific locations on the surface of the user's hand. Additionally, the XR system generates labels for the interactive virtual objects that are dynamically oriented toward the user as the user moves their hands and head when interacting with an XR environment.

Claims (46)

1 . A machine-implemented method, comprising:

capturing, using one or more sensors of an extended Reality (XR) system, tracking data of a user, the tracking data including hand tracking data of a hand of the user and pose data of the XR system;

while continuously capturing the tracking data and the pose data, performing operations comprising:

generating, using the tracking data, a hand-located user interface including an interactive virtual object associated with a location on a surface of the hand;

generating, using the interactive virtual object and the pose data, a label associated with the interactive virtual object, the label orientated to a viewpoint of the user;

continuously adjusting an orientation of the label based on changes of the viewpoint of the user and a position of the hand to maintain readability of the label regardless of hand movement or head movement of the user by counteracting rotation of the hand by applying an inverse of a hand orientation quaternion derived from the hand tracking data to a label orientation quaternion of the label to maintain the orientation of the label relative to a visual axis of the viewpoint of the user; and

providing the hand-located user interface to the user.

2 . The machine-implemented method of claim 1 , wherein the surface is a dorsal surface of the hand.

3 . The machine-implemented method of claim 1 , wherein the surface is a palmar surface of the hand.

4 . The machine-implemented method of claim 1 , further comprising:

measuring, using the tracking data, a distance between a first landmark on the hand and a second landmark on the hand; and

adjusting a size of the interactive virtual object using the distance.

5 . The machine-implemented method of claim 4 , wherein the first landmark is a wrist landmark and the second landmark is a middle knuckle landmark.

6 . The machine-implemented method of claim 4 , wherein the size is adjusted in steps using a fixed interval.

7 . The machine-implemented method of claim 1 , wherein the XR system is a head-wearable apparatus.

8 . A machine comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, cause the machine to perform operations comprising:

capturing, using one or more sensors of an extended Reality (XR) system, tracking data of a user, the tracking data including hand tracking data of a hand of the user and pose data of the XR system;

while continuously capturing the tracking data and the pose data, performing operations comprising:

generating, using the tracking data, a hand-located user interface including an interactive virtual object associated with a location on a surface of the hand;

generating, using the interactive virtual object and the pose data, a label associated with the interactive virtual object, the label orientated to a viewpoint of the user;

continuously adjusting an orientation of the label based on changes of the viewpoint of the user and a position of the hand to maintain readability of the label regardless of hand movement or head movement of the user by counteracting rotation of the hand by applying an inverse of a hand orientation quaternion derived from the hand tracking data to a label orientation quaternion of the label to maintain the orientation of the label relative to a visual axis of the viewpoint of the user; and

providing the hand-located user interface to the user.

9 . The machine of claim 8 , wherein the surface is a dorsal surface of the hand.

10 . The machine of claim 8 , wherein the surface is a palmar surface of the hand.

11 . The machine of claim 8 , wherein the operations further comprise:

measuring, using the tracking data, a distance between a first landmark on the hand and a second landmark on the hand; and

adjusting a size of the interactive virtual object using the distance.

12 . The machine of claim 11 , wherein the first landmark is a wrist landmark and the second landmark is a middle knuckle landmark.

13 . The machine of claim 11 , wherein the size is adjusted in steps using a fixed interval.

14 . The machine of claim 8 , wherein the XR system is a head-wearable apparatus.

15 . A machine-storage medium, the machine-storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:

capturing, using one or more sensors of an extended Reality (XR) system, tracking data of a user, the tracking data including hand tracking data of a hand of the user and pose data of the XR system;

while continuously capturing the tracking data and the pose data, performing operations comprising:

generating, using the tracking data, a hand-located user interface including an interactive virtual object associated with a location on a surface of the hand;

generating, using the interactive virtual object and the pose data, a label associated with the interactive virtual object, the label orientated to a viewpoint of the user;

continuously adjusting an orientation of the label based on changes of the viewpoint of the user and a position of the hand to maintain readability of the label regardless of hand movement or head movement of the user by counteracting rotation of the hand by applying an inverse of a hand orientation quaternion derived from the hand tracking data to a label orientation quaternion of the label to maintain the orientation of the label relative to a visual axis of the viewpoint of the user; and

providing the hand-located user interface to the user.

16 . The machine-storage medium of claim 15 , wherein the surface is a dorsal surface of the hand.

17 . The machine-storage medium of claim 15 , wherein the surface is a palmar surface of the hand.

18 . The machine-storage medium of claim 15 , wherein the operations further comprise:

measuring, using the tracking data, a distance between a first landmark on the hand and a second landmark on the hand; and

adjusting a size of the interactive virtual object using the distance.

19 . The machine-storage medium of claim 18 , wherein the first landmark is a wrist landmark and the second landmark is a middle knuckle landmark.

20 . The machine-storage medium of claim 15 , wherein the XR system is a head-wearable apparatus.