Method and device for surfacing and detaching a tools palette

1 . A method comprising:

at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a display device and one or more input devices via a communication interface:

detecting, via the one or more input devices, a control device within a physical environment;

detecting, via the one or more input devices, a first user input that corresponds to surfacing a plurality of selectable UI elements associated with a function or a usage of the control device, wherein the first user input comprises a first touch input directed to a surface disposed on a barrel of the control device;

in response to detecting the first user input, presenting, via the display device, the plurality of selectable UI elements associated with the function or the usage of the control device in a first location and a first orientation relative to the control device according to a first visual modality, wherein the first visual modality is an object locked mode and the first orientation is parallel to a longitudinal dimension of the control device, and wherein the plurality of selectable UI elements is anchored to the control device in the first location and the first orientation while presented in the first visual modality;

while presenting the plurality of selectable UI elements according to the first visual modality, detecting, via the one or more input devices, a second user input that corresponds to detaching the plurality of selectable UI elements, wherein the second user input comprises a second touch input directed to the surface disposed on the barrel of the control device;

in response to detecting the second user input, presenting, via the display device, the plurality of selectable UI elements associated with the function or the usage of the control device at a second location and in a second orientation according to a second visual modality different from the first visual modality, wherein the plurality of selectable UI elements are anchored to a point within the physical environment at the second location and the second orientation while presented in the second visual modality, and wherein the second location corresponds to a point within the physical environment associated with a position of a tip of the control device when the second user input was detected;

while presenting the plurality of selectable UI elements at the second location and in the second orientation according to the second visual modality, detecting, via the one or more input devices, a third user input that comprises a third touch input directed to the surface disposed on the barrel of the control device; and

in response to detecting the third user input, presenting, via the display device, the plurality of selectable UI elements at the first location and in the first orientation relative to the control device according to the first visual modality, wherein the plurality of selectable UI elements is anchored to the control device in the first location and the first orientation while presented in the first visual modality.

2 . The method of claim 1 , further comprising:

while presenting the plurality of selectable UI elements according to the first visual modality:

detecting movement of the control device within the physical environment; and

in response to detecting movement of the control device, updating presentation of the plurality of selectable UI elements to maintain the first location and the first orientation relative to the control device.

3 . The method of claim 1 , wherein the first visual modality corresponds to an object-locked mode relative to the control device, and wherein the second visual modality corresponds to one of a head-locked mode or a world-locked mode relative to the point within the physical environment.

4 . The method of claim 1 , wherein the plurality of selectable UI elements includes a selectable undo affordance, a selectable redo affordance, a drawing tools accelerator affordance for presenting a drawing tools menu, and a color picker affordance for presenting a color picker menu for an output color of the control device.

5 . The method of claim 1 , wherein the second orientation is perpendicular to a head pose.

6 . The method of claim 1 , wherein the first location corresponds to a spatial offset relative to a centroid of the control device, and wherein the second location corresponds to a point within the physical environment.

7 . The method of claim 1 , wherein the second location corresponds to a point within the physical environment associated with a position of a portion of the control device when the second user input was detected.

8 . The method of claim 1 , further comprising:

while presenting the plurality of selectable UI elements according to the second visual modality, maintaining presentation of the selectable UI elements at the second location when the control device moves.

9 . The method of claim 1 , further comprising:

while presenting the plurality of selectable UI elements at the first location and in the first orientation according to the first visual modality:

detecting a fourth user input directed to a respective selectable UI element among the plurality of selectable UI elements; and

in response to detecting the fourth user input, updating a respective function or usage of the control device based on the respective selectable UI element selected with the fourth user input.

10 . The method of claim 1 , further comprising:

while presenting the plurality of selectable UI elements at the second location and in the second orientation according to the second visual modality:

detecting a fourth user input directed to a respective selectable UI element among the plurality of selectable UI elements; and

in response to detecting the fourth user input, updating a respective function or usage of the control device based on the respective selectable UI element selected with the fourth user input.

11 . The method of claim 1 , further comprising:

while presenting the plurality of selectable UI elements according to the first visual modality, detecting a fifth user input; and

in response to detecting the fifth user input, ceasing presentation of the plurality of selectable UI elements.

12 . The method of claim 1 , wherein the first user input corresponds to selecting a virtual affordance within an extended reality environment with a gaze input or a hand tracking input to surface the plurality of selectable UI elements.

13 . The method of claim 1 , wherein the first user input corresponds to a tap input directed to a portion of a touch-sensitive surface (TSS) of the control device associated with surfacing the plurality of selectable UI elements.

14 . The method of claim 1 , wherein the first user input corresponds to a voice command associated with surfacing the plurality of selectable UI elements.

15 . The method of claim 1 , wherein the first user input corresponds to gazing at a virtual sketchpad or input region within an extended reality environment.

16 . The method of claim 1 , wherein the second user input corresponds to a double tap input directed to a portion of a touch-sensitive surface (TSS) of the control device associated with detaching the plurality of selectable UI elements.

17 . The method of claim 1 , wherein the second user input corresponds to an outward slide gesture directed to a portion of a touch-sensitive surface (TSS) of the control device associated with detaching the plurality of selectable UI elements.

18 . The method of claim 1 , wherein the second user input corresponds to a voice command associated with detaching the plurality of selectable UI elements.

19 . The method of claim 1 , wherein detecting the control device includes performing six-degree of freedom (6DOF) tracking on the control device based on at least one of one or more computer vision (CV) techniques, hand/extremity tracking data, magnetic tracking data, inertial measurement unit (IMU) data, accelerometer data, or gyroscope data.

20 . The method of claim 1 , wherein the control device corresponds to an electronic device in communication with the computing system.

21 . The method of claim 1 , wherein the control device corresponds to a physical object that cannot communicate with the computing system.

22 . The method of claim 1 , wherein the second user input is performed on a touch-sensitive surface (TSS) of the control device and the third user input is an inverse of the second user input.

23 . A device comprising:

one or more processors;

a non-transitory memory;

an interface for communicating with a display device, one or more audio output devices, and one or more input devices; and

one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to:

detect, via the one or more input devices, a control device within a physical environment;

detect, via the one or more input devices, a first user input that corresponds to surfacing a plurality of selectable UI elements associated with a function or a usage of the control device, wherein the first user input comprises a first touch input directed to a surface disposed on a barrel of the control device;

in response to detecting the first user input, present, via the display device, the plurality of selectable UI elements associated with the function or the usage of the control device in a first location and a first orientation relative to the control device according to a first visual modality, wherein the first visual modality is an object locked mode and the first orientation is parallel to a longitudinal dimension of the control device, and wherein the plurality of selectable UI elements is anchored to the control device in the first location and the first orientation while presented in the first visual modality;

while presenting the plurality of selectable UI elements according to the first visual modality, detect, via the one or more input devices, a second user input that corresponds to detaching the plurality of selectable UI elements, wherein the second user input comprises a second touch input directed to the surface disposed on the barrel of the control device;

in response to detecting the second user input, present, via the display device, the plurality of selectable UI elements associated with the function or the usage of the control device at a second location and in a second orientation according to a second visual modality different from the first visual modality, wherein the plurality of selectable UI elements are anchored to a point within the physical environment at the second location and the second orientation while presented in the second visual modality, and wherein the second location corresponds to a point within the physical environment associated with a position of a tip of the control device when the second user input was detected;

while presenting the plurality of selectable UI elements at the second location and in the second orientation according to the second visual modality, detecting, via the one or more input devices, a third user input that comprises a third touch input directed to the surface disposed on the barrel of the control device; and

in response to detecting the third user input, presenting, via the display device, the plurality of selectable UI elements at the first location and in the first orientation relative to the control device according to the first visual modality, wherein the plurality of selectable UI elements is anchored to the control device in the first location and the first orientation while presented in the first visual modality.

24 . A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with an interface for communicating with a display device, one or more audio output devices, and one or more input devices, cause the device to:

detect, via the one or more input devices, a control device within a physical environment;

detect, via the one or more input devices, a first user input that corresponds to surfacing a plurality of selectable UI elements associated with a function or a usage of the control device, wherein the first user input comprises a first touch input directed to a surface disposed on a barrel of the control device;

in response to detecting the first user input, present, via the display device, the plurality of selectable UI elements associated with the function or the usage of the control device in a first location and a first orientation relative to the control device according to a first visual modality, wherein the first visual modality is an object locked mode and the first orientation is parallel to a longitudinal dimension of the control device, and wherein the plurality of selectable UI elements is anchored to the control device in the first location and the first orientation while presented in the first visual modality;

while presenting the plurality of selectable UI elements according to the first visual modality, detect, via the one or more input devices, a second user input that corresponds to detaching the plurality of selectable UI elements, wherein the second user input comprises a second touch input directed to the surface disposed on the barrel of the control device;

in response to detecting the second user input, present, via the display device, the plurality of selectable UI elements associated with the function or the usage of the control device at a second location and in a second orientation according to a second visual modality different from the first visual modality, wherein the plurality of selectable UI elements are anchored to a point within the physical environment at the second location and the second orientation while presented in the second visual modality, and wherein the second location corresponds to a point within the physical environment associated with a position of a tip of the control device when the second user input was detected;

while presenting the plurality of selectable UI elements at the second location and in the second orientation according to the second visual modality, detecting, via the one or more input devices, a third user input that comprises a third touch input directed to the surface disposed on the barrel of the control device; and

in response to detecting the third user input, presenting, via the display device, the plurality of selectable UI elements at the first location and in the first orientation relative to the control device according to the first visual modality, wherein the plurality of selectable UI elements is anchored to the control device in the first location and the first orientation while presented in the first visual modality.