Granted Patent B2

US 12,645,340 · App. 17/950,770 · Granted Jun 2, 2026

Inventors: Benjamin R. Blachnitzky (San Francisco, CA); Aaron M. Burns (Sunnyvale, CA); Anette L. Freiin von Kapri (Mountain View, CA); Arun Rakesh Yoganandan (San Francisco, CA); Benjamin H. Boesel (Sunnyvale, CA); Evgenii Krivoruchko (Cupertino, CA); Jonathan Ravasz (Sunnyvale, CA); Shih-Sang Chiu (Menlo Park, CA); Stephen O. Lemay (Palo Alto, CA)

Assignee: APPLE INC.

G06F3/04815G06F3/013G06T19/00

Abstract

Claims (60)

1 . A method comprising:

at an electronic device including a non-transitory memory, one or more processors, a display, and one or more input devices:

detecting a first peripheral device at a first location within a three-dimensional (3D) environment via a computer vision technique;

after detecting the first peripheral device at the first location, receiving, via the one or more input devices, a first user input that is directed to the first location within the 3D environment;

in response to receiving the first user input that is directed to the first location within the 3D environment, displaying, on the display, a first pairing affordance that is world-locked to the first peripheral device within the 3D environment;

after displaying the first pairing affordance that is world-locked to the first peripheral device within the 3D environment, receiving, via the one or more input devices, a second user input that indicates that attention of the user is directed away from the first location within the 3D environment; and

in response to receiving the second user input that indicates that attention of the user is directed away from the first location within the 3D environment, ceasing to display the first pairing affordance that is world-locked to the first peripheral device within the 3D environment.

2 . The method of claim 1 , wherein the one or more input devices includes an eye tracker, wherein receiving the first user input includes receiving first eye tracking data from the eye tracker, wherein the first eye tracking data indicates a first gaze that is directed to the first peripheral device, and wherein receiving the second user input includes receiving second eye tracking data that indicates a second gaze that is directed away from the first peripheral device.

3 . The method of claim 1 , further comprising:

after ceasing to display the first pairing affordance, receiving, via the one or more input devices, a third user input that is directed to the first location within the 3D environment;

in response to receiving the third user input, redisplaying, on the display, the first pairing affordance;

receiving, via the one or more input devices, a fourth user input that is directed to the first pairing affordance; and

in response to receiving the fourth user input, pairing the electronic device with the first peripheral device.

4 . The method of claim 3 , further comprising, in response to pairing the electronic device with the first peripheral device, displaying, on the display, a status indicator that is associated with the first peripheral device.

5 . The method of claim 3 , further comprising, in response to pairing the electronic device with the first peripheral device, displaying, on the display, a computer-generated representation of a numeric keypad near the first peripheral device.

6 . The method of claim 3 , wherein the electronic device includes a communication interface provided to communicate with the first peripheral device while the electronic device is paired with the first peripheral device.

7 . The method of claim 6 , further comprising, while the electronic device is paired with the first peripheral device:

receiving, via the communication interface, sensor data from the first peripheral device; and

performing a content manipulation operation based on the sensor data.

8 . The method of claim 7 , wherein the first peripheral device corresponds to a keyboard device including a plurality of keys, wherein the sensor data indicates a first keypress of a first key of the plurality of keys.

9 . The method of claim 7 , further comprising, while the electronic device is paired with the first peripheral device, displaying a content manipulation region on the display, and wherein performing the content manipulation operation includes manipulating content within the content manipulation region.

10 . The method of claim 9 , further comprising displaying, on the display, one or more application affordances near the first peripheral device, wherein displaying the content manipulation region is in response to detecting, via the one or more input devices, a fifth user input directed to a first application affordance of the one or more application affordances.

11 . The method of claim 1 , wherein displaying the first pairing affordance world-locked to the first peripheral device includes displaying the first pairing affordance at a first position on the display, the method further comprising:

detecting a change of a viewpoint of a user of the electronic device; and

in response to detecting the change of the viewpoint of the user of the electronic device, moving the first pairing affordance from the first position to a second position on the display in order to maintain the first pairing affordance world-locked to the first peripheral device.

12 . The method of claim 1 , further comprising detecting a physical surface associated with the first peripheral device, wherein displaying the first pairing affordance world-locked to the first peripheral device includes displaying the first pairing affordance as substantially parallel with respect to the physical surface.

13 . The method of claim 1 , further comprising:

detecting a second peripheral device at a second location within the 3D environment via the computer vision technique,

wherein the second user input that indicates that attention of the user is directed away from the first location within the 3D environment is directed to the second peripheral device at the second location within the 3D environment, and wherein ceasing to display the first pairing affordance that is world-locked is based on the second user input is directed to the second peripheral device at the second location within the 3D environment; and

in response to receiving the second user input, displaying, on the display, a second pairing affordance that is world-locked to the second peripheral device within the 3D environment.

14 . The method of claim 13 , further comprising:

receiving, via the one or more input devices, a third user input that is directed to the second pairing affordance; and

in response to receiving the third user input, pairing the electronic device with the second peripheral device.

15 . The method of claim 1 , wherein the second user input that indicates that attention of the user is directed away from the first location within the 3D environment causes the first peripheral device to exit a field of view of an image sensor.

16 . The method of claim 1 , wherein the second user input includes a combination of an extremity directed to the first pairing affordance with a gaze directed away from the first pairing affordance.

17 . An electronic device comprising:

one or more processors;

a non-transitory memory;

a display;

one or more input devices; and

one or more programs, wherein the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

detecting a first peripheral device at a first location within a 3D environment via a computer vision technique;

after detecting the first peripheral device at the first location, receiving, via the one or more input devices, a first user input that is directed to the first location within the 3D environment;

in response to receiving the first user input that is directed to the first location within the 3D environment, displaying, on the display, a first pairing affordance that is world-locked to the first peripheral device within the 3D environment;

after displaying the first pairing affordance that is world-locked to the first peripheral device within the 3D environment, receiving, via the one or more input devices, a second user input that indicates that attention of the user is directed away from the first location within the 3D environment; and

in response to receiving the second user input that indicates that attention of the user is directed away from the first location within the 3D environment, ceasing to display the first pairing affordance that is world-locked to the first peripheral device within the 3D environment.

18 . The electronic device of claim 17 , wherein the one or more programs include instructions for:

after ceasing to display the first pairing affordance, receiving, via the one or more input devices, a third user input that is directed to the first location within the 3D environment;

in response to receiving the third user input, redisplaying, on the display, the first pairing affordance;

receiving, via the one or more input devices, a fourth user input that is directed to the first pairing affordance; and

in response to receiving the fourth user input, pairing the electronic device with the first peripheral device.

19 . The electronic device of claim 18 , wherein the one or more programs include instructions for, while the electronic device is paired with the first peripheral device:

receiving, via a communication interface, sensor data from the first peripheral device; and

performing a content manipulation operation based on the sensor data.

20 . A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with one or processors, a display, and one or more input devices, cause the electronic device to:

detect a first peripheral device at a first location within a 3D environment via a computer vision technique;

after detecting the first peripheral device at the first location, receive, via the one or more input devices, a first user input that is directed to the first location within the 3D environment;

in response to receiving the first user input that is directed to the first location within the 3D environment, display, on the display, a first pairing affordance that is world-locked to the first peripheral device within the 3D environment;

after displaying the first pairing affordance that is world-locked to the first peripheral device within the 3D environment, receiving, via the one or more input devices, a second user input that indicates that attention of the user is directed away from the first location within the 3D environment; and

in response to receiving the second user input that indicates that attention of the user is directed away from the first location within the 3D environment, ceasing to display the first pairing affordance that is world-locked to the first peripheral device within the 3D environment.

Assignments (3)

Continuity (3)

References Cited (12)