IP Library › Granted Patent US 12688657

Granted Patent B2

US 12688657 · App. 18/437,634 · Granted Jul 21, 2026

Systems, methods, and graphical user interfaces for applying virtual effects in three-dimensional environments

Inventors: Joanna M. Newman (San Jose, CA); Daniel Escudero (San Francisco, CA); Corbin F. Peters (San Mateo, CA); Joseph-Alexander P. Weil (San Anselmo, CA); Juan Gril (San Francisco, CA); Jose Moises Tafolla Carmona (San Jose, CA)

Assignee: APPLE INC.

G06T19/006 G06T13/40 G06V20/44 G06V40/20

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12688657
App. No.: 18/437,634
Granted: Jul 21, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A computer system, in response to detecting the occurrence of a first event, applies a first virtual effect to the representation of the field of view of the one or more cameras, including: applying the first virtual effect with a first set of values for a first characteristic of the first virtual effect, wherein the first set of values for the first characteristic of the first virtual effect are selected in accordance with segmentation of the view of the first physical object from the view of the portion of the first physical environment; and applying the first virtual effect with a second set of values for a second characteristic of the first virtual effect, wherein the second set of values for the second characteristic of the first virtual effect are selected in accordance with tracking of movement of one or more portions of the first physical object.

Claims (135)

1 . A method, comprising:

at a computer system in communication with a display generation component and one or more cameras:

while a representation of a field of view of the one or more cameras, including a view of a first physical object that moves in a portion of a first physical environment, is visible via the display generation component, detecting occurrence of a first event;

in response to detecting the occurrence of the first event, applying a first virtual effect to the representation of the field of view of the one or more cameras, including:

applying the first virtual effect with a first set of values for a first characteristic of the first virtual effect, wherein the first set of values for the first characteristic of the first virtual effect are selected in accordance with segmentation of the view of the first physical object from the view of the portion of the first physical environment; and

applying the first virtual effect with a second set of values for a second characteristic of the first virtual effect, wherein the second set of values for the second characteristic of the first virtual effect are selected in accordance with tracking of movement of one or more portions of the first physical object in the portion of the first physical environment.

2 . The method of claim 1 , including:

detecting first movement of the one or more portions of the first physical object in the portion of the first physical environment from a first position to a second position, wherein applying the first virtual effect with the second set of values for the second characteristic of the first virtual effect includes:

in response to detecting the first movement of the one or more portions of the first physical object from the first position to the second position, displaying animated movements of the first virtual effect in accordance with the first movement of the one or more portions of the first physical object from the first position to the second position.

3 . The method of claim 1 , including:

detecting movement of the one or more portions of the first physical object in the portion of the first physical environment from a first pose to a second pose, wherein applying the first virtual effect with the second set of values for the second characteristic of the first virtual effect includes:

in response to detecting the movement of the one or more portions of the first physical object from the first pose to the second pose, updating display of the first virtual effect in accordance with the movement of the one or more portions of the first physical object from the first pose to the second pose.

4 . The method of claim 1 , wherein applying the first virtual effect with the first set of values for the first characteristic of the first virtual effect includes:

displaying one or more virtual background elements in the view of the portion of the first physical environment, and

forgoing display, based on the segmentation of the view of the first physical object from the view of the portion of the first physical environment, of one or more portions of the one or more virtual background elements in the view of the portion of the first physical environment in accordance with a determination that respective positions of the one or more portions of the one or more virtual background elements are behind the first physical object relative to a current viewpoint of a user that corresponds to the view of the portion of the first physical environment.

5 . The method of claim 1 , wherein:

the first virtual effect includes a first set of one or more virtual elements; and

applying the first virtual effect with the second set of values for the second characteristic of the first virtual effect includes displaying animated movements of the first set of one or more virtual elements, in the view of the portion of the first physical environment, that follow tracked movement of the one or more portions of the first physical object.

6 . The method of claim 5 , wherein displaying the animated movements of the first set of one or more virtual elements that follow the tracked movement of the one or more portions of the first physical object further includes:

forgoing display, based on the segmentation of the view of the first physical object from the view of the portion of the first physical environment, of one or more portions of the first set of one or more virtual elements in the view of the portion of the first physical environment in accordance with a determination that respective positions of the one or more portions of the first set of one or more virtual elements are behind the first physical object relative to a current viewpoint of a user that corresponds to the view of the portion of the first physical environment.

7 . The method of claim 5 , wherein the first set of the one or more virtual elements that follow the tracked movement of the one or more portions of the first physical object includes a plurality of virtual elements that move away from the one or more portions of the first physical object during the tracked movement of the one or more portions of the first physical object.

8 . The method of claim 5 , wherein the first set of the one or more virtual elements that follow the tracked movement of the one or more portions of the first physical object includes a set of one or more virtual outlines having respective shapes that correspond to a shape of the one or more portions of the first physical object.

9 . The method of claim 1 , including:

detecting movement of at least a first portion of the first physical object relative to the first physical environment; and

in response to detecting the movement of at least the first portion of the first physical object, in accordance with a determination that the movement satisfies motion criteria, updating the first set of values for the first characteristic of the first virtual effect to an updated first set of values for the first characteristic, wherein the updated first set of values for the first characteristic are selected at least in part in accordance with the movement of at least the first portion of the first physical object that satisfies the motion criteria.

10 . The method of claim 9 , wherein detecting the movement of at least the first portion of the first physical object that satisfies the motion criteria includes detecting a first gesture of the first physical object formed by the movement of at least the first portion of the first physical object.

11 . The method of claim 9 , wherein detecting the movement of at least the first portion of the first physical object that satisfies the motion criteria includes detecting that the movement of at least the first portion of the first physical object causes the at least the first portion of the first physical object to come into contact or break contact with one or more physical surfaces in the first physical environment.

12 . The method of claim 1 , wherein:

applying the first virtual effect with the first set of values for the first characteristic of the first virtual effect includes displaying a second set of one or more virtual elements in the view of the portion of the first physical environment, wherein the first set of values for the first characteristic of the first virtual effect defines respective first positions at which the second set of one or more virtual elements are displayed within the view of the portion of the first physical environment in accordance with respective current positions of the one or more portions of the first physical object, and defines which portions of the second set of one or more virtual elements are occluded by the one or more portions of the first physical object in the view of the portion of the first physical environment; and

the method includes:

detecting movement of at least a portion of the one or more portions of the first physical object in the portion of the first physical environment; and

in response to detecting the movement of the at least a portion of the first physical object in the portion of the first physical environment, updating the first set of values for the first characteristic of the first virtual effect to define respective second positions at which the second set of one or more virtual elements are displayed within the view of the portion of the first physical environment in accordance with the respective current positions of the one or more portions of the first physical object, and to update which portions of the second set of one or more virtual elements are occluded by the one or more portions of the first physical object in the view of the portion of the first physical environment.

13 . The method of claim 1 , wherein:

applying the first virtual effect with the first set of values for the first characteristic of the first virtual effect includes displaying one or more textual elements in the view of the portion of the first physical environment, wherein the first set of values for the first characteristic of the first virtual effect defines a respective position at which the textual element is displayed within the view of the portion of the first physical environment in accordance with segmentation of the view of the first physical object from the view of the portion of the first physical environment.

14 . The method of claim 13 , wherein the respective position at which the textual element is displayed within the view of the portion of the first physical environment is selected in accordance with a requirement that the textual element displayed at the respective position would not intersect with the view of the first physical object in the view of the portion of the first physical environment.

15 . The method of claim 1 , wherein:

the representation of the field of view of the one or more cameras includes a second physical object, distinct from the first physical object, that moves in the portion of the first physical environment; and

the method includes:

applying the first virtual effect with a third set of values for a third characteristic of the first virtual effect, wherein the third set of values for the third characteristic of the first virtual effect are selected in accordance with segmentation of the view of the second physical object from the view of the portion of the first physical environment; and

applying the first virtual effect with a fourth set of values for a fourth characteristic of the first virtual effect, wherein the fourth set of values for the fourth characteristic of the first virtual effect are selected in accordance with tracking of movement of one or more portions of the second physical object in the portion of the first physical environment.

16 . The method of claim 1 , wherein applying the first virtual effect includes displaying a first virtual object at a respective position, relative to the view of the first physical environment, that corresponds to a first portion of the first physical object, wherein at least a portion of an outline of a visible portion of the first virtual object is based on an outline of the view of the first physical object that is determined based on segmenting the view of the first physical object from the view of the portion of the first physical environment.

17 . The method of claim 1 , wherein:

applying the first virtual effect includes displaying a second virtual object at a respective position, relative to the view of the first physical environment, that is separated from the view of the first physical object, and

the method includes:

detecting movement of at least one of the one or more portions of the first physical object in the portion of the first physical environment, wherein applying the first virtual effect with the second set of values for the second characteristic of the first virtual effect includes, in response to detecting the movement of the at least one of the one or more portions of the first physical object, animating the second virtual object to move the second virtual object away from the at least one portion of the one or more portions of the first physical object in accordance with motion tracking of the at least one portion of the one or more portions of the first physical object.

18 . The method of claim 1 , including:

while applying the first virtual effect to the representation of the field of view of the one or more cameras, detecting the occurrence of a second event;

in response to detecting the occurrence of the second event:

ceasing to apply the first virtual effect to the representation of the field of view of the one or more cameras; and

applying a fourth virtual effect, distinct from the first virtual effect to the representation of the field of view of the one or more cameras.

19 . A computer system, comprising:

a display generation component;

one or more cameras;

one or more processors; and

memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for:

in response to detecting the occurrence of the first event, applying a first virtual effect to the representation of the field of view of the one or more cameras, including:

20 . The computer system of claim 19 , the one or more programs further including instructions for:

21 . The computer system of claim 19 , the one or more programs further including instructions for:

22 . The computer system of claim 19 , wherein applying the first virtual effect with the first set of values for the first characteristic of the first virtual effect includes:

displaying one or more virtual background elements in the view of the portion of the first physical environment, and

23 . The computer system of claim 19 , wherein:

the first virtual effect includes a first set of one or more virtual elements; and

24 . The computer system of claim 19 , the one or more programs further including instructions for:

detecting movement of at least a first portion of the first physical object relative to the first physical environment; and

25 . The computer system of claim 19 , wherein:

the one or more programs further include instructions for:

detecting movement of at least a portion of the one or more portions of the first physical object in the portion of the first physical environment; and

26 . The computer system of claim 19 , wherein:

27 . The computer system of claim 19 , wherein:

the one or more programs further include instructions for:

28 . The computer system of claim 19 , wherein applying the first virtual effect includes displaying a first virtual object at a respective position, relative to the view of the first physical environment, that corresponds to a first portion of the first physical object, wherein at least a portion of an outline of a visible portion of the first virtual object is based on an outline of the view of the first physical object that is determined based on segmenting the view of the first physical object from the view of the portion of the first physical environment.

29 . The computer system of claim 19 , wherein applying the first virtual effect includes displaying a second virtual object at a respective position, relative to the view of the first physical environment, that is separated from the view of the first physical object, and the one or more programs further include instructions for:

30 . The computer system of claim 19 , the one or more programs further including instructions for:

while applying the first virtual effect to the representation of the field of view of the one or more cameras, detecting the occurrence of a second event;

in response to detecting the occurrence of the second event:

ceasing to apply the first virtual effect to the representation of the field of view of the one or more cameras; and

applying a fourth virtual effect, distinct from the first virtual effect to the representation of the field of view of the one or more cameras.

31 . A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system that is in communication with a display generation component and one or more cameras, cause the computer system to:

in response to detecting the occurrence of the first event, apply a first virtual effect to the representation of the field of view of the one or more cameras, including:

32 . The non-transitory computer readable storage medium of claim 31 , the one or more programs further comprising instructions that cause the computer system to:

detect first movement of the one or more portions of the first physical object in the portion of the first physical environment from a first position to a second position, wherein applying the first virtual effect with the second set of values for the second characteristic of the first virtual effect includes:

33 . The non-transitory computer readable storage medium of claim 31 , the one or more programs further comprising instructions that cause the computer system to:

detect movement of the one or more portions of the first physical object in the portion of the first physical environment from a first pose to a second pose, wherein applying the first virtual effect with the second set of values for the second characteristic of the first virtual effect includes:

34 . The non-transitory computer readable storage medium of claim 31 , wherein applying the first virtual effect with the first set of values for the first characteristic of the first virtual effect includes:

displaying one or more virtual background elements in the view of the portion of the first physical environment, and

35 . The non-transitory computer readable storage medium of claim 31 , wherein:

the first virtual effect includes a first set of one or more virtual elements; and

36 . The non-transitory computer readable storage medium of claim 31 , the one or more programs further comprising instructions that cause the computer system to:

detect movement of at least a first portion of the first physical object relative to the first physical environment; and

in response to detecting the movement of at least the first portion of the first physical object, in accordance with a determination that the movement satisfies motion criteria, update the first set of values for the first characteristic of the first virtual effect to an updated first set of values for the first characteristic, wherein the updated first set of values for the first characteristic are selected at least in part in accordance with the movement of at least the first portion of the first physical object that satisfies the motion criteria.

37 . The non-transitory computer readable storage medium of claim 31 , wherein:

the one or more programs further comprise instructions that cause the computer system to:

detect movement of at least a portion of the one or more portions of the first physical object in the portion of the first physical environment; and

in response to detecting the movement of the at least a portion of the first physical object in the portion of the first physical environment, update the first set of values for the first characteristic of the first virtual effect to define respective second positions at which the second set of one or more virtual elements are displayed within the view of the portion of the first physical environment in accordance with the respective current positions of the one or more portions of the first physical object, and to update which portions of the second set of one or more virtual elements are occluded by the one or more portions of the first physical object in the view of the portion of the first physical environment.

38 . The non-transitory computer readable storage medium of claim 31 , wherein:

39 . The non-transitory computer readable storage medium of claim 31 , wherein:

the one or more programs further comprise instructions that cause the computer system to:

apply the first virtual effect with a third set of values for a third characteristic of the first virtual effect, wherein the third set of values for the third characteristic of the first virtual effect are selected in accordance with segmentation of the view of the second physical object from the view of the portion of the first physical environment; and

apply the first virtual effect with a fourth set of values for a fourth characteristic of the first virtual effect, wherein the fourth set of values for the fourth characteristic of the first virtual effect are selected in accordance with tracking of movement of one or more portions of the second physical object in the portion of the first physical environment.

40 . The non-transitory computer readable storage medium of claim 31 , wherein applying the first virtual effect includes displaying a first virtual object at a respective position, relative to the view of the first physical environment, that corresponds to a first portion of the first physical object, wherein at least a portion of an outline of a visible portion of the first virtual object is based on an outline of the view of the first physical object that is determined based on segmenting the view of the first physical object from the view of the portion of the first physical environment.

41 . The non-transitory computer readable storage medium of claim 31 , wherein applying the first virtual effect includes displaying a second virtual object at a respective position, relative to the view of the first physical environment, that is separated from the view of the first physical object, and

the one or more programs further comprise instructions that cause the computer system to:

detect movement of at least one of the one or more portions of the first physical object in the portion of the first physical environment, wherein applying the first virtual effect with the second set of values for the second characteristic of the first virtual effect includes, in response to detecting the movement of the at least one of the one or more portions of the first physical object, animating the second virtual object to move the second virtual object away from the at least one portion of the one or more portions of the first physical object in accordance with motion tracking of the at least one portion of the one or more portions of the first physical object.

42 . The non-transitory computer readable storage medium of claim 31 , the one or more programs further comprising instructions that cause the computer system to:

while applying the first virtual effect to the representation of the field of view of the one or more cameras, detect the occurrence of a second event;

in response to detecting the occurrence of the second event:

cease to apply the first virtual effect to the representation of the field of view of the one or more cameras; and

apply a fourth virtual effect, distinct from the first virtual effect to the representation of the field of view of the one or more cameras.