IP Library › Granted Patent US 10,375,313

Granted Patent B1

US 10,375,313 · App. 16/110,514 · Granted Aug 6, 2019

Creative camera

Inventors: Marcel Van Os (San Francisco, CA); Jessica Aboukasm (San Francisco, CA); David R. Black (San Jose, CA); Robert Chinn (Redwood City, CA); Gregory L. Dudey (Los Gatos, CA); Katherine K. Ernst (Cedar Rapids, IA); Grant Paul (San Francisco, CA); William A. Sorrentino, III (San Francisco, CA); Brian E. Walsh (Sunnyvale, CA); Jean-Francois Albouze (Santa Cruz, CA); Jae Woo Chang (San Jose, CA); Aurelio Guzman (San Jose, CA); Christopher J. Moulios (Palo Alto, CA); Joanna M. Newman (Santa Clara, CA); Nicolas Scapel (Sunnyvale, CA); Joseph-Alexander P. Weil (Greenbrae, CA); Christopher Wilson (San Francisco, CA)

Assignee: Apple Inc.

H04N5/232935 H04N5/272 G06F3/04842 G06F3/04883 G06T7/70

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 10,375,313
App. No.: 16/110,514
Filed: Aug 23, 2018
Granted: Aug 6, 2019
Kind: B1
Examiner: LE, TUAN H
Art Unit: 2699
USPC: 348/333.02

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

The present disclosure generally relates to displaying visual effects in image data. In some examples, visual effects include an avatar displayed on a user's face. In some examples, visual effects include stickers applied to image data. In some examples, visual effects include screen effects. In some examples, visual effects are modified based on depth data in the image data.

Claims (230)

1. An electronic device, comprising:

a camera;

a display apparatus;

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:

displaying, via the display apparatus, a camera user interface, the camera user interface including:

a camera display region including a representation of image data captured via the camera, wherein the representation includes a representation of a subject and a representation of a background, and the image data corresponds to depth data for the subject; and

a first affordance associated with a first camera display mode;

identifying the subject based on the depth data;

while the subject is positioned within a field of view of the camera and the representation of the subject and the representation of the background are displayed in the camera display region, detecting a gesture directed to the first affordance;

in response to detecting the gesture directed to the first affordance, activating the first camera display mode, wherein activating the first camera display mode includes:

displaying an avatar selection region including a plurality of avatar options; and

in response to detecting a selection of one of the plurality of avatar options, displaying a representation of the selected avatar option over a first portion of the representation of the identified subject in the camera display region and while maintaining display of at least a second portion of the representation of the identified subject;

while the first camera display mode is active, detecting a change in pose of the subject;

in response to detecting the change in pose of the subject, changing an appearance of the displayed representation of the selected avatar option based on the detected change in pose of the subject while maintaining display of the second portion of the representation of the identified subject and at least a portion of the representation of the background;

while displaying the representation of the selected avatar option over the first portion of the representation of the identified subject in the camera display region and while maintaining display of at least the second portion of the representation of the identified subject, detecting a gesture directed to a filter affordance that is concurrently displayed on the display with the representation of the selected avatar; and

in response to detecting the gesture directed to the filter affordance, activating a filter mode, wherein activating the filter mode includes:

changing an appearance of the displayed representation of the selected avatar option based on the selected filter; and

changing an appearance of at least a portion of the representation of the background based on the selected filter.

2. The electronic device of claim 1 , wherein activating the first camera display mode further includes:

prior to displaying the representation of the selected avatar option over the first portion of the representation of the subject in the camera display region:

displaying the representation of the subject in the camera display region without displaying the representation of the selected avatar option over the first portion of the representation of the subject.

3. The electronic device of claim 1 , wherein the avatar selection region further includes an option for ceasing to display the representation of the selected avatar option over the first portion of the representation of the subject in the camera display, the one or more programs further including instructions for:

receiving a user input corresponding to selection of the option for ceasing to display the representation of the selected avatar option over the first portion of the representation of the subject in the camera display region; and

in response to receiving a user input corresponding to selection of the option for ceasing to display the representation of the selected avatar option over the first portion of the representation of the subject in the camera display region, ceasing to display the representation of the selected avatar option over the first portion of the representation of the subject in the camera display region.

4. The electronic device of claim 1 , the one or more programs further including instructions for:

detecting a horizontal swipe gesture on the avatar selection region; and

in response to detecting the horizontal swipe gesture, displaying an avatar creation affordance associated with a function for adding a new avatar option to the plurality of avatar options.

5. The electronic device of claim 1 , the one or more programs further including instructions for:

while the first camera display mode is active, detecting a swipe gesture on the avatar selection region; and

in response to detecting the swipe gesture on the avatar selection region, changing an appearance of the displayed representation of the selected avatar option in the camera display region from a first appearance to a second appearance, wherein the second appearance corresponds to a different one of the plurality of avatar options.

6. The electronic device of claim 1 , wherein the representation of image data captured via the camera is a live camera preview.

7. The electronic device of claim 6 , the one or more programs further including instructions for:

while the first camera display mode is active, detecting a swipe gesture on the camera display region; and

in response to detecting the swipe gesture on the camera display region, changing an appearance of the displayed representation of the selected avatar option in the camera display region from a first appearance to a second appearance, wherein the second appearance corresponds to a different one of the plurality of avatar options.

8. The electronic device of claim 1 , wherein activating the first camera display mode further includes:

displaying the selected avatar option with a static appearance in the avatar selection region;

updating the selected avatar option to have a dynamic appearance that changes based on the detected change in pose of the subject; and

displaying an animation of the selected avatar having the dynamic appearance moving from the avatar selection region to the first portion of the representation of the subject in the camera display region.

9. The electronic device of claim 1 , the one or more programs further including instructions for:

while the first camera display mode is active:

in response to a determination that the subject is no longer positioned in the field of view of the camera, displaying an animation of the representation of the selected avatar option moving to a center location in the camera display region.

10. The electronic device of claim 1 , the one or more programs further including instructions for:

while the first camera display mode is active:

in response to a determination that the subject is no longer positioned in the field of view of the camera, modifying the visual appearance of the background displayed in the camera display region.

11. The electronic device of claim 1 , wherein the camera user interface further includes a second affordance associated with a second camera display mode, the one or more programs further including instructions for:

while the subject is positioned within the field of view of the camera and the representation of the subject and the representation of the background are displayed in the camera display region, detecting a gesture directed to the second affordance;

in response to detecting the gesture directed to the second affordance, activating the second camera display mode, wherein activating the second camera display mode includes:

displaying a visual effects selection region including a plurality of graphical objects;

while the second camera display mode is active, detecting a selection of one of the plurality of graphical objects in the visual effects selection region; and

in response to detecting the selection, displaying a representation of the selected graphical object in the camera display region.

12. The electronic device of claim 1 , wherein the representation of image data captured via the camera is a media item, and wherein the camera user interface further includes a third affordance associated with a third camera display mode, the one or more programs further including instructions for:

detecting a gesture directed to the third affordance;

in response to detecting the gesture directed to the third affordance, activating the third camera display mode, wherein activating the third camera display mode includes:

displaying a visual effects selection region including a plurality of graphical objects;

while the third camera display mode is active, detecting a selection of one of the plurality of graphical objects in the visual effects selection region; and

in response to detecting the selection, displaying a representation of the selected graphical object on the media item in the camera display region.

13. The electronic device of claim 1 , wherein the camera user interface further includes a sticker affordance associated with a sticker display mode, the one or more programs further including instructions for:

while displaying the image data in the camera display region, detecting a gesture directed to the sticker affordance;

in response to detecting the gesture directed to the sticker affordance, activating the sticker display mode, wherein activating the sticker display mode includes:

displaying a sticker selection region including a plurality of sticker options;

detecting a selection of one of the plurality of sticker options in the sticker selection region; and

in response to detecting the selection, displaying a representation of the selected sticker option on the image data in the camera display region.

14. The electronic device of claim 13 , the one or more programs further including instructions for:

while displaying the representation of the selected sticker option on the image data in the camera display region:

detecting lateral movement of the subject in the field of view of the camera; and

in response to detecting the lateral movement of the subject in the field of view of the camera, moving the representation of the selected sticker option laterally in accordance with the movement of the subject in the field of view of the camera.

15. The electronic device of claim 13 , the one or more programs further including instructions for:

while displaying the representation of the selected sticker option on the image data in the camera display region:

detecting rotation of the subject in the field of view of the camera; and

in response to detecting the rotation of the subject in the field of view of the camera:

in accordance with a determination that the representation of the selected sticker option has a first relationship to the subject, rotating the representation of the selected sticker option in accordance with a magnitude and direction of the rotation of the subject; and

in accordance with a determination that the representation of the selected sticker option does not have the first relationship to the subject, forgoing rotating the representation of the selected sticker option in accordance with the magnitude and direction of the rotation of the subject.

16. The electronic device of claim 13 , the one or more programs further including instructions for:

while displaying the representation of the selected sticker option on the image data in the camera display region:

detecting movement of the subject toward the camera; and

in response to detecting the movement of the subject toward the camera:

in accordance with a determination that the representation of the selected sticker option has the first relationship to the subject, enlarging the representation of the selected sticker option in accordance with a magnitude of movement of the subject toward the camera; and

in accordance with a determination that the representation of the selected sticker option does not have the first relationship to the subject, forgoing enlarging the representation of the selected sticker option in accordance with the magnitude of movement of the subject toward the camera.

17. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a camera and a display apparatus, the one or more programs including instructions for:

displaying, via the display apparatus, a camera user interface, the camera user interface including:

a first affordance associated with a first camera display mode;

identifying the subject based on the depth data;

in response to detecting the gesture directed to the first affordance, activating the first camera display mode, wherein activating the first camera display mode includes:

displaying an avatar selection region including a plurality of avatar options; and

while the first camera display mode is active, detecting a change in pose of the subject;

in response to detecting the gesture directed to the filter affordance, activating a filter mode, wherein activating the filter mode includes:

changing an appearance of the displayed representation of the selected avatar option based on the selected filter; and

changing an appearance of at least a portion of the representation of the background based on the selected filter.

18. The non-transitory computer-readable storage medium of claim 17 , wherein activating the first camera display mode further includes:

prior to displaying the representation of the selected avatar option over the first portion of the representation of the subject in the camera display region:

19. The non-transitory computer-readable storage medium of claim 17 , wherein the avatar selection region further includes an option for ceasing to display the representation of the selected avatar option over the first portion of the representation of the subject in the camera display, the one or more programs further including instructions for:

20. The non-transitory computer-readable storage medium of claim 17 , the one or more programs further including instructions for:

detecting a horizontal swipe gesture on the avatar selection region; and

in response to detecting the horizontal swipe gesture, displaying an avatar creation affordance associated with a function for adding a new avatar option to the plurality of avatar options.

21. The non-transitory computer-readable storage medium of claim 17 , the one or more programs further including instructions for:

while the first camera display mode is active, detecting a swipe gesture on the avatar selection region; and

22. The non-transitory computer-readable storage medium of claim 17 , wherein the representation of image data captured via the camera is a live camera preview.

23. The non-transitory computer-readable storage medium of claim 22 , the one or more programs further including instructions for:

while the first camera display mode is active, detecting a swipe gesture on the camera display region; and

24. The non-transitory computer-readable storage medium of claim 17 , wherein activating the first camera display mode further includes:

displaying the selected avatar option with a static appearance in the avatar selection region;

updating the selected avatar option to have a dynamic appearance that changes based on the detected change in pose of the subject; and

25. The non-transitory computer-readable storage medium of claim 17 , the one or more programs further including instructions for:

while the first camera display mode is active:

26. The non-transitory computer-readable storage medium of claim 17 , the one or more programs further including instructions for:

while the first camera display mode is active:

in response to a determination that the subject is no longer positioned in the field of view of the camera, modifying the visual appearance of the background displayed in the camera display region.

27. The non-transitory computer-readable storage medium of claim 17 , wherein the camera user interface further includes a second affordance associated with a second camera display mode, the one or more programs further including instructions for:

in response to detecting the gesture directed to the second affordance, activating the second camera display mode, wherein activating the second camera display mode includes:

displaying a visual effects selection region including a plurality of graphical objects;

while the second camera display mode is active, detecting a selection of one of the plurality of graphical objects in the visual effects selection region; and

in response to detecting the selection, displaying a representation of the selected graphical object in the camera display region.

28. The non-transitory computer-readable storage medium of claim 17 , wherein the representation of image data captured via the camera is a media item, and wherein the camera user interface further includes a third affordance associated with a third camera display mode, the one or more programs further including instructions for:

detecting a gesture directed to the third affordance;

in response to detecting the gesture directed to the third affordance, activating the third camera display mode, wherein activating the third camera display mode includes:

displaying a visual effects selection region including a plurality of graphical objects;

while the third camera display mode is active, detecting a selection of one of the plurality of graphical objects in the visual effects selection region; and

in response to detecting the selection, displaying a representation of the selected graphical object on the media item in the camera display region.

29. The non-transitory computer-readable storage medium of claim 17 , wherein the camera user interface further includes a sticker affordance associated with a sticker display mode, the one or more programs further including instructions for:

while displaying the image data in the camera display region, detecting a gesture directed to the sticker affordance;

in response to detecting the gesture directed to the sticker affordance, activating the sticker display mode, wherein activating the sticker display mode includes:

displaying a sticker selection region including a plurality of sticker options;

detecting a selection of one of the plurality of sticker options in the sticker selection region; and

in response to detecting the selection, displaying a representation of the selected sticker option on the image data in the camera display region.

30. The non-transitory computer-readable storage medium of claim 29 , the one or more programs further including instructions for:

while displaying the representation of the selected sticker option on the image data in the camera display region:

detecting lateral movement of the subject in the field of view of the camera; and

31. The non-transitory computer-readable storage medium of claim 29 , the one or more programs further including instructions for:

while displaying the representation of the selected sticker option on the image data in the camera display region:

detecting rotation of the subject in the field of view of the camera; and

in response to detecting the rotation of the subject in the field of view of the camera:

32. The non-transitory computer-readable storage medium of claim 29 , the one or more programs further including instructions for:

while displaying the representation of the selected sticker option on the image data in the camera display region:

detecting movement of the subject toward the camera; and

in response to detecting the movement of the subject toward the camera:

33. A method comprising:

at an electronic device having a camera and a display apparatus:

displaying, via the display apparatus, a camera user interface, the camera user interface including:

a first affordance associated with a first camera display mode;

identifying the subject based on the depth data;

in response to detecting the gesture directed to the first affordance, activating the first camera display mode, wherein activating the first camera display mode includes:

displaying an avatar selection region including a plurality of avatar options; and

while the first camera display mode is active, detecting a change in pose of the subject;

in response to detecting the gesture directed to the filter affordance, activating a filter mode, wherein activating the filter mode includes:

changing an appearance of the displayed representation of the selected avatar option based on the selected filter; and

changing an appearance of at least a portion of the representation of the background based on the selected filter.

34. The method of claim 33 , wherein activating the first camera display mode further includes:

prior to displaying the representation of the selected avatar option over the first portion of the representation of the subject in the camera display region:

35. The method of claim 33 , wherein the avatar selection region further includes an option for ceasing to display the representation of the selected avatar option over the first portion of the representation of the subject in the camera display, the method further comprising:

36. The method of claim 33 , the method further comprising:

detecting a horizontal swipe gesture on the avatar selection region; and

in response to detecting the horizontal swipe gesture, displaying an avatar creation affordance associated with a function for adding a new avatar option to the plurality of avatar options.

37. The method of claim 33 , the method further comprising:

while the first camera display mode is active, detecting a swipe gesture on the avatar selection region; and

38. The method of claim 33 , wherein the representation of image data captured via the camera is a live camera preview.

39. The method of claim 38 , the method further comprising:

while the first camera display mode is active, detecting a swipe gesture on the camera display region; and

40. The method of claim 33 , wherein activating the first camera display mode further includes:

displaying the selected avatar option with a static appearance in the avatar selection region;

updating the selected avatar option to have a dynamic appearance that changes based on the detected change in pose of the subject; and

41. The method of claim 33 , the method further comprising:

while the first camera display mode is active:

42. The method of claim 33 , the method further comprising:

while the first camera display mode is active:

in response to a determination that the subject is no longer positioned in the field of view of the camera, modifying the visual appearance of the background displayed in the camera display region.

43. The method of claim 33 , wherein the camera user interface further includes a second affordance associated with a second camera display mode, the method further comprising:

in response to detecting the gesture directed to the second affordance, activating the second camera display mode, wherein activating the second camera display mode includes:

displaying a visual effects selection region including a plurality of graphical objects;

while the second camera display mode is active, detecting a selection of one of the plurality of graphical objects in the visual effects selection region; and

in response to detecting the selection, displaying a representation of the selected graphical object in the camera display region.

44. The method of claim 33 , wherein the representation of image data captured via the camera is a media item, and wherein the camera user interface further includes a third affordance associated with a third camera display mode, the method further comprising:

detecting a gesture directed to the third affordance;

in response to detecting the gesture directed to the third affordance, activating the third camera display mode, wherein activating the third camera display mode includes:

displaying a visual effects selection region including a plurality of graphical objects;

while the third camera display mode is active, detecting a selection of one of the plurality of graphical objects in the visual effects selection region; and

in response to detecting the selection, displaying a representation of the selected graphical object on the media item in the camera display region.

45. The method of claim 33 , wherein the camera user interface further includes a sticker affordance associated with a sticker display mode, the method further comprising:

while displaying the image data in the camera display region, detecting a gesture directed to the sticker affordance;

in response to detecting the gesture directed to the sticker affordance, activating the sticker display mode, wherein activating the sticker display mode includes:

displaying a sticker selection region including a plurality of sticker options;

detecting a selection of one of the plurality of sticker options in the sticker selection region; and

in response to detecting the selection, displaying a representation of the selected sticker option on the image data in the camera display region.

46. The method of claim 45 , the method further comprising:

while displaying the representation of the selected sticker option on the image data in the camera display region:

detecting lateral movement of the subject in the field of view of the camera; and

47. The method of claim 45 , the method further comprising:

while displaying the representation of the selected sticker option on the image data in the camera display region:

detecting rotation of the subject in the field of view of the camera; and

in response to detecting the rotation of the subject in the field of view of the camera:

48. The method of claim 45 , the method further comprising:

while displaying the representation of the selected sticker option on the image data in the camera display region:

detecting movement of the subject toward the camera; and

in response to detecting the movement of the subject toward the camera:

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 4, 2018

From: VAN OS, MARCEL; ABOUKASM, JESSICA; BLACK, DAVID R.; CHINN, ROBERT; DUDEY, GREGORY L.; ERNST, KATHERINE K.; PAUL, GRANT; SORRENTINO, WILLIAM A., III; WALSH, BRIAN E.; ALBOUZE, JEAN-FRANCOIS; CHANG, JAE WOO; GUZMAN, AURELIO; MOULIOS, CHRISTOPHER J.; NEWMAN, JOANNA M.; SCAPEL, NICOLAS; WEIL, JOSEPH-ALEXANDER P.; WILSON, CHRISTOPHER

To: APPLE INC.

Reel/Frame 047672/0518 →

Continuity (2)

Provisional Application 62679934 · Jun 3, 2018

Provisional Application 62668227 · May 7, 2018

Cited By (19)

US 1,119,941 US 12,190,407 US 12,192,617 US 12,223,612 US 12,287,913 US 12,314,553 US 12,340,481 US 12,379,834 US 12,394,077 US 12,417,596 US 12,422,977 US 12,482,161 US 12,602,142 US 12,619,345 US 12,638,913 US 12,647,673 US 12,681,625 US 12,701,334 US 12,713,125