IP Library › Granted Patent US 12,449,946
Granted Patent B2
US 12,449,946 · App. 18/152,656 · Granted Oct 21, 2025

Methods for displaying user interface elements relative to media content

Inventors: Wesley M. Holder (Union City, CA); Earl M. Olson (Santa Clara, CA); Matan Stauber (San Francisco, CA); Stephen O. Lemay (Palo Alto, CA); Maneli Noorkami (Menlo Park, CA); Negar Nejati (Palo Alto, CA); Afshin Taghavi Nasrabadi (Los Gatos, CA)
Assignee: Apple Inc.
G06F3/04815G06F3/04847
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,449,946
App. No.
18/152,656
Filed
Jan 10, 2023
Granted
Oct 21, 2025
Kind
B2
Art Unit
2171
USPC
715/716
Abstract

In some embodiments, a computer system displays a caption for a media item at different depths depending on the depth of the portion of the media item over which the caption is displayed. In some embodiments, a computer system displays a user interface element that includes information associated with the media item at different locations relative to the media item depending on attention of the user. In some embodiments, a computer system displays a user interface element that includes information associated with the media item with different visual appearances depending on visual characteristics of the portion of the media item over which the user interface element is displayed.

Claims (108)

1. A method comprising:

at a computer system that is in communication with one or more display generation components and one or more input devices:

while displaying, via the one or more display generation components, a representation of a media item that has content at different depths, displaying, overlaid on a first portion or a second portion, different from the first portion, of the representation of the media item, a respective user interface element that includes a caption for the media item that is being played by the computer system, wherein displaying the respective user interface element overlaid on the first portion or the second portion of the representation of the media item includes:

in accordance with a determination that the first portion of the representation of the media item is being presented at a first depth in the representation of the media item, and that the first portion of the representation of the media item is closer to a viewpoint of a user than the second portion of the representation of the media item, the respective user interface element is displayed at a first distance from the viewpoint of the user of the computer system, wherein the respective user interface element is overlaid on the first portion of the representation of the media item; and

in accordance with a determination that the second portion of the representation of the media item is being presented at a second depth in the representation of the media item, and that the second portion of the representation of the media item is closer to the viewpoint of the user than the first portion of the representation of the media item, the respective user interface element is displayed at a second distance, different from the first distance, from the viewpoint of the user, wherein the respective user interface element is overlaid on the second portion of the representation of the media item.

2. The method of claim 1 , further comprising:

after presenting the first portion of the representation of the media item at the first depth in the representation of the media item, wherein the respective user interface element was overlaid on the first portion of the representation of the media item, displaying the first portion of the representation of the media item at a third depth, different from the first depth, in the representation of the media item, and

while displaying the first portion of the representation of the media item at the third depth in the representation of the media item, displaying the respective user interface element at a third distance, different from the first distance, from the viewpoint of the user.

3. The method of claim 1 , wherein the respective user interface element is displayed at the first distance from the viewpoint of the user when the respective user interface element is displayed overlaid on the first portion of the representation of the media item, the method further comprising:

after displaying, overlaid on the first portion of the representation of the media item, the respective user interface element at the first distance from the viewpoint of the user, displaying, via the one or more display generation components, the respective user interface element overlaid on the second portion, different from the first portion, of the representation of the media item, wherein displaying the respective user interface element overlaid on the second portion of the representation of the media item includes:

in accordance with a determination that the second portion of the representation of the media item is being presented at a third depth, different from the first depth, in the representation of the media item, displaying the respective user interface element at a third distance, different from the first distance, from the viewpoint of the user, wherein the respective user interface element is overlaid on the second portion of the representation of the media item.

4. The method of claim 1 , wherein the respective user interface element is overlaid on the first portion of the representation of the media item when one or more criteria are satisfied, including a criterion that is satisfied when a target focal plane corresponds to the first portion of the representation of the media item, and is not overlaid on the first portion of the representation of the media item when the one or more criteria are not satisfied.

5. The method of claim 1 , further comprising:

while displaying the representation of the media item with the respective user interface element overlaid on the first or second portion of the representation of the media item, receiving, via the one or more input devices, a first input directed to the representation of the media item; and

in response to receiving the first input:

displaying, via the one or more display generation components, a playback controls user interface that includes one or more selectable user interface elements for modifying playback of the representation of the media item; and

ceasing display of the respective user interface element overlaid on the first portion of the representation of the media item.

6. The method of claim 5 , wherein the playback controls user interface is displayed when one or more criteria are satisfied and is not displayed when the one or more criteria are not satisfied, the method further comprising:

after ceasing display of the respective user interface element and while displaying the playback controls user interface, determining that the one or more criteria are not satisfied; and

in response to determining that the one or more criteria are not satisfied:

ceasing display, via the one or more display generation components, of the playback controls user interface; and

displaying, via the one or more display generation components, the respective user interface element.

7. The method of claim 5 , further comprising:

in response to receiving the first input and while displaying the playback controls user interface:

displaying, via the one or more display generation components, the respective user interface element overlaid on a third portion, different from the first portion, of the representation of the media item.

8. The method of claim 1 , further comprising:

while displaying the respective user interface element overlaid on the first portion of the representation of the media item, detecting a change in a relative location of the representation of the media item relative to the viewpoint of the user; and

while detecting the change in the relative location of the representation of the media item relative to the viewpoint of the user:

moving the respective user interface element from overlaying the first portion of the representation of the media item to overlaying a third portion, different from the first portion, of the representation of the media item in accordance with the change of the relative location of the representation of the media item relative to the viewpoint of the user.

9. The method of claim 1 , wherein the respective user interface element is overlaid on the first portion of the representation of the media item when one or more criteria are satisfied, including a criterion that is satisfied when audio of the media item corresponds to the first portion of the representation of the media item, and is not overlaid on the first portion of the representation of the media item when the one or more criteria are not satisfied.

10. The method of claim 1 , wherein:

in accordance with a determination that the representation of the media item is at a first playback position, the caption for the media item includes first content that corresponds to audio present at the first playback position; and

in accordance with a determination that representation of the media item is at a second playback position, different from the first playback position, the caption for the media item includes second content, different from the first content, that corresponds to audio present at the second playback position.

11. The method of claim 1 , further comprising:

while displaying the respective user interface element and while the computer system is playing the representation of the media item, receiving, via the one or more input devices, an input corresponding to a request to cease displaying the respective user interface element; and

in response to receiving the input, continuing playback of the representation of the media item without displaying the respective user interface element.

12. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with one or more display generation components and one or more input devices, the one or more programs including instructions for:

while displaying, via the one or more display generation components, a representation of a media item that has content at different depths, displaying, overlaid on a first portion or a second portion, different from the first portion, of the representation of the media item, a respective user interface element that includes a caption for the media item that is being played by the computer system, wherein displaying the respective user interface element overlaid on the first portion or the second portion of the representation of the media item includes:

in accordance with a determination that the first portion of the representation of the media item is being presented at a first depth in the representation of the media item, and that the first portion of the representation of the media item is closer to a viewpoint of a user than the second portion of the representation of the media item, the respective user interface element is displayed at a first distance from the viewpoint of the user of the computer system, wherein the respective user interface element is overlaid on the first portion of the representation of the media item; and

in accordance with a determination that the second portion of the representation of the media item is being presented at a second depth in the representation of the media item, and that the second portion of the representation of the media item is closer to the viewpoint of the user than the first portion of the representation of the media item, the respective user interface element is displayed at a second distance, different from the first distance, from the viewpoint of the user, wherein the respective user interface element is overlaid on the second portion of the representation of the media item.

13. The non-transitory computer readable storage medium of claim 12 , wherein the one or more programs further including instructions for:

after presenting the first portion of the representation of the media item at the first depth in the representation of the media item, wherein the respective user interface element was overlaid on the first portion of the representation of the media item, displaying the first portion of the representation of the media item at a third depth, different from the first depth, in the representation of the media item, and

while displaying the first portion of the representation of the media item at the third depth in the representation of the media item, displaying the respective user interface element at a third distance, different from the first distance, from the viewpoint of the user.

14. The non-transitory computer readable storage medium of claim 12 , wherein the respective user interface element is displayed at the first distance from the viewpoint of the user when the respective user interface element is displayed overlaid on the first portion of the representation of the media item, the one or more programs further including instructions for:

after displaying, overlaid on the first portion of the representation of the media item, the respective user interface element at the first distance from the viewpoint of the user, displaying, via the one or more display generation components, the respective user interface element overlaid on the second portion, different from the first portion, of the representation of the media item, wherein displaying the respective user interface element overlaid on the second portion of the representation of the media item includes:

in accordance with a determination that the second portion of the representation of the media item is being presented at a third depth, different from the first depth, in the representation of the media item, displaying the respective user interface element at a third distance, different from the first distance, from the viewpoint of the user, wherein the respective user interface element is overlaid on the second portion of the representation of the media item.

15. The non-transitory computer readable storage medium of claim 12 , wherein the respective user interface element is overlaid on the first portion of the representation of the media item when one or more criteria are satisfied, including a criterion that is satisfied when a target focal plane corresponds to the first portion of the representation of the media item, and is not overlaid on the first portion of the representation of the media item when the one or more criteria are not satisfied.

16. The non-transitory computer readable storage medium of claim 12 , the one or more programs further including instructions for:

while displaying the representation of the media item with the respective user interface element overlaid on the first or second portion of the representation of the media item, receiving, via the one or more input devices, a first input directed to the representation of the media item; and

in response to receiving the first input:

displaying, via the one or more display generation components, a playback controls user interface that includes one or more selectable user interface elements for modifying playback of the representation of the media item; and

ceasing display of the respective user interface element overlaid on the first portion of the representation of the media item.

17. The non-transitory computer readable storage medium of claim 16 , wherein the playback controls user interface is displayed when one or more criteria are satisfied and is not displayed when the one or more criteria are not satisfied, the one or more programs further including instructions for:

after ceasing display of the respective user interface element and while displaying the playback controls user interface, determining that the one or more criteria are not satisfied; and

in response to determining that the one or more criteria are not satisfied:

ceasing display, via the one or more display generation components, of the playback controls user interface; and

displaying, via the one or more display generation components, the respective user interface element.

18. The non-transitory computer readable storage medium of claim 16 , the one or more programs further including instructions for:

in response to receiving the first input and while displaying the playback controls user interface:

displaying, via the one or more display generation components, the respective user interface element overlaid on a third portion, different from the first portion, of the representation of the media item.

19. The non-transitory computer readable storage medium of claim 12 , the one or more programs further including instructions for:

while displaying the respective user interface element overlaid on the first portion of the representation of the media item, detecting a change in a relative location of the representation of the media item relative to the viewpoint of the user; and

while detecting the change in the relative location of the representation of the media item relative to the viewpoint of the user:

moving the respective user interface element from overlaying the first portion of the representation of the media item to overlaying a third portion, different from the first portion, of the representation of the media item in accordance with the change of the relative location of the representation of the media item relative to the viewpoint of the user.

20. The non-transitory computer readable storage medium of claim 12 , wherein the respective user interface element is overlaid on the first portion of the representation of the media item when one or more criteria are satisfied, including a criterion that is satisfied when audio of the media item corresponds to the first portion of the representation of the media item, and is not overlaid on the first portion of the representation of the media item when the one or more criteria are not satisfied.

21. The non-transitory computer readable storage medium of claim 12 , wherein:

in accordance with a determination that the representation of the media item is at a first playback position, the caption for the media item includes first content that corresponds to audio present at the first playback position; and

in accordance with a determination that representation of the media item is at a second playback position, different from the first playback position, the caption for the media item includes second content, different from the first content, that corresponds to audio present at the second playback position.

22. The non-transitory computer readable storage medium of claim 12 , the one or more programs further including instructions for:

while displaying the respective user interface element and while the computer system is playing the representation of the media item, receiving, via the one or more input devices, an input corresponding to a request to cease displaying the user interface element; and

in response to receiving the input, continuing playback of the representation of the media item without displaying the respective user interface element.

23. A computer system that is in communication with one or more display generation components and one or more input devices, the computer system comprising:

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:

while displaying, via the one or more display generation components, a representation of a media item that has content at different depths, displaying, overlaid on a first portion or a second portion, different from the first portion, of the representation of the media item, a respective user interface element that includes a caption for the media item that is being played by the computer system, wherein displaying the respective user interface element overlaid on the first portion or the second portion of the representation of the media item includes:

in accordance with a determination that the first portion of the representation of the media item is being presented at a first depth in the representation of the media item, and that the first portion of the representation of the media item is closer to a viewpoint of a user than the second portion of the representation of the media item, the respective user interface element is displayed at a first distance from the viewpoint of the user of the computer system, wherein the respective user interface element is overlaid on the first portion of the representation of the media item; and

in accordance with a determination that the second portion of the representation of the media item is being presented at a second depth in the representation of the media item, and that the second portion of the representation of the media item is closer to the viewpoint of the user than the first portion of the representation of the media item, the respective user interface element is displayed at a second distance, different from the first distance, from the viewpoint of the user, wherein the respective user interface element is overlaid on the second portion of the representation of the media item.

24. The computer system of claim 23 , the one or more programs further including instructions for:

after presenting the first portion of the representation of the media item at the first depth in the representation of the media item, wherein the respective user interface element was overlaid on the first portion of the representation of the media item, displaying the first portion of the representation of the media item at a third depth, different from the first depth, in the representation of the media item, and

while displaying the first portion of the representation of the media item at the third depth in the representation of the media item, displaying the respective user interface element at a third distance, different from the first distance, from the viewpoint of the user.

25. The computer system of claim 23 , wherein the respective user interface element is displayed at the first distance from the viewpoint of the user when the respective user interface element is displayed overlaid on the first portion of the representation of the media item, the one or more programs further including instructions for:

after displaying, overlaid on the first portion of the representation of the media item, the respective user interface element at the first distance from the viewpoint of the user, displaying, via the one or more display generation components, the respective user interface element overlaid on the second portion, different from the first portion, of the representation of the media item, wherein displaying the respective user interface element overlaid on the second portion of the representation of the media item includes:

in accordance with a determination that the second portion of the representation of the media item is being presented at a third depth, different from the first depth, in the representation of the media item, displaying the respective user interface element at a third distance, different from the first distance, from the viewpoint of the user, wherein the respective user interface element is overlaid on the second portion of the representation of the media item.

26. The computer system of claim 23 , wherein the one or more programs further including instructions for overlaying the respective user interface element on the first portion of the representation of the media item when one or more criteria are satisfied, including a criterion that is satisfied when a target focal plane corresponds to the first portion of the representation of the media item, and to not overlay on the first portion of the representation of the media item when the one or more criteria are not satisfied.

27. The computer system of claim 23 , the one or more programs further including instructions for:

while displaying the representation of the media item with the respective user interface element overlaid on the first or second portion of the representation of the media item, receiving, via the one or more input devices, a first input directed to the representation of the media item; and

in response to receiving the first input:

displaying, via the one or more display generation components, a playback controls user interface that includes one or more selectable user interface elements for modifying playback of the representation of the media item; and

ceasing display of the respective user interface element overlaid on the first portion of the representation of the media item.

28. The computer system of claim 27 , wherein the playback controls user interface is displayed when one or more criteria are satisfied and is not displayed when the one or more criteria are not satisfied, the one or more programs further including instructions for:

after ceasing display of the respective user interface element and while displaying the playback controls user interface, determining that the one or more criteria are not satisfied; and

in response to determining that the one or more criteria are not satisfied:

ceasing display, via the one or more display generation components, of the playback controls user interface; and

displaying, via the one or more display generation components, the respective user interface element.

29. The computer system of claim 27 , the one or more programs further including instructions for:

in response to receiving the first input and while displaying the playback controls user interface:

displaying, via the one or more display generation components, the respective user interface element overlaid on a third portion, different from the first portion, of the representation of the media item.

30. The computer system of claim 23 , the one or more programs further including instructions for:

while displaying the respective user interface element overlaid on the first portion of the representation of the media item, detecting a change in a relative location of the representation of the media item relative to the viewpoint of the user; and

while detecting the change in the relative location of the representation of the media item relative to the viewpoint of the user:

moving the respective user interface element from overlaying the first portion of the representation of the media item to overlaying a third portion, different from the first portion, of the representation of the media item in accordance with the change of the relative location of the representation of the media item relative to the viewpoint of the user.

31. The computer system of claim 23 , wherein the respective user interface element is overlaid on the first portion of the representation of the media item when one or more criteria are satisfied, including a criterion that is satisfied when audio of the media item corresponds to the first portion of the representation of the media item, and is not overlaid on the first portion of the representation of the media item when the one or more criteria are not satisfied.

32. The computer system of claim 23 , wherein:

in accordance with a determination that the representation of the media item is at a first playback position, the caption for the media item includes first content that corresponds to audio present at the first playback position; and

in accordance with a determination that representation of the media item is at a second playback position, different from the first playback position, the caption for the media item includes second content, different from the first content, that corresponds to audio present at the second playback position.

33. The computer system of claim 23 , the one or more programs further including instructions for:

while displaying the respective user interface element and while the computer system is playing the representation of the media item, receiving, via the one or more input devices, an input corresponding to a request to cease displaying the respective user interface element; and

in response to receiving the input, continuing playback of the representation of the media item without displaying the respective user interface element.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE CONVEYING PARTY DATA PREVIOUSLY RECORDED AT REEL: 62497 FRAME: 220. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Mar 11, 2025
From: HOLDER, WESLEY M.; OLSON, EARL M.; STAUBER, MATAN; LEMAY, STEPHEN O.; NOORKAMI, MANELI; NEJATI, NEGAR; TAGHAVI NASRABADI, AFSHIN
To: APPLE INC.
Reel/Frame 070481/0259 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 26, 2023
From: HOLDER, WESLEY M.; OLSON, EARL M.; STAUBER, MATAN; LEMAY, STEPHEN O.
To: APPLE INC.
Reel/Frame 062497/0220 →
Continuity (2)
Provisional Application 63266625 · Jan 10, 2022
Related Publication 20230221833A1 · Jul 13, 2023
References Cited (61)
US 6323846B1 · Westerman et al. · 2001 [cited by applicant]
US 6570557B1 · Westerman et al. · 2003 [cited by applicant]
US 6677932B1 · Westerman · 2004 [cited by applicant]
US 7614008B2 · Ording · 2009 [cited by applicant]
US 7633076B2 · Huppi et al. · 2009 [cited by applicant]
US 7653883B2 · Hotelling et al. · 2010 [cited by applicant]
US 7657849B2 · Chaudhri et al. · 2010 [cited by applicant]
US 7663607B2 · Hotelling et al. · 2010 [cited by applicant]
US 7844914B2 · Andre et al. · 2010 [cited by applicant]
US 7957762B2 · Herz et al. · 2011 [cited by applicant]
US 8006002B2 · Kalayjian et al. · 2011 [cited by applicant]
US 8239784B2 · Hotelling et al. · 2012 [cited by applicant]
US 8279180B2 · Hotelling et al. · 2012 [cited by applicant]
US 8381135B2 · Hotelling et al. · 2013 [cited by applicant]
US 8479122B2 · Hotelling et al. · 2013 [cited by applicant]
US 8767045B2 · Kitazato · 2014 [cited by examiner]
US 9237334B2 · Cheng · 2016 [cited by examiner]
US 9241149B2 · Redmann · 2016 [cited by examiner]
US 9348458B2 · Hotelling et al. · 2016 [cited by applicant]
US 9436357B2 · Pallakoff · 2016 [cited by examiner]
US 9588651B1 · Buchanan · 2017 [cited by examiner]
US 9933937B2 · Lemay et al. · 2018 [cited by applicant]
US 10484641B2 · Zhou · 2019 [cited by examiner]
US 10645332B2 · Zhang · 2020 [cited by examiner]
US 20020015024A1 · Westerman et al. · 2002 [cited by applicant]
US 20040230912A1 · Clow et al. · 2004 [cited by applicant]
US 20050190059A1 · Wehrenberg · 2005 [cited by applicant]
US 20060017692A1 · Wehrenberg et al. · 2006 [cited by applicant]
US 20060033724A1 · Chaudhri et al. · 2006 [cited by applicant]
US 20060034590A1 · Teramoto · 2006 [cited by applicant]
US 20060197753A1 · Hotelling · 2006 [cited by applicant]
US 20100188572A1 · Card, II · 2010 [cited by examiner]
US 20100328432A1 · Tanaka · 2010 [cited by applicant]
US 20130010062A1 · Redmann · 2013 [cited by examiner]
US 20130307945A1 · Cheng · 2013 [cited by examiner]
US 20130326341A1 · Nonaka · 2013 [cited by examiner]
US 20140362111A1 · Kim · 2014 [cited by examiner]
US 20160224122A1 · Dietz et al. · 2016 [cited by applicant]
US 20160373714A1 · Lee · 2016 [cited by examiner]
US 20170302903A1 · Ng · 2017 [cited by examiner]
US 20180220195A1 · Panchaksharaiah et al. · 2018 [cited by applicant]
US 20190279407A1 · McHugh · 2019 [cited by examiner]
US 20200099989A1 · Niemirska et al. · 2020 [cited by applicant]
US 20220253136A1 · Holder et al. · 2022 [cited by applicant]
US 20230007335A1 · Gupta · 2023 [cited by examiner]
US 20230092874A1 · Krivoruchko et al. · 2023 [cited by applicant]
US 20230093979A1 · Stauber et al. · 2023 [cited by applicant]
US 20230394755A1 · Negoita · 2023 [cited by examiner]
US 20230396854A1 · Sanders et al. · 2023 [cited by applicant]
US 20240070948A1 · Bradley · 2024 [cited by examiner]
US 20240152256A1 · Dascola · 2024 [cited by examiner]
US 20240205509A1 · Kim · 2024 [cited by examiner]
US 20250005855A1 · Holder et al. · 2025 [cited by applicant]
JP H11289555A · 1999 [cited by applicant]
WO 2013169849A2 · 2013 [cited by applicant]
WO 2014105276A1 · 2014 [cited by applicant]
Final Office Action received for U.S. Appl. No. 18/327,844, mailed on Nov. 29, 2024, 25 pages. [cited by applicant]
International Search Report received for PCT Patent Application No. PCT/US2023/060429, mailed on Jun. 28, 2023, 7 pages. [cited by applicant]
Non-Final Office Action received for U.S. Appl. No. 18/327,844, mailed on May 17, 2024, 20 pages. [cited by applicant]
Yue , et al., “Optimal Region Selection for Stereoscopic Video Subtitle Insertion”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, No. 11, Nov. 2018, 13 pages. [cited by applicant]
International Search Report received for PCT Patent Application No. PCT/US2024/032455, mailed on Sep. 25, 2024, 3 pages. [cited by applicant]