IP Library › Granted Patent US 12,555,183
Granted Patent B2
US 12,555,183 · App. 18/162,171 · Granted Feb 17, 2026

System and method for providing spatiotemporal visual guidance within 360-degree video

Inventors: Sean Liu (Toronto, CA); Rorik Henrikson (Toronto, CA); Mark Parent (Toronto, CA); Michael Glueck (Toronto, CA); Tovi Grossman (Toronto, CA)
Assignee: Meta Platforms Technologies, LLC
G06T3/16G06T19/006G06V10/25G06V20/20G06V20/44
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,555,183
App. No.
18/162,171
Granted
Feb 17, 2026
Kind
B2
Abstract

Features described herein generally relate to providing spatiotemporal guidance within a 360-degree video. Particularly, while a 360-degree video is being displayed to a user, a current location of the view of the user within the 360-degree video is determined. Additionally, regions of interest may be identified within the 360-degree video, along with their location and a time in which the regions of interest are active within the 360-degree video. A visual guide is then overlaid onto the 360-degree video that indicates a current location of the view of the user, as well as locations of one or more regions of interest, as well as the time during which each region of interest is active (able to be viewed) within the 360-degree video. By viewing the visual guide, the user may be able to anticipate a location and timing of regions of interest within the 360-degree video and adjust their gaze to the correct location at the correct time to view such regions of interest.

Claims (99)

1 . An extended reality headset comprising:

a display configured to display content to a user;

one or more sensors configured to capture input data; and

one or more processors configured to:

cause identification of a location of a region of interest within a 360-degree video;

cause identification of a time during which the region of interest is active within the 360-degree video;

receive information used to present a visual guide, the visual guide including one or more visual indicators indicating:

the location of the region of interest within the 360-degree video with respect to a user's current view;

the time during which the region of interest is active within the 360-degree video; and

wherein the visual guide includes one or more additional visual indicators in accordance with a determination that the region of interest is active and the user's current view does not enable the region of interest to be viewed; and

cause output of the visual guide at the display.

2 . The extended reality headset of claim 1 , wherein the one or more processors are configured to determine the user's current view within a 360-degree video based on the input data, utilizing the one or more sensors within the extended reality headset.

3 . The extended reality headset of claim 1 , wherein the one or more processors are configured to receive a label for the region of interest, the label including:

a bounding box indicating the location of the region of interest within the 360-degree video, and

a start time and an end time for the region of interest within the 360-degree video.

4 . The extended reality headset of claim 1 , wherein the visual guide includes a visual widget that represents a top-down overview of the 360-degree video.

5 . The extended reality headset of claim 1 , wherein;

the visual guide includes a first shape representing the user; and

the first shape aligns with the user's current view.

6 . The extended reality headset of claim 5 , wherein:

the visual guide includes a second shape representing the region of interest; and

a distance of the second shape from the first shape corresponds to a difference between a current time in the 360-degree video and a time in which the region of interest is active.

7 . The extended reality headset of claim 1 , wherein:

the visual guide includes a central object that aligns with the user's current view;

the region of interest is displayed as a geometric shape positioned around the central object

the location of the region of interest with respect to the user's current view corresponds to an angular location of the geometric shape with respect to the central object;

the time during which the region of interest is active within the 360-degree video corresponds to a distance of the geometric shape from a center of the central object;

a decrease in the distance between the geometric shape and the center of the central object signifies a decrease in a time until the region of interest is active within the 360-degree video;

a duration of the region of interest within the 360-degree video is represented by a radial thickness of the geometric shape; or

the geometric shape intersects with the central object when the region of interest is active within the 360-degree video.

8 . The extended reality headset of claim 7 , wherein the one or more processors are configured to:

alter one or more characteristics of the central object in response to determining that:

the region of interest is active, and

the user's current view does not enable the region of interest to be viewed;

alter a shape of the geometric shape in response to determining that the region of interest is currently moving horizontally within the 360-degree video; or

alter one or more characteristics of the geometric shape in response to determining that the region of interest is active and the user's current view enables the region of interest to be viewed.

9 . A method comprising:

identifying a location of a region of interest within a 360-degree video;

identifying a time during which the region of interest is active within the 360-degree video;

generating a visual guide, the visual guide including one or more visual indicators indicating:

the location of the region of interest within the 360-degree video with respect to a user's current view;

the time during which the region of interest is active within the 360-degree video; and

wherein the visual guide includes one or more additional visual indicators in accordance with a determination that the region of interest is active and the user's current view does not enable the region of interest to be viewed; and

outputting the visual guide at a display.

10 . The method of claim 9 , comprising determining the user's current view within a 360-degree video based on input data, utilizing one or more sensors within a head-mounted device.

11 . The method of claim 9 , comprising receiving a label for the region of interest, the label including:

a bounding box indicating the location of the region of interest within the 360-degree video, and

a start time and an end time for the region of interest within the 360-degree video.

12 . The method of claim 9 , wherein:

the visual guide includes:

a first shape representing a user; and

a second shape representing the region of interest, wherein a distance of the second shape from the first shape corresponds to a difference between a current time in the 360-degree video and a time in which the region of interest is active; and

the first shape aligns with the user's current view.

13 . The method of claim 9 , wherein:

the visual guide includes a central object that aligns with the user's current view;

the region of interest is displayed as a geometric shape positioned around the central object

the location of the region of interest with respect to the user's current view corresponds to an angular location of the geometric shape with respect to the central object;

the time during which the region of interest is active within the 360-degree video corresponds to a distance of the geometric shape from a center of the central object;

a decrease in the distance between the geometric shape and the center of the central object signifies a decrease in a time until the region of interest is active within the 360-degree video;

a duration of the region of interest within the 360-degree video is represented by a radial thickness of the geometric shape; or

the geometric shape intersects with the central object when the region of interest is active within the 360-degree video.

14 . The method of claim 13 , comprising:

altering one or more characteristics of the central object in response to determining that:

the region of interest is active, and

the user's current view does not enable the region of interest to be viewed;

altering a shape of the geometric shape in response to determining that the region of interest is currently moving horizontally within the 360-degree video; or

altering one or more characteristics of the geometric shape in response to determining that the region of interest is active and the user's current view enables the region of interest to be viewed.

15 . One or more non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processing systems, cause the one or more processing systems to perform operations including:

identifying a location of a region of interest (ROI) within a 360-degree video;

identifying a time during which the region of interest is active within the 360-degree video;

generating a visual guide, the visual guide including one or more visual indicators indicating:

the location of the region of interest within the 360-degree video with respect to a user's current view;

the time during which the region of interest is active within the 360-degree video; and

wherein the visual guide includes one or more additional visual indicators in accordance with a determination that the region of interest is active and the user's current view does not enable the region of interest to be viewed; and

outputting the visual guide at a display.

16 . The one or more non-transitory computer-readable media of claim 15 , wherein the operations include:

determining the user's current view within a 360-degree video based on input data, utilizing one or more sensors within a head-mounted device.

17 . The one or more non-transitory computer-readable media of claim 15 , wherein the operations include receiving a label for the region of interest, the label including:

a bounding box indicating the location of the region of interest within the 360-degree video, and

a start time and an end time for the region of interest within the 360-degree video.

18 . The one or more non-transitory computer-readable media of claim 15 , wherein:

the visual guide includes:

a first shape representing a user; and

a second shape representing the region of interest, wherein a distance of the second shape from the first shape corresponds to a difference between a current time in the 360-degree video and a time in which the region of interest is active; and

the first shape aligns with the user's current view.

19 . The one or more non-transitory computer-readable media of claim 15 , wherein:

the visual guide includes a central object that aligns with the user's current view;

the region of interest is displayed as a geometric shape positioned around the central object

the location of the region of interest with respect to the user's current view corresponds to an angular location of the geometric shape with respect to the central object;

the time during which the region of interest is active within the 360-degree video corresponds to a distance of the geometric shape from a center of the central object;

a decrease in the distance between the geometric shape and the center of the central object signifies a decrease in a time until the region of interest is active within the 360-degree video;

a duration of the region of interest within the 360-degree video is represented by a radial thickness of the geometric shape; or

the geometric shape intersects with the central object when the region of interest is active within the 360-degree video.

20 . The one or more non-transitory computer-readable media of claim 19 , wherein the operations include:

altering one or more characteristics of the central object in response to determining that:

the region of interest is active, and

the user's current view does not enable the region of interest to be viewed;

altering a shape of the geometric shape in response to determining that the region of interest is currently moving horizontally within the 360-degree video; or

altering one or more characteristics of the geometric shape in response to determining that the region of interest is active and the user's current view enables the region of interest to be viewed.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 9, 2023
From: LIU, SEAN; HENRIKSON, RORIK; PARENT, MARK; GLUECK, MICHAEL; GROSSMAN, TOVI
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 064537/0574 →
Continuity (2)
Provisional Application 63375600 · Sep 14, 2022
Related Publication 20240095877A1 · Mar 21, 2024
References Cited (46)
US 20200053351A1 · Han · 2020 [cited by examiner]
US 20200382755A1 · DiVerdi · 2020 [cited by examiner]
Alcove VR, “360 3D Marseille, France—Guided Tour—Virtual Travel 4K (Full series on Alcove for Quest),” Youtube, Apr. 19, 2022, 1 page, Retrieved from the Internet: URL: https://www.youtube.com/watch?v=gURIhqukTSQ. [cited by applicant]
Baudisch P., et al., “Halo: A Technique for Visualizing Off-Screen Objects,” CHI 03: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 5, 2003, vol. 05, No. 01, pp. 481-488. [cited by applicant]
Beat Games, “Beat Saber”, 2018, Available online at https://beatsaber.com, 18 pages. [cited by applicant]
Brillhart J., “In the Blink of a Mind—Attention,” The Language of VR, Feb. 5, 2016, 12 pages, Retrieved from the Internet: URL: https://medium.com/the-language-of-vr/in-the-blink-of-a-mind-attention-1fdff60fa045. [cited by applicant]
Bruder G., et al., “Tuning Self-Motion Perception in Virtual Reality with Visual Illusions,” IEEE Transactions on Visualization and Computer Graphics, Jul. 2012, vol. 18, No. 07, pp. 1068-1078. [cited by applicant]
Cao C., et al., “Automatic Generation of Diegetic Guidance in Cinematic Virtual Reality,” 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Nov. 9, 2020, pp. 600-607. [cited by applicant]
Grogorick S., et al., “Comparing Unobtrusive Gaze Guiding Stimuli in Head-Mounted Displays,” 2018 IEEE International Conference on Image Processing (ICIP), Oct. 7, 2018, pp. 2805-2809. [cited by applicant]
Grogorick S., et al., “Stereo Inverse Brightness Modulation for Guidance in Dynamic Panorama Videos in Virtual Reality,” Computer Graphics Forum, Aug. 29, 2020, vol. 39, No. 06, 12 pages. [cited by applicant]
Gustafson S., et al., “Wedge: Clutter-Free Visualization of Off-Screen Locations,” CHI 08: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 5, 2008, pp. 787-796. [cited by applicant]
Harmonix, “Guitar Hero (Video Game),” Wikipedia, Nov. 8, 2005, 10 pages. [cited by applicant]
Hart S.G., et al., “Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research,” Human Mental Workload, 1988, vol. 52, pp. 139-183. [cited by applicant]
Kaul O.B., et al., “HapticHead: A Spherical Vibrotactile Grid around the Head for 3D Guidance in Virtual and Augmented Reality,” CHI 17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, May … [cited by applicant]
Kinnunen J., “Spatial Guiding through Haptic Cues in Omnidirectional Video,” Master's Thesis University of Tampere, Jun. 2018, 56 pages. [cited by applicant]
Konami, “Dance Dance Revolution”, 1998, 1 page, Available online at: https://www.youtube.com/watch?v=K2IammLWeNE, 1 page. [cited by applicant]
“Renderheads,” AVPro Video, 2022, 1 page, Retrieved from the Internet: https://www.renderheads.com/content/docs/AVProVideo/articles/intro.html. [cited by applicant]
Li J., et al., “Route Tapestries: Navigating 360 Virtual Tour Videos Using Slit-Scan Visualizations,” UIST 21: The 34th Annual ACM Symposium on User Interface Software and Technology, Oct. 12, 2021, pp. 223-238. [cited by applicant]
Lilija K., et al., “Who Put That There? Temporal Navigation of Spatial Recordings by Direct Manipulation,” CHI 20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Apr. 25-30, 2020, 11 pages. [cited by applicant]
Lin Y-C., et al., “Tell Me Where to Look: Investigating Ways for Assisting Focus in 360° Video,” CHI 17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, May 6-11, 2017, pp. 2535-2545. [cited by applicant]
Lin Y-T., “Outside-In: Visualizing Out-of-Sight Regions-of-Interest in a 360° Video Using Spatial Picture-in-Picture Previews,” UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Techn… [cited by applicant]
Liu S.J., et al., “View-Dependent Video Textures for 360° Video,” Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, Oct. 20-23, 2019, pp. 249-262. [cited by applicant]
Maranes C., et al., “Exploring The Impact Of 360 Movie Cuts In Users' Attention,” IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Mar. 22-26, 2020, pp. 73-82. [cited by applicant]
Maranes C., et al., “Towards Assisting The Decision-Making Process For Content Creators In Cinematic Virtual Reality Through The Analysis Of Movie Cuts And Their Influence On Viewers' Behavior,” International Transactio… [cited by applicant]
Masia B., et al., “Influence of Directional Sound Cues on Users' Exploration Across 360° Movie Cuts,” IEEE Computer Graphics and Applications, Mar. 11, 2021, vol. 41, No. 4, 14 pages. [cited by applicant]
McCrae J., et al., “Exploring the Design Space of Multiscale 3D Orientation,” In Proceedings of the International Conference on Advanced Visual Interfaces., May 26, 2010, pp. 81-88. [cited by applicant]
McCrae J., et al., “Multiscale 3D Navigation,” Proceedings of the 2009 symposium on Interactive 3D graphics and games, Feb. 27, 2009, pp. 7-14. [cited by applicant]
National Geographic, “Elephant Encounter in 360—Ep. 2 The Okavango Experience,” Youtube, Dec. 18, 2018, 2 pages, Retrieved from the Internet: URL: https://www.youtube.com/watch?v=HI7mTlxNotQ. [cited by applicant]
Nguyen C., et al., “Vremiere: In-Headset Virtual Reality Video Editing,” Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, May 6-11, 2017, pp. 5428-5438. [cited by applicant]
Nielsen L. T., et al., “. Missing the Point: An Exploration of How to Guide Users' Attention during Cinematic Virtual Reality,” In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, Nov. … [cited by applicant]
Pavel A., et al., “Shot Orientation Controls for Interactive Cinematography with 360 Video,” Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, 2017, pp. 289-297. [cited by applicant]
Rockstar, “Grand Theft Auto IV”, PlayStation 3, Apr. 29, 2008, 3 pages. [cited by applicant]
Rothe S., et al., “GazeRecall: Using Gaze Direction to Increase Recall of Details in Cinematic Virtual Reality,” In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia, Nov. 25, 2018, pp… [cited by applicant]
Rothe S., et al., “Guidance in Cinematic Virtual Reality-Taxonomy, Research Status and Challenges,” Multimodal Technologies and Interaction, Mar. 19, 2019, 23 pages. [cited by applicant]
Rothe S., et al., “Guiding The Viewer In Cinematic Virtual Reality By Diegetic Cues,” In 5 th International Conference on AVR 2018: Augmented Reality, Virtual Reality and Computer Graphics, Jun. 24-27, 2018, 18 pages. [cited by applicant]
School of Rock the Musical, “School of Rock: The Musical—“You're in the Band” (360 Video),” Youtube, Oct. 14, 2015, 1 page, Retrieved from the Internet: URL: https://www.youtube.com/watch?v=GFRPXRhBYOl. [cited by applicant]
Serrano A., et al., “Movie Editing and Cognitive Event Segmentation in Virtual Reality Video,” ACM Transactions on Graphics, vol. 36, No. 4, Article 47, 2017, pp. 47:1-47:12. [cited by applicant]
Sheikh A., et al., “Directing Attention In 360-Degree Video,” IBC 2016 Conference, Sep. 8-12, 2016, 9 pages. [cited by applicant]
Shirazi S., “Timeline Visualization Of Omnidirectional Videos,” Master's thesis, May 2018, 52 pages. [cited by applicant]
Sitzmann V., et al., “How Do People Explore Virtual Environments?,” arXiv: 1612.04335v2, Sep. 19, 2017, 10 pages. [cited by applicant]
Speicher M., et al., “Exploring Visual Guidance In 360-Degree Videos,” In Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video, Jun. 4, 2019, pp. 1-12. [cited by applicant]
Wallgrun J.O., et al., “A Comparison Of Visual Attention Guiding Approaches For 360 Image-Based VR Tours,” In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) IEEE Annual International Symposium Virtu… [cited by applicant]
Wikipedia, “Call of Duty 4: Modern Warfare,” Activision, Nov. 5, 2007, 19 pages. [cited by applicant]
Woolwson R.F., “Wilcoxon Signed-Rank Test,” Wiley Encyclopedia of Clinical Trials, Mar. 9, 2007, pp. 1-3. [cited by applicant]
Youtube, “360 Google Spotlight Stories: HELP,” Google Spotlight Stories, Apr. 18, 2016, 2 pages, Retrieved from the Internet: URL: https://www.youtube.com/watch?v=G-XZhKqQAHU. [cited by applicant]
Youtube, “Pokemon 360—Catch Em All in VR1!,” Indie Film School, Sep. 13, 2016, 1 page, Retrieved from the Internet: URL: https://www.youtube.com/watch?v=pHUVS_GrleM. [cited by applicant]