Granted Patent B2

US 12,541,929 · App. 18/207,013 · Granted Feb 3, 2026

Inventors: Samuel Edward Hare (Los Angeles, CA); Andrew James McPhee (Culver City, CA); Tony Mathew (Los Angeles, CA)

Assignee: Snap Inc.

G06T19/006G06F3/04817G06F3/04845G06T7/70G11B27/036H04N9/8715G06F3/0488G06T2200/04G06T2200/24

Abstract

Claims (76)

1 . A method comprising:

playing back a stored video depicting a real-world scene;

during playback of the video, detecting interaction with an individual frame of the video for a threshold period of time, the individual frame corresponding to a paused play position;

in response to detecting the interaction with the individual frame of the video for the threshold period of time, pausing playback of the video;

while the video is paused on the individual frame, receiving a request to augment the stored video with a virtual object; and

adding the virtual object to a first frame of the video that precedes the individual frame, presented when the video is paused, instead of adding the virtual object to the individual frame of the video to create augmented video content depicting the real-world scene and the virtual object.

2 . The method of claim 1 , further comprising:

capturing movement information collected by a camera-enabled device during capture of the video;

storing the movement information comprising a plurality of inertial measurement unit (IMU) frames associated with respective timestamps;

processing the stored video to identify a real-world object in the real-world scene.

3 . The method of claim 2 , further comprising:

in response to receiving a request to augment the stored video with a virtual object:

retrieving the plurality of IMU frames associated with the stored video; and

matching the plurality of IMU frames with the stored video by correlating timestamps of one or more of the plurality of IMU frames with a timestamp of a frame of the stored video;

generating an interactive augmented reality display; and

adjusting, during playback of the video, an on-screen position of the virtual object within the augmented video based at least in part on matching the plurality of IMU frames with the stored video.

4 . The method of claim 1 , further comprising:

determining that a camera-enabled device is associated with a first classification of a plurality of classifications; and

in response to determining that the camera-enabled device is associated with the first classification, storing a first set of IMU information, wherein a second set of IMU information is stored for devices associated with a second classification of the plurality of classifications, wherein motion information represented by the second set of IMU information is of a different type and quality than motion information represented by the first set of IMU information, the first set of IMU information comprising preprocessed device motion data and the second set of IMU information comprising raw IMU data.

5 . The method of claim 4 , wherein the second set of IMU information comprise an orientation matrix of the camera-enabled device, raw accelerometer information for the camera-enabled device, and gyroscope information for the camera-enabled device, and wherein:

the virtual object comprises a virtual three-dimensional object; and

adding the virtual object comprises attaching the virtual three-dimensional object to the real-world scene.

6 . The method of claim 1 further comprising:

displaying a plurality of icons representing virtual objects while the video plays; and

receiving a user selection of the virtual object from the plurality of icons while the video plays.

7 . The method of claim 6 , further comprising resuming playback of the augmented video content in response to determining that further user input has not been received after selection of the virtual object after a given time interval.

8 . The method of claim 1 further comprising:

determining that the virtual object overlaps boundaries of first and second real-world objects depicted in the video;

computing an overlap amount of each boundary of the first and second real-world objects that the virtual object overlaps;

determining that the overlap amount of the second real-world object exceeds the overlap amount of the first real-world object; and

in response to determining that the overlap amount of the second real-world object exceeds the overlap amount of the first real-world object, selecting the second real-world object as a target real-world object to track.

9 . The method of claim 8 , further comprising:

attaching the virtual object to the target real-world object in response to receiving a user input that presses and holds the virtual object on a given frame for a threshold amount of time; and

attaching the virtual object to a second real-world object in response to:

receiving a first user input that touches the virtual object; and

receiving, within the threshold amount of time, a second user input that drags the virtual object to a position of the second real-world object.

10 . The method of claim 9 , further comprising presenting a grid underneath the virtual object in response to receiving the second user input.

11 . The method of claim 1 , further comprising presenting a plurality of icons each having a circular ground shadow positioned relative to a respective icon and receiving a user selection of one of the plurality of icons.

12 . The method of claim 1 , further comprising replacing the virtual object with another virtual object in response to receiving a user selection of the another virtual object, wherein the another virtual object retains properties of the virtual object that is replaced and is attached to a same real-world object to which the virtual object that is replaced was attached.

13 . The method of claim 1 , further comprising automatically rewinding the augmented video content at a predetermined rate to a beginning of the augmented video after adding the virtual object to the stored video that has been paused.

14 . A system comprising:

at least one processor configured to perform operations comprising:

playing back a stored video depicting a real-world scene;

during playback of the video, detecting interaction with an individual frame of the video for a threshold period of time, the individual frame corresponding to a paused play position;

in response to detecting the interaction with the individual frame of the video for the threshold period of time, pausing playback of the video;

while the video is paused on the individual frame, receiving a request to augment the stored video with a virtual object; and

adding the virtual object to a first frame of the video that precedes the individual frame, presented when the video is paused, instead of adding the virtual object to the individual frame of the video to create augmented video content depicting the real-world scene and the virtual object.

15 . The system of claim 14 , wherein the at least one processor is further configured to perform operations comprising:

capturing movement information collected by a camera-enabled device during capture of the video;

storing the movement information comprising a plurality of inertial measurement unit (IMU) frames associated with respective timestamps;

processing the stored captured video to identify a real-world object in the real-world scene.

16 . The system of claim 15 , wherein the at least one processor is further configured to perform operations comprising:

in response to receiving a request to augment the stored video with a virtual object:

retrieving the plurality of IMU frames associated with the stored video; and

matching the plurality of IMU frames with the stored video by correlating the timestamps of one or more of the plurality of IMU frames with a timestamp of a frame of the stored video;

generating an interactive augmented reality display; and

adjusting, during playback of the augmented video, an on-screen position of the virtual object within the augmented video content based at least in part on matching the plurality of IMU frames with the stored video.

17 . A non-transitory machine-readable storage medium including an augmented reality system that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

playing back a stored video depicting a real-world scene;

during playback of the video, detecting interaction with an individual frame of the video for a threshold period of time, the individual frame corresponding to a paused play position;

in response to detecting the interaction with the individual frame of the video for the threshold period of time, pausing playback of the video;

while the video is paused on the individual frame, receiving a request to augment the stored video with a virtual object; and

adding the virtual object to a first frame of the video that precedes the individual frame, presented when the video is paused, instead of adding the virtual object to the individual frame of the video to create augmented video content depicting the real-world scene and the virtual object.

18 . The non-transitory machine-readable storage medium of claim 17 , wherein the operations further comprise:

capturing movement information collected by a camera-enabled device during capture of the video;

storing the movement information comprising a plurality of inertial measurement unit (IMU) frames associated with respective timestamps;

processing the stored captured video to identify a real-world object in the real-world scene.

19 . The non-transitory machine-readable storage medium of claim 18 , wherein the operations further comprise:

in response to receiving a request to augment the stored video with a virtual object:

retrieving the plurality of IMU frames associated with the stored video; and

matching the plurality of IMU frames with the stored video by correlating the timestamps of one or more of the plurality of IMU frames with a timestamp of a frame of the stored video;

generating an interactive augmented reality display; and

adjusting, during playback of the augmented video content, an on-screen position of the virtual object within the augmented video based at least in part on matching the plurality of IMU frames with the stored video.

20 . The non-transitory machine-readable storage medium of claim 17 , wherein the operations further comprise:

determining that a camera-enabled device is associated with a first classification of a plurality of classifications; and

in response to determining that the camera-enabled device is associated with the first classification, storing a first set of IMU information, wherein a second set of IMU information is stored for devices associated with a second classification of the plurality of classifications, wherein motion information represented by the second set of IMU information is of a different type and quality than motion information represented by the first set of IMU information, the first set of IMU information comprising preprocessed device motion data and the second set of IMU information comprising raw IMU data.

Assignments (1)

Continuity (4)

References Cited (400)