IP Library Granted Patent US 11,315,301
Granted Patent B1
US 11,315,301 · App. 17/012,856 · Granted Apr 26, 2022

Rendering post-capture artificial-reality effects based on artificial-reality state information

Inventors: Jonathan Lim (San Francisco, CA); Yuanshuo Lu (Fremont, CA); Mohmmad Feisal Saleh Amir Rasras (Pittsburgh, PA)
Assignee: Meta Platforms, Inc.
G06T15/005G06K9/00201G06K9/00228G06T19/006G06T19/20H04N21/8146
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,315,301
App. No.
17/012,856
Granted
Apr 26, 2022
Kind
B1
Abstract

In one embodiment, a method includes retrieving a video stream that was recorded while a first artificial-reality effect was being displayed on the video stream, where each frame of the video stream comprises a real-world scene without the first artificial-reality effect, retrieving an artificial-reality state information stream corresponding to the video stream, where the artificial-reality state information stream comprises state information associated with the first artificial-reality effect, retrieving one or more contextual data streams corresponding to the video stream, where the first artificial-reality effect displayed on the video stream was rendered based on at least a portion of the one or more contextual data streams, rendering a second artificial-reality effect based on at least a portion of the artificial-reality state information stream and a portion of the one or more contextual data streams, and displaying the second artificial-reality effect on the video stream.

Claims (114)

1. A method comprising, by a computing device:

accessing a video stream that was recorded while a first artificial-reality effect was being displayed on the video stream, the first artificial-reality effect having one or more first non-deterministic features generated based on a randomness data stream generated using a randomness model;

retrieving the randomness data stream used for generating the one or more first non-deterministic features of the first artificial-reality effect displayed on the video stream;

rendering a second artificial-reality effect using the retrieved randomness data stream used for generating the one or more first non-deterministic features of the first artificial-reality effect, wherein the retrieved randomness data stream is used for generating one or more second non-deterministic features of the second artificial-reality effect; and

displaying the second artificial-reality effect on the video stream.

2. The method of claim 1 , further comprises:

retrieving one or more contextual data streams corresponding to the video stream, wherein the first artificial-reality effect displayed on the video stream was rendered based on at least a portion of the one or more contextual data streams, wherein the one or more contextual data streams comprise a sensor data stream generated by one or more sensors while the video stream is being captured.

3. The method of claim 2 , wherein the one or more sensors comprises one or more of:

an inertial measurement unit (IMU);

an accelerometer;

a device orientation sensor;

a motion sensor;

a velocity sensor;

a device position sensor;

a microphone;

a light sensor;

a touch sensor;

a stylus sensor;

a depth sensor;

a temperature sensor;

a GPS sensor; or

a user input sensor.

4. The method of claim 2 , wherein the one or more contextual data streams comprise a computed data stream generated by an object tracking algorithm performed on content of the video stream.

5. The method of claim 4 , wherein the computed data comprises one or more of:

face recognition data;

face tracking points;

person segmentation data;

object recognition data;

object tracking points;

object segmentation data;

body tracking points;

world tracking points;

a depth;

a point in a three-dimensional space;

a line in a three-dimensional space;

a surface in a three-dimensional space; or

a point cloud.

6. The method of claim 2 , wherein the second artificial-reality effect is the first artificial-reality effect, wherein the computing device renders the second artificial-reality effect identical to the first artificial-reality effect based on at least the randomness data stream and a portion of the one or more contextual data streams, and wherein one or more non-deterministic features of the second artificial-reality effect are generated based at least on the randomness data stream.

7. The method of claim 2 , further comprising:

receiving an indication, from one or more input sensors associated with the computing device, that a user associated with the computing device wants to switch to a third artificial-reality effect in a middle of replaying the video stream;

stopping rendering the second artificial-reality effect on the video stream;

rendering the third artificial-reality effect based on at least the randomness data stream and a portion of the one or more contextual data streams; and

displaying the third artificial-reality effect on the video stream.

8. The method of claim 1 , wherein the second artificial-reality effect is determined based on an input of a user associated with the computing device.

9. The method of claim 8 , wherein determining the second artificial-reality effect based on the input of the user comprises:

presenting choices for the second artificial-reality effect to the user;

receiving an indication of a user choice from one or more input sensors associated with the computing device; and

determining the second artificial-reality effect based on the indication of the user choice.

10. The method of claim 9 , wherein the choices for the second artificial-reality effect comprise no artificial-reality effect, the first artificial-reality effect, or one or more artificial-reality effects different from the first artificial-reality effect.

11. The method of claim 1 , wherein the one or more non-deterministic features comprise:

a size of a rain drop;

a path of a rain drop;

a timing of a rain drop;

a size of a snowflake;

a path of a snowflake;

a timing of a snowflake;

a direction of a flying arrow;

a trajectory of a flying arrow;

a timing of a flying arrow;

a size of a bubble;

a moving path of a bubble; or

a moving speed of a bubble.

12. The method of claim 1 , wherein the first artificial-reality effect comprises one or more of:

a virtual object;

a three-dimensional effect;

an interaction effect;

a displaying effect;

a sound effect;

a lighting effect; or

a tag.

13. The method of claim 1 , wherein each frame of the video stream comprises a real-world scene without the first artificial-reality effect.

14. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

access a video stream that was recorded while a first artificial-reality effect was being displayed on the video stream, the first artificial-reality effect having one or more first non-deterministic features generated based on a randomness data stream generated using a randomness model;

retrieve the randomness data stream used for generating the one or more first non-deterministic features of the first artificial-reality effect while displayed on the video stream;

render a second artificial-reality effect using the retrieved randomness data stream used for generating the one or more first non-deterministic features of the first artificial-reality effect, wherein the retrieved randomness data stream is used for generating one or more second non-deterministic features of the second artificial-reality effect; and

display the second artificial-reality effect on the video stream.

15. The media of claim 14 , wherein the software is further operable when executed to:

retrieve one or more contextual data streams corresponding to the video stream, wherein the first artificial-reality effect displayed on the video stream was rendered based on at least a portion of the one or more contextual data streams, wherein the one or more contextual data streams comprise a sensor data stream generated by one or more sensors while the video stream is being captured.

16. The media of claim 15 , wherein the one or more sensors comprises one or more of:

an inertial measurement unit (IMU);

an accelerometer;

a device orientation sensor;

a motion sensor;

a velocity sensor;

a device position sensor;

a microphone;

a light sensor;

a touch sensor;

a stylus sensor;

a depth sensor;

a temperature sensor;

a GPS sensor; or

a user input sensor.

17. The media of claim 15 , wherein the one or more contextual data streams comprise a computed data stream generated by an object tracking algorithm performed on content of the video stream.

18. The media of claim 17 , wherein the computed data comprises one or more of:

face recognition data;

face tracking points;

person segmentation data;

object recognition data;

object tracking points;

object segmentation data;

body tracking points;

world tracking points;

a depth;

a point in a three-dimensional space;

a line in a three-dimensional space;

a surface in a three-dimensional space; or

a point cloud.

19. The media of claim 14 , wherein each frame of the video stream comprises a real-world scene without the first artificial-reality effect.

20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:

access a video stream that was recorded while a first artificial-reality effect was being displayed on the video stream, the first artificial-reality effect having one or more first non-deterministic features generated based on a randomness data stream generated using a randomness model;

retrieve the randomness data stream used for generating the one or more first non-deterministic features of the first artificial-reality effect while displayed on the video stream;

render a second artificial-reality effect using the retrieved randomness data stream used for generating the one or more first non-deterministic features of the first artificial-reality effect, wherein the retrieved randomness data stream is used for generating one or more second non-deterministic features of the second artificial-reality effect; and

display the second artificial-reality effect on the video stream.

Assignments (1)
CHANGE OF NAME Recorded Dec 20, 2021
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058553/0802 →