IP Library Granted Patent US 11,816,757
Granted Patent B1
US 11,816,757 · App. 16/710,738 · Granted Nov 14, 2023

Device-side capture of data representative of an artificial reality environment

Inventors: Melissa Erin Summers (Charlotte, NC); Lindsay Young (San Francisco, CA); Arthur Bodolec (New York City, NY); Adam Hupp (San Francisco, CA); Bryan Alger (Dallas, TX); Peter Aubrey Bartholomew Griess (Dallas, TX); Nataraja Kumar Koduri (Irving, TX); Carl Taylor (Palo Alto, CA); Bryan Justice (Sunnyvale, CA); Andrea Zeller (Bothell, WA); Ayfer Gokalp (Kirkland, WA)
Assignee: META PLATFORMS TECHNOLOGIES, LLC
G06T1/60G06F3/011G06F3/017G06T17/20G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,816,757
App. No.
16/710,738
Granted
Nov 14, 2023
Kind
B1
Abstract

The disclosure describes artificial reality systems and techniques for saving and exporting artificial reality data. For example, an artificial reality system includes an application engine configured to generate artificial reality content based on a pose of a user participating in an artificial reality environment and a head-mounted display (HMD) configured to output the artificial reality content to the user. The HMD includes a buffer configured to hold data representative of the artificial reality environment during a time window on a rolling basis and a capture engine configured to, in response to user input, capture the data representative of the artificial reality environment held in the buffer at a point in time at which the user input was received.

Claims (65)

1 . An artificial reality system comprising:

an application engine configured to generate artificial reality content based on a pose of a user participating in an artificial reality environment; and

a head-mounted display (HMD) configured to output the artificial reality content to the user, the HMD comprising:

a buffer configured to hold data representative of the artificial reality environment during a time window on a rolling basis, wherein the data representative of the artificial reality environment represents a view of the artificial reality environment from a position of the user, wherein the data representative of the artificial reality environment represents full state data that is more comprehensive than the artificial reality content output to the user, wherein the view of the artificial reality environment represents a 360 degree view of the artificial reality environment from the position of the user, wherein the view of the artificial reality environment includes data representative of one or more objects that are within a field of view of the HMD and data representative of one or more objects that are outside of the field of view of the HMD, and wherein to hold the data on the rolling basis, the buffer is configured to:

continuously save the full state data representative of the artificial reality environment representing the view of the artificial reality environment that is generated during the time window immediately preceding a current point in time, the view of the artificial reality environment including the data representative of the one or more objects that are within the field of view of the HMD and the data representative of the one or more objects that are outside of the field of view of the HMD; and

continuously drop the full state data representative of the artificial reality environment representing the view of the artificial reality environment that is generated prior to the time window, the view of the artificial reality environment including the data representative of the one or more objects that are within the field of view of the HMD and the data representative of the one or more objects that are outside of the field of view of the HMD; and

a capture engine configured to, in response to user input, capture the full state data representative of the artificial reality environment held in the buffer at a point in time at which the user input was received.

2 . The artificial reality system of claim 1 , wherein the full state data representative of the artificial reality environment held in the buffer at the point in time includes at least the view of the artificial reality environment including the data representative of the one or more objects that are within the field of view of the HMD and the data representative of the one or more objects that are outside of the field of view of the HMD immediately preceding the point in time at which the user input was received.

3 . The artificial reality system of claim 1 , wherein the application engine is further configured to:

receive information indicative of a request to save the full state data representative of the artificial reality environment during the time window permanently in a memory; and

save, in response to receiving the information, the full state data representative of the artificial reality environment during the time window permanently in the memory.

4 . The artificial reality system of claim 3 , wherein the application engine is further configured to perform one or both of:

replay, using the HMD or another device, the permanently saved full state data representative of the artificial reality environment during the time window; or

upload the permanently saved full state data representative of the artificial reality environment to one or more remote devices configured to host a social media account associated with the user in order to share the data with one or more other social media accounts connected to the social media account associated with the user.

5 . The artificial reality system of claim 1 , wherein the application engine is further configured to:

receive information indicative of a request to export, to one or more remote devices, the full state data representative of the artificial reality environment during the time window in order to share the data with one or more users of the one or more remote devices; and

export, in response to receiving the information, the full state data representative of the artificial reality environment to the one or more remote devices.

6 . The artificial reality system of claim 1 , wherein the application engine is further configured to:

receive information indicative of a request to export the full state data representative of the artificial reality environment during the time window to a content moderator system in order to report one or more rules infractions by at least one other user participating in the artificial reality environment; and

export, in response to receiving the information, the full state data representative of the artificial reality environment to the content moderator system.

7 . The artificial reality system of claim 1 , further comprising:

an image capture device configured to capture image data representative of a physical environment of the user;

a gesture detector configured to identify, from the image data, a user interface (UI) activation gesture performed by the user in a field of view of the image capture device; and

a rendering engine configured to, in response to identifying the UI activation gesture, render an options menu as an overlay to the artificial reality content for display on the HMD, wherein the options menu comprises a set of options.

8 . The artificial reality system of claim 7 , wherein the gesture detector is further configured to:

identify, from the image data, a user selection gesture performed by the user in a field of view of the image capture device,

wherein the user selection gesture indicates a user selected option of the set of options, and

wherein based on the user selected option, the capture engine captures the full state data representative of the artificial reality environment from the buffer.

9 . The artificial reality system of claim 8 , wherein the application engine is further configured to perform one or more of:

store, based on the user selected option of the set of options, the full state data representative of the artificial reality environment permanently in a memory;

export, based on the user selected option of the set of options, the full state data representative of the artificial reality environment to one or more remote devices in order to share the full state data representative of the artificial reality environment with one or more users of the one or more remote devices; and

export, based on the user selected option of the set of options, the full state data representative of the artificial reality environment to a content moderator system in order to report one or more rules infractions by at least one other user participating in the artificial reality environment.

10 . A method comprising:

generating, using an application engine, artificial reality content based on a pose of a user participating in an artificial reality environment;

outputting, using a head-mounted display (HMD), the artificial reality content to the user;

holding, using a buffer of the HMD, data representative of the artificial reality environment during a time window on a rolling basis, wherein the data representative of the artificial reality environment represents a view of the artificial reality environment from a position of the user, wherein the data representative of the artificial reality environment represents full state data that is more comprehensive than the artificial reality content output to the user, wherein the view of the artificial reality environment represents a 360 degree view of the artificial reality environment from the position of the user, wherein the view of the artificial reality environment includes data representative of one or more objects that are within a field of view of the HMD and data representative of one or more objects that are outside of the field of view of the HMD, and wherein holding the data on the rolling basis comprises:

continuously saving the full state data representative of the artificial reality environment representing the view of the artificial reality environment that is generated during the time window immediately preceding a current point in time, the view of the artificial reality environment including the data representative of the one or more objects that are within the field of view of the HMD and the data representative of the one or more objects that are outside of the field of view of the HMD; and

continuously dropping the full state data representative of the artificial reality environment representing the view of the artificial reality environment that is generated prior to the time window, the view of the artificial reality environment including the data representative of the one or more objects that are within the field of view of the HMD and the data representative of the one or more objects that are outside of the field of view of the HMD; and

capturing, by a capture engine of the HMD in response to user input, the full state data representative of the artificial reality environment held in the buffer at a point in time at which the user input was received.

11 . The method of claim 10 , further comprising:

receiving, by the application engine, information indicative of a request to save the full state data representative of the artificial reality environment during the time window permanently in a memory; and

saving, by the application engine in response to receiving the information, the full state data representative of the artificial reality environment during the time window permanently in the memory.

12 . The method of claim 11 , further comprising one or both of:

replaying, using the HMD or another device, the permanently saved full state data representative of the artificial reality environment during the time window; or

uploading the permanently saved full state data representative of the artificial reality environment to one or more remote devices configured to host a social media account associated with the user in order to share the data with one or more other social media accounts connected to the social media account associated with the user.

13 . The method of claim 10 , further comprising:

receiving information indicative of a request to export, to one or more remote devices, the data representative of the artificial reality environment during the time window in order to share the full state data with one or more users of the one or more remote devices; and

exporting, in response to receiving the information, the full state data representative of the artificial reality environment to the one or more remote devices.

14 . The method of claim 10 , further comprising:

receiving information indicative of a request to export the full state data representative of the artificial reality environment during the time window to a content moderator system in order to report one or more rules infractions by at least one other user participating in the artificial reality environment; and

exporting, in response to receiving the information, the full state data representative of the artificial reality environment to the content moderator system.

15 . The method of claim 10 , further comprising:

capturing, using an image capture device, image data representative of a physical environment of the user;

identifying, by a gesture detector from the image data, a user interface (UI) activation gesture performed by the user in a field of view of the image capture device; and

rendering, by a rendering engine in response to identifying the UI activation gesture using the gesture detector, an options menu as an overlay to the artificial reality content for display on the HMD, wherein the options menu comprises a set of options.

16 . The method of claim 15 , further comprising:

identifying, from the image data, a user selection gesture performed by the user in a field of view of the image capture device, wherein the user selection gesture indicates a user selected option of the set of options; and

capturing, by the capture engine, the full state data representative of the artificial reality environment based on the user selected option.

17 . A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to:

generate artificial reality content based on a pose of a user participating in an artificial reality environment;

output the artificial reality content to the user;

hold data representative of the artificial reality environment during a time window on a rolling basis, wherein the data representative of the artificial reality environment represents a view of the artificial reality environment from a position of the user, wherein the data representative of the artificial reality environment represents full state data that is more comprehensive than the artificial reality content output to the user, wherein the view of the artificial reality environment represents a 360 degree view of the artificial reality environment from the position of the user, wherein the view of the artificial reality environment includes data representative of one or more objects that are within a field of view of a head-mounted display (HMD) and data representative of one or more objects that are outside of the field of view of the HMD, and wherein to hold the data on the rolling basis, the instructions cause the one or more processors to:

continuously save the full state data representative of the artificial reality environment representing the view of the artificial reality environment that is generated during the time window immediately preceding a current point in time, the view of the artificial reality environment including the data representative of the one or more objects that are within the field of view of the HMD and the data representative of the one or more objects that are outside of the field of view of the HMD; and

continuously drop the full state data representative of the artificial reality environment representing the view of the artificial reality environment that is generated prior to the time window, the view of the artificial reality environment including the data representative of the one or more objects that are within the field of view of the HMD and the data representative of the one or more objects that are outside of the field of view of the HMD; and

capture the full state data representative of the artificial reality environment at a point in time at which the user input was received.

Assignments (2)
CHANGE OF NAME Recorded Jul 21, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060802/0799 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 15, 2020
From: SUMMERS, MELISSA ERIN; YOUNG, LINDSAY; BODOLEC, ARTHUR; ALGER, BRYAN; GRIESS, PETER AUBREY BARTHOLOMEW; KODURI, NATARAJA KUMAR; TAYLOR, CARL; JUSTICE, BRYAN; ZELLER, ANDREA; GOKALP, AYFER; HUPP, ADAM
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 051524/0892 →
Cited By (5)
US 12,222,930 US 12,361,661 US 12,462,470 US 12,561,005 US 12,646,261