Granted Patent B2

US 12,737,996 · App. 18/497,754 · Granted Sep 15, 2026

Inventors: James Allan Booth (Pacifica, CA); Mahdi Salmani Rahimi (San Francisco, CA); Gioacchino Noris (Zurich, CH)

Assignee: Meta Platforms Technologies, LLC

G06T19/006G06N20/00G06T15/205G06V40/10

Abstract

Claims (67)

1 . A method for providing a secured self representation of a user in an artificial reality (XR) environment, the method comprising:

receiving, by a secure application with permission to write to an output frame buffer, one or more images captured in real time by one or more cameras on an XR system;

selecting, by the secure application and a machine learning model trained to identify a portion of the user in an image, a secured self representation in each of the one or more images, wherein the secured self representation includes the portion of the user;

accessing, by the secure application, the frame buffer through which one or more other applications are providing XR content in the XR environment;

determining a portion of the frame buffer relative to the secured self representation; and

displaying, in the XR environment, the secured self representation by writing the secured self representation to the portion of the frame buffer after the one or more other applications have written to the frame buffer, wherein:

the one or more other applications cannot access the secured self representation written to the frame buffer; and

the one or more other applications cannot access the one or more images received by the secure application.

2 . The method of claim 1 , further comprising:

identifying a user movement based on identified movement of a controller or a tracked body part of the user;

determining one or more distances and directions of the user movement; and

based on the one or more determined distances and directions of the user movement, adjusting the secured self representation to conform to the identified movement.

3 . The method of claim 2 , wherein:

adjusting the secured self representation includes warping portions of the secured self representation that match the tracked body part of the user in accordance with the identified movement for the tracked body part of the user; and

warping portions of the secured self representation includes moving and/or resizing portions of the secured self representation that match the tracked body part of the user.

4 . The method of claim 1 further comprising adjusting at least part of the one or more images to appear to be from a user's perspective according to one or more distances between A) at least one eye of the user and B) multiple cameras on an artificial reality system.

5 . The method of claim 1 , wherein selecting the secured self representation includes:

generating an image mask based on an output of the machine learning model; and

applying the image mask to at least a portion of the one or more images to obtain the secured self representation of each of the one or more images.

6 . The method of claim 1 , wherein the machine learning model is trained using a set of images with portions of each image tagged to indicate whether that portion depicts a respective portion of a user or not.

7 . The method of claim 1 , wherein selecting the secured self representation in each respective image of the one or more images includes classifying parts of the respective image as depicting particular body parts of the user.

8 . The method of claim 7 further comprising:

receiving, from one of the one or more other applications, an indication of an effect to apply to a depiction of a particular body part of the user; and

applying, based on the classified parts of the respective image as depicting the particular body parts of the user, the effect to the depiction of the particular body part of the user.

9 . The method of claim 1 , further comprising

receiving, from one of the one or more other applications, an indication of an effect to apply to at least part of the displayed secured self representation; and

applying the effect to the secured self representation before the secured self representation is written to the portion of the frame buffer.

10 . The method of claim 9 , wherein the effect comprises one or more of: a color; a shading; a warp or distortion field; a composite layer to overlay onto the at least part of the secured self representation; or any combination thereof.

11 . A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for providing a secured self representation of a user in an artificial reality (XR) environment, the operations comprising:

receiving, by a secure application with permission to write to an output frame buffer, one or more images captured in real time by one or more cameras on an XR system;

selecting, by the secure application and a machine learning model trained to identify a portion of the user in an image, a secured self representation in each of the one or more images, wherein the secured self representation includes the portion of the user;

accessing, by the secure application, a frame buffer through which one or more other applications are providing XR content in the XR environment;

determining a portion of the frame buffer for the secured self representation; and

displaying, in the XR environment, the secured self representation by writing the secured self representation to the portion of the frame buffer after the one or more other applications have written to the frame buffer, wherein:

the one or more other applications cannot access the secured self representation written to the frame buffer; and

the one or more other applications cannot access the one or more images received by the secure application.

12 . The non-transitory computer-readable storage medium of claim 11 , wherein the operations further comprise:

identifying a user movement based on identified movement of a controller or a tracked body part of the user;

determining one or more distances and directions of the user movement; and

based on the one or more determined distances and directions of the user movement, adjusting the secured self representation to conform to the identified movement.

13 . The non-transitory computer-readable storage medium of claim 12 , wherein;

adjusting the secured self representation includes warping portions of the secured self representation that match the tracked body part of the user in accordance with the identified movements for the tracked body part of the user; and

warping portions of the secured self representation includes moving and/or resizing portions of the secured self representation that match the tracked body part of the user.

14 . The non-transitory computer-readable storage medium of claim 11 , wherein the operations further comprise adjusting at least part of the one or more images to appear to be from a user's perspective according to one or more distances between A) at least one eye of the user and B) multiple cameras on an artificial reality system.

15 . The non-transitory computer-readable storage medium of claim 11 , wherein selecting the secured self representation includes:

generating an image mask based on output of one or more machine learning models; and

applying the image mask to at least a portion of the one or more images to obtain the secured self representation portion of each of the one or more images.

16 . The non-transitory computer-readable storage medium of claim 15 , wherein the one or more machine learning models are trained using a set of images with portions of each image tagged to indicate whether that portion depicts a respective portion of a user or not.

17 . A computing system for providing a secured self representation of a user in an artificial reality (XR) environment, the computing system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:

receiving, by a secure application with permission to write to an output frame buffer, one or more images captured in real time by one or more cameras on an XR system;

selecting, by the secure application and a machine learning model trained to identify a portion of the user in an image, a secured self representation in each of the one or more images, wherein the secured self representation includes the portion of the user;

accessing, by the secure application, a frame buffer through which one or more other applications are providing XR content in the XR environment;

determining a portion of the frame buffer for the secured self representation; and

displaying, in the XR environment, the secured self representation by writing the secured self representation to the portion of the frame buffer after the one or more other applications have written to the frame buffer, wherein:

the one or more other applications cannot access the secured self representation written to the frame buffer; and

the one or more other applications cannot access the one or more images received by the secure application.

18 . The computing system of claim 17 ,

wherein selecting the secured self representation in each respective image of the one or more images includes classifying parts of the respective image as depicting particular body parts of the user; and

wherein the process further comprises:

receiving, from one of the one or more other applications, an indication of an effect to apply to a depiction of a particular body part of the user; and

applying, based on the classified parts of the respective image as depicting the particular body parts of the user, the effect to the depiction of the particular body part of the user.

19 . The computing system of claim 17 , wherein the process further comprises:

receiving, from one of the one or more other applications, an indication of an effect to apply to at least part of the displayed secured self representation; and

applying the effect to the secured self representation before the secured self representation is written to the portion of the frame buffer.

20 . The computing system of claim 19 , wherein the effect comprises one or more of: a color; a shading; a warp or distortion field; a composite layer to overlay onto the at least part of the secured self representation; or any combination thereof.

Assignments (1)

Continuity (3)

References Cited (267)