Self presence in artificial reality

1. A method for providing a self representation of a user in an artificial reality environment, the method comprising:

as a first process:

receiving one or more images captured in real time by one or more cameras on an artificial reality system;

classifying a self portion in each of the one or more images by applying, to the one or more images, a machine learning model trained to identify a user's own body in an image; and

displaying, in the artificial reality environment and as the self representation, the self portion of at least one of the one or more images, at a virtual location relative to the user's perspective in the artificial reality environment; and

as a second process, before a new self portion is classified and displayed as the self representation:

identifying a user movement based on identified movement of a controller or a tracked body part of the user;

determining one or more distances and directions of the user movement; and

based on the one or more determined distances and directions of the user movement, adjusting the displayed self representation of the at least one image to conform to the identified movement.

2. The method of claim 1 ,

wherein the method further comprises, as a third process:

receiving one or more new images captured in real time by the one or more cameras on the artificial reality system;

classifying the new self portion in each of the one or more new images by applying, to the one or more new images, the machine learning model trained to identify a user's own body in an image; and

displaying, in the artificial reality environment and as the self representation, the new self portion of at least one of the one or more new images, at a new virtual location relative to the user's perspective in the artificial reality environment.

3. The method of claim 1 ,

wherein the identifying the user movement includes identifying movements for one or more user body parts; and

wherein the adjusting the displayed self representation includes warping portions of the self representation A) that match the identified one or more body parts and B) according to the identified movements for those one or more user body parts, wherein the warping includes moving and/or resizing portions of the self representation that match the identified one or more body parts.

4. The method of claim 1 ,

wherein the self representation is converted into a 3D object; and

wherein the adjusting the displayed self representation includes moving the 3D object to match the identified user movement.

5. The method of claim 1 ,

wherein the identified user movement is identified as a positional change represented by a set of vectors; and

wherein the adjusting the displayed self representation includes applying the set of vectors to move portions the displayed self representation.

6. The method of claim 1 further comprising determining that the identified user movement is below a threshold movement amount, wherein the adjusting the displayed self representation is in response to the determining that the identified user movement is below the threshold movement amount.

7. The method of claim 1 further comprising adjusting at least part of the one or more images to appear to be from a user's perspective according to one or more distances between A) at least one eye of the user and B) multiple cameras on an artificial reality system.

8. The method of claim 1 , wherein classifying the self portion includes:

generating an image mask based on the output of the machine learning model; and

applying the image mask to at least a portion of the one or more images to obtain the self portion of each of the one or more images.

9. The method of claim 1 , wherein the machine learning model is trained using a set of images with portions of each image tagged to indicate whether that portion depicts a self portion of a user or not.

10. The method of claim 1 , wherein classifying the self portion in each specific image of the one or more images includes classifying parts of the specific image as depicting particular body parts.

11. The method of claim 10 further comprising:

receiving, from an application controlling part of the artificial reality environment, an indication of an effect to apply to a depiction of a particular body part of the user; and

applying, based on the classified parts of the specific image as depicting particular body parts, the effect to the depiction of the particular body part of the user.

12. The method of claim 1 ,

wherein displaying the self portion includes overwriting a portion of a frame buffer, written to by an application controlling part of the artificial reality environment, with data for the self portion; and

wherein the application controlling part of the artificial reality environment does not have access to all of the data for the self portion.

13. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for providing a user self representation in an artificial reality environment, the operations comprising:

as a first process:

receiving one or more images captured in real time by one or more cameras on an artificial reality system;

classifying a self portion in each of the one or more images by applying, to the one or more images, a machine learning model trained to identify a user's own body in an image; and

displaying, in the artificial reality environment and as the self representation, the self portion of at least one of the one or more images, at a virtual location relative to the user's perspective in the artificial reality environment; and

as a second process, before a new self portion is classified and displayed as the self representation:

identifying a user movement based on identified movement of a controller or a tracked body part of the user;

determining one or more distances and directions of the user movement; and

based on the one or more determined distances and directions of the user movement, adjusting the displayed self representation of the at least one image to conform to the identified movement.

14. The non-transitory computer-readable storage medium of claim 13 ,

wherein the operations further comprise, as a third process:

receiving one or more new images captured in real time by the one or more cameras on the artificial reality system;

classifying the new self portion in each of the one or more new images by applying, to the one or more new images, the machine learning model trained to identify a user's own body in an image; and

displaying, in the artificial reality environment and as the self representation, the new self portion of at least one of the one or more new images, at a new virtual location relative to the user's perspective in the artificial reality environment.

15. The non-transitory computer-readable storage medium of claim 13 ,

wherein the identifying the user movement includes identifying movements for one or more user body parts; and

wherein the adjusting the displayed self representation includes warping portions of the self representation A) that match the identified one or more body parts and B) according to the identified movements for those one or more user body parts, wherein the warping includes moving and/or resizing portions of the self representation that match the identified one or more body parts.

16. The non-transitory computer-readable storage medium of claim 13 ,

wherein the identified user movement is identified as a positional change represented by a set of vectors; and

wherein the adjusting the displayed self representation includes applying the set of vectors to move portions the displayed self representation.

17. A computing system for providing a user self representation in an artificial reality environment, the computing system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a method comprising:

as a first process:

receiving one or more images captured in real time by one or more cameras on an artificial reality system;

classifying a self portion in each of the one or more images by applying, to the one or more images, a machine learning model trained to identify a user's own body in an image; and

displaying, in the artificial reality environment and as the self representation, the self portion of at least one of the one or more images, at a virtual location relative to the user's perspective in the artificial reality environment; and

as a second process, before a new self portion is classified and displayed as the self representation:

identifying a user movement based on identified movement of a controller or a tracked body part of the user;

determining one or more distances and directions of the user movement; and

based on the one or more determined distances and directions of the user movement, adjusting the displayed self representation of the at least one image to conform to the identified movement.

18. The computing system of claim 17 , wherein the method further comprises determining that the identified user movement is below a threshold movement amount, wherein the adjusting the displayed self representation is in response to the determining that the identified user movement is below the threshold movement amount.

19. The computing system of claim 17 ,

wherein displaying the self portion includes overwriting a portion of a frame buffer, written to by an application controlling part of the artificial reality environment, with data for the self portion; and

wherein the application controlling part of the artificial reality environment does not have access to all of the data for the self portion.