Collaboration spaces in extended reality conference sessions
Extended reality (XR) software application programs establish remote collaboration sessions in which a host device and one or more remote devices can interact. When initiating a remote collaboration session, an XR application in a host device determines a collaboration area. The collaboration area corresponds to a portion of a real-world environment that is shared by the host device with the one or more remote devices. In some embodiments, the collaboration area can be determined automatically and/or based on user input. The XR application causes sensors associated with the host device to scan the collaboration area. Then, the XR application transmits, to the one or more remote devices, a three-dimensional representation of the collaboration area for rendering in one or more remote XR environments.
1. A computer-implemented method comprising:
initiating a remote conference session;
receiving, from one or more depth sensors associated with a host device, depth data associated with a three-dimensional (3D) environment;
receiving, from one or more image sensors associated with the host device, image data associated with the 3D environment;
determining a collaboration area around at least one automatically detected portion of a first user in the 3D environment, wherein the collaboration area is at least one of (i) a predefined 3D shape encompassing the at least one automatically detected portion of the first user, or (ii) determined based on a boundary of the at least one automatically detected portion of the first user, and wherein the at least one automatically detected portion can be moved or re-sized responsive to user interaction to control a size or location of the collaboration area;
generating a first 3D representation of the at least one automatically detected portion of the first user based on the depth data, the image data, and the collaboration area; and
transmitting the first 3D representation for rendering within a first extended reality (XR) environment associated with the remote conference session.
2. The method of claim 1 , further comprising:
receiving a second 3D representation of at least a portion of a second user; and
rendering, within a second XR environment associated with the remote conference session, the second 3D representation of the at least a portion of the second user.
3. The method of claim 1 , wherein determining the collaboration area comprises:
detecting, based on a machine learning model and at least one of the depth data or the image data, the boundary of the at least one automatically detected portion of the first user in the 3D environment; and
generating the collaboration area based on the boundary.
4. The method of claim 1 , wherein determining the collaboration area comprises:
detecting, based on a saliency model and at least one of the depth data or the image data, the boundary of the at least one automatically detected portion of the first user in the 3D environment; and
generating the collaboration area based on the boundary.
5. The method of claim 1 , wherein determining the collaboration area comprises:
determining, based on the depth data, a plane associated with the 3D environment;
receiving, from the first user, a selection of a location;
generating the collaboration area based on the location and the plane;
receiving, via one or more input devices, one or more interactions by the first user; and
modifying at least one of a size or a position of the collaboration area based on the one or more interactions.
6. The method of claim 1 , wherein determining the collaboration area comprises:
detecting the at least one automatically detected portion of the first user in the image data;
determining a mapping between the at least one automatically detected portion of the first user in the image data and the depth data; and
generating the collaboration area based on the mapping.
7. The method of claim 1 , wherein the collaboration area is a 3D region around the at least one automatically detected portion of the first user and one or more objects within the 3D environment.
8. The method of claim 1 , further comprising displaying, via a display device associated with the host device, an indication that image data associated with at least a portion of the collaboration area needs to be captured via the one or more image sensors.
9. The method of claim 1 , wherein generating the first 3D representation comprises:
generating, based on the depth data, one or more 3D meshes, wherein each 3D mesh included in the one or more 3D meshes comprises a set of vertices and a set of faces formed from edges between pairs of vertices in the set of vertices;
determining one or more vertices and one or more faces included in the one or more 3D meshes that are within the collaboration area; and
determining one or more textures associated with the one or more faces based on the image data.
10. The method of claim 1 , further comprising:
generating a 3D background associated with a portion of the 3D environment that is outside the collaboration area; and
transmitting, to the remote device, the 3D background for rendering in the first XR environment.
11. The method of claim 1 , further comprising:
receiving, from the one or more depth sensors, additional depth data associated with the 3D environment;
receiving, from the one or more image sensors, additional image data associated with the 3D environment;
determining, based on the additional depth data, changes in one or more geometries within the collaboration area; and
generating a second 3D representation of the first user based on the additional depth data, the additional image data, and the collaboration area; and
transmitting, to the remote device, the second 3D representation for rendering in the first XR environment.
12. The method of claim 1 , further comprising:
detecting, based on at least one of the depth data or the image data, a third user within the 3D environment;
determining another collaboration area around at least a portion of the third user in the 3D environment;
generating a third 3D representation of the at least a portion of the third user based on the depth data, the image data, and the another collaboration area; and
transmitting, to the remote device, the third 3D representation for rendering in the first XR environment.
13. One or more non-transitory computer-readable media storing program instructions that, when executed by at least one processor, cause the at least one processor to perform the steps of:
initiating a remote conference session;
receiving, from one or more depth sensors associated with a host device, depth data associated with a three-dimensional (3D) environment;
receiving, from one or more image sensors associated with the host device, image data associated with the 3D environment;
determining a collaboration area around at least one automatically detected portion of a first user in the 3D environment, wherein the collaboration area is at least one of (i) a predefined 3D shape encompassing the at least one automatically detected portion of the first user, or (ii) determined based on a boundary of the at least one automatically detected portion of the first user, and wherein the at least one automatically detected portion can be moved or re-sized responsive to user interaction to control a size or location of the collaboration area;
generating a first 3D representation of the at least one automatically detected portion of the first user based on the depth data, the image data, and the collaboration area; and
transmitting, to a remote device, the first 3D representation for rendering in a first extended reality (XR) environment associated with the remote conference session.
14. The one or more non-transitory computer-readable media of claim 13 , the steps further comprising:
receiving a second 3D representation of at least a portion of a second user; and
rendering, in a second XR environment associated with the remote conference session, the second 3D representation of the at least a portion of the second user.
15. The one or more non-transitory computer-readable media of claim 13 , wherein determining the collaboration area comprises:
detecting, based on at least one of a machine learning model or a saliency model and at least one of the depth data or the image data, the boundary of the at least one automatically detected portion of the first user in the 3D environment; and
generating the collaboration area based on the boundary.
16. The one or more non-transitory computer-readable media of claim 13 , wherein determining the collaboration area comprises:
determining, based on the depth data, a plane associated with the 3D environment;
receiving, from the first user, a selection of a location;
generating the collaboration area based on the location and the plane;
receiving, via one or more input devices, one or more interactions by the first user; and
modifying at least one of a size or a position of the collaboration area based on the one or more interactions.
17. The one or more non-transitory computer-readable media of claim 13 , wherein the collaboration area is a 3D region around the at least one automatically detected portion of the first user and one or more objects within the 3D environment.
18. The one or more non-transitory computer-readable media of claim 13 , wherein generating the first 3D representation comprises:
generating, based on the depth data, one or more 3D meshes, wherein each 3D mesh included in the one or more 3D meshes comprises a set of vertices and a set of faces formed from edges between pairs of vertices in the set of vertices;
determining one or more vertices and one or more faces included in the one or more 3D meshes that are within the collaboration area; and
determining one or more textures associated with the one or more faces based on the image data.
19. The one or more non-transitory computer-readable media of claim 13 , the steps further comprising:
generating a two-dimensional (2D) or 3D background associated with a portion of the 3D environment that is outside the collaboration area; and
transmitting, to the remote device, the 2D or 3D background for rendering in the first XR environment.
20. A system, comprising:
one or more memories storing instructions; and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
initiate a remote conference session,
receive, from one or more depth sensors associated with a host device, depth data associated with a three-dimensional (3D) environment, receive, from one or more image sensors associated with the host device, image data associated with the 3D environment,
determine a collaboration area around at least one automatically detected portion of a first user in the 3D environment, wherein the collaboration area is at least one of (i) a predefined 3D shape encompassing the at least one automatically detected portion of the first user, or (ii) determined based on a boundary of the at least one automatically detected portion of the first user, and wherein the at least one automatically detected portion can be moved or re-sized responsive to user interaction to control a size or location of the collaboration area,
generate a first 3D representation of the at least one automatically detected portion of the first user based on the depth data, the image data, and the collaboration area, and
transmit, to a remote device, the first 3D representation for rendering in a first extended reality (XR) environment associated with the remote conference session.