Showing context in a communication session
Various implementations provide a representation of at least a portion of a user within a three-dimensional (3D) environment other than the user's physical environment. Based on detecting a condition, a representation of another object of the user's physical environment is shown to provide context. As examples, a representation of a sitting surface may be shown based on detecting that the user is sitting down, representations of a table and coffee cup may be shown based on detecting that the user is reaching out to pick up a coffee cup, a representation of a second user may be shown based on detecting a voice or the user turning his attention towards a moving object or sound, and a depiction of a puppy may be shown when the puppy's bark is detected.
1 . A method comprising:
at one or more processors:
generating a representation of at least a portion of a user based on an image of the user in a physical environment;
providing the representation for inclusion in a view of a three-dimensional (3D) environment provided by an electronic device that is remote from the physical environment;
determining that a condition to provide context regarding the physical environment is satisfied; and
based on determining that the condition to provide context is satisfied:
identifying an object of the physical environment;
determining characteristics of a context feature representing the object to limit an amount of information about the physical environment to reveal; and
providing the context feature representing the object of the physical environment for inclusion in the view of the 3D environment provided by the electronic device that is remote from the physical environment.
2 . The method of claim 1 , wherein the view of the 3D environment is provided during a communication session between electronic devices in different physical environments.
3 . The method of claim 2 , wherein the representation of at least the portion of the user is generated based on sensor data obtained during the communication session.
4 . The method of claim 1 , wherein determining that the condition to provide context is satisfied comprises determining that an action of the user corresponds to an interaction with the object or an intent to interact with the object.
5 . The method of claim 1 , wherein determining that the condition to provide context is satisfied comprises determining that an action of the user corresponds to sitting on the object or an intent to sit on the object, wherein the context feature representing the object comprises a representation of a sitting surface.
6 . The method of claim 1 , wherein determining that the condition to provide context is satisfied comprises determining that an action of the user corresponds to leaning on the object or an intent to lean on the object.
7 . The method of claim 1 , wherein determining that the condition to provide context is satisfied comprises determining that an action of the user corresponds to grasping the object or an intent to grasp the object.
8 . The method of claim 1 , wherein determining that the condition to provide context is satisfied comprises determining that an activity in the physical environment has a characteristic corresponding to attracting attention of the user or inspiring action by the user.
9 . The method of claim 1 , wherein determining that the condition to provide context is satisfied comprises using object classification or object classification to determine that the object is of a predefined type of object.
10 . The method of claim 1 , wherein providing the context feature comprises transmitting data defining the context feature to the electronic device.
11 . The method of claim 10 , wherein the data transmitted to the electronic device comprises:
a point cloud representing the object;
a point cloud representing an abstraction of the object;
a reconstruction of the object,
a generic object model based on the object;
object type information;
object pose information; or
object characteristic information.
12 . The method of claim 10 , wherein the electronic device includes the context feature by including a representation of the object in the view based on the transmitted data.
13 . The method of claim 12 , wherein the electronic device includes the context feature by:
including a received point cloud, a received reconstruction, or a received generic object model in the 3D environment; and
generating the view of the 3D environment with the received point cloud, the received reconstruction, or the received generic object model included.
14 . The method of claim 12 , wherein the electronic device includes the context feature by:
generating 3D content based on received object type information, received object pose information, or received object characteristic information;
including the 3D content in the 3D environment; and
generating the view of the 3D environment with the 3D content included.
15 . The method of claim 1 , wherein the context feature representing the object provides a photo-realistic representation of the object.
16 . The method of claim 1 , wherein the context feature representing the object comprises a generic representation based on a type of the object.
17 . The method of claim 1 , wherein the context feature representing the object comprises a generic representation based on a size and/or shape of the object.
18 . The method of claim 1 , wherein the context feature representing the object comprises a generic representation based on a function of the object or a type of interaction with the object.
19 . The method of claim 1 , wherein the characteristics of the context feature are determined to obfuscate a type of the object.
20 . The method of claim 1 , wherein the characteristics of the context feature are determined to obfuscate a size of the object.
21 . The method of claim 1 , wherein the characteristics of the context feature are determined to obfuscate a shape of the object.
22 . The method of claim 1 , wherein determining the characteristics of the context feature representing the object to limit the amount of information about the physical environment to reveal is based on receiving a consent of the user.
23 . The method of claim 1 , wherein determining the characteristics of the context feature representing the object to limit the amount of information about the physical environment to reveal is based on receiving a preference of the user.
24 . The method of claim 23 , wherein the preference of the user identifies whether particular objects or types of objects will be revealed.
25 . The method of claim 23 , wherein the preference of the user identifies an amount of detail to reveal.
26 . The method of claim 23 , wherein the preference of the user identifies to whom contextual features representing objects will be revealed.
27 . A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
generating a representation of at least a portion of a user based on an image of the user in a physical environment;
providing the representation for inclusion in a view of a three-dimensional (3D) environment provided by an electronic device that is remote from the physical environment;
determining that a condition to provide context regarding the physical environment is satisfied; and
based on determining that the condition to provide context is satisfied:
identifying an object of the physical environment;
determining characteristics of a context feature representing the object to limit an amount of information about the physical environment to reveal; and
providing the context feature representing the object of the physical environment for inclusion in the view of the 3D environment provided by the electronic device that is remote from the physical environment.
28 . A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
generating a representation of at least a portion of a user based on an image of the user in a physical environment;
providing the representation for inclusion in a view of a three-dimensional (3D) environment provided by an electronic device that is remote from the physical environment;
determining that a condition to provide context regarding the physical environment is satisfied; and
based on determining that the condition to provide context is satisfied:
identifying an object of the physical environment;
determining characteristics of a context feature representing the object to limit an amount of information about the physical environment to reveal; and
providing the context feature representing the object of the physical environment for inclusion in the view of the 3D environment provided by the electronic device that is remote from the physical environment.