System and method enabling interactions in virtual environments with virtual presence

1. A system enabling spatial video-based virtual presence, comprising:

one or more server computers, at least one sending client device, and at least one receiving client device;

wherein the one or more server computers comprise at least one processor and memory storing data and instructions implementing a virtual environment platform comprising at least one virtual environment configured to open a peer-to-peer communication channel between the at least one sending client device and the at least one receiving client device;

wherein the at least one sending client device comprises at least one processor and memory storing data and instructions configured to, when executed by the at least one processor, cause the at least one sending client device to:

generate a user real-time 3D virtual cutout to transmit via the peer-to-peer communication channel by:

obtaining a 2D or 3D live video data feed from at least one camera;

performing, by the at least one sending client device, a background removal on the live video data feed; and

generating a polygonal structure to be used as a virtual frame to support the live video data feed having the background removed; and

transmit, by the at least one sending client device via the peer-to-peer communication channel, the user real-time 3D virtual cutout to the at least one receiving client device for presentation; and

wherein the at least one receiving client device comprises at least one processor and memory storing data and instructions configured to, when executed by the at least one processor, cause the at least one receiving client device to:

receive, by the at least one receiving client device via the peer-to-peer communication channel, the user real-time 3D virtual cutout from the at least one sending client device; and

insert and combine the user real-time 3D virtual cutout with the virtual environment by presenting the live video data feed having the background removed with the polygonal structure within the virtual environment; and

wherein the virtual environment is shared by the at least one sending client device and the at least one receiving client device and corresponding user real-time 3D virtual cutouts, enabling multi-user shared experiences with spatial video-based virtual presence within the virtual environment.

2. The system of claim 1 , wherein the data and instructions further cause the at least one sending client device to establish and automatically update a viewing perspective of the virtual environment provided to the at least one sending client device by using one or more virtual cameras placed virtually and aligned with respect to the user real-time 3D virtual cutout.

3. The system of claim 2 , wherein the alignment of the one or more virtual cameras is in front of the user real-time 3D virtual cutout and tracks movement of the user real-time 3D virtual cutout.

4. The system of claim 3 , wherein an orientation of the user real-time 3D virtual cutout is updated automatically by tracking and analyzing one or more of user eye-and-head-tilting data and head-rotation data using computer vision methods.

5. The system of claim 3 , wherein the one or more virtual cameras point outward from eye level, or wherein the one or more virtual cameras include two virtual cameras, one per eye, that point outward from the two-eye level, or wherein the one or more virtual cameras point outward from the center of the head-position of the user real-time 3D virtual cutout, or wherein the one or more virtual cameras point outward from the center of the user real-time 3D virtual cutout.

6. The system of claim 1 , wherein the one or more server computers include a spatially analyzed media server configured to manage, analyze and process incoming data of a plurality of sending client devices and in such analysis manages or optimizes transmission of data streams to a plurality of receiving client devices.

7. The system of claim 6 , wherein the spatially analyzed media server is further configured to:

in response to detecting that a distance between two or more user real-time 3D virtual cutouts has changed:

calculate a distance difference between the two or more user real-time 3D virtual cutouts;

update at least one of a position and an orientation of each user real-time 3D virtual cutout within the virtual environment;

update a corresponding viewing perspective respective to each other for each user real-time 3D virtual cutout; and

send the updated position, orientation and corresponding viewing perspective of each user real-time 3D virtual cutout to corresponding client devices.

8. The system of claim 1 , wherein the at least one sending client device is further configured to perform further processing or improvements on the user real-time 3D virtual cutout.

9. The system of claim 1 , wherein the background removal is performed by applying image segmentation through one or more of instance segmentation or semantic segmentation, and usage of deep neural networks by the sending client device or one or more server computers.

10. The system of claim 1 , wherein the instructions further cause the at least one sending client device to:

generate a user 3D virtual cutout by:

obtaining a photo of a user; and

generating a 3D mesh or point cloud for the user 3D virtual cutout based on the photo; and

selectively transmitting the user 3D virtual cutout or the user real-time 3D virtual cutout for presentation by the at least one receiving client device.

11. A method enabling spatial video-based presence, comprising:

opening a peer-to-peer communication channel between at least one sending client device and at least one receiving client device;

obtaining, by the at least one sending client device, a 2D or 3D live video data feed;

generating a user real-time 3D virtual cutout by performing actions including:

performing, by the at least one sending client device, a background removal on the live video data feed; and

generating a polygonal structure to be used as a virtual frame to support the live video data feed;

and transmitting, by the at least one sending client device via the peer-to-peer communication channel, the user real-time 3D virtual cutout to the at least one receiving client device, wherein the at least one receiving client device is configured to insert and combine the user real-time 3D virtual cutout with a virtual environment implemented by at least one server computer for presentation by the at least one receiving client device of the user real-time 3D virtual cutout within the virtual environment,

wherein the virtual environment is shared by the at least one sending client device and the at least one receiving client device and corresponding user real-time 3D virtual cutouts, enabling multi-user shared experiences with virtual presence within the virtual environment.

12. The method of claim 11 , further comprising establishing and automatically updating a viewing perspective of the virtual environment provided to the at least one sending client device using one or more virtual cameras placed virtually and aligned with respect to the user real-time 3D virtual cutout.

13. The method of claim 12 , wherein the alignment of the one or more virtual cameras is in front of the user real-time 3D virtual cutout and tracks movement of the user real-time 3D virtual cutout.

14. The method of claim 13 , further comprising updating an orientation of the user real-time 3D virtual cutout by tracking and analyzing one or more of user eye-and-head-tilting data and head-rotation data using computer vision methods.

15. The method of claim 14 , wherein the one or more virtual cameras point outward from eye level, or wherein the one or more virtual cameras include two virtual cameras, one per eye, that point outward from the two-eye level, or wherein the one or more virtual cameras point outward from the center of the head-position of the user real-time 3D virtual cutout, or wherein the one or more virtual cameras point outward from the center of the user real-time 3D virtual cutout.

16. The method of claim 11 , wherein the at least one server computer is a spatially analyzed media server configured to manage, analyze and process incoming data of a plurality of sending client devices and in such analysis manages or optimizes the transmission of data streams to a plurality of receiving client devices.

17. The method of claim 16 , further comprising:

in response to detecting that the distance between two or more user real-time 3D virtual cutouts has changed, calculating, by the spatially analyzed media server, a distance difference between the two or more user real-time 3D virtual cutouts;

updating at least one of a position and an orientation of each user real-time 3D virtual cutout within the virtual environment;

updating a corresponding viewing perspective respective to each other for each user real-time 3D virtual cutout, and

sending the updated position, orientation and corresponding viewing perspective of each user real-time 3D virtual cutout to corresponding client devices.

18. The method of claim 11 , further comprising performing further processing or improvements on the user real-time 3D virtual cutout by the at least one sending client device.

19. The method of claim 11 , wherein the background removal is performed by applying image segmentation through one or more of instance segmentation or semantic segmentation, and usage of deep neural networks by the sending client device or one or more server computers.

20. A non-transitory computer readable medium having stored thereon instructions configured to cause a computer system comprising at least one sending client device comprising a processor and memory to perform steps comprising:

obtaining, by the at least one sending client device, a 2D or 3D live video data feed;

generating, by the at least one sending client device, a user real-time 3D virtual cutout by performing actions including performing, by the at least one sending client device, a background removal on the live video data feed;

generating a polygonal structure to be used as a virtual frame to support the live video data feed; and

transmitting, by the at least one sending client device via a peer-to-peer communication channel, the user real-time 3D virtual cutout to at least one receiving client device, wherein the at least one receiving client device is configured to insert and combine the user real-time 3D virtual cutout with a virtual environment implemented by at least one server computer for presentation by the at least one receiving client device of the user real-time 3D virtual cutout within the virtual environment,

wherein the virtual environment is shared by the at least one sending client device and the at least one receiving client device and corresponding user real-time 3D virtual cutouts, enabling multi-user shared experiences with virtual presence within the virtual environment.