Virtual viewpoint for a participant in an online communication
Implementations provide an in-person communication experience by generating a changable virtual viewpoint for a participant in an online communication. For instance, techniques described herein capture visual data about participants in an online communication, and create a realistic geometric proxy from the visual data. A virtual scene geometry is generated that mimics an arrangement of an in-person meeting for the online communication. According to various implementations, a virtual viewpoint displays a changing virtual viewpoint, such as based on a change in position of a participants face.
1. A system comprising:
one or more processors; and
one or more computer-readable storage media storing instructions that are executable by the one or more processors to perform operations including:
generating a scene geometry including a geometric proxy for a particular participant and a different geometric proxy for a different participant in an online communication, the scene geometry visually simulating an in-person conversation;
causing a virtual viewpoint to be displayed for the particular participant that includes the different geometric proxy for the different participant along with a background that is part of the scene geometry, the virtual viewpoint being displayed based on a detected position of the particular participant's face such that a change in the detected position of the particular participant's face causes a change in the virtual viewpoint of the different geometric proxy of the different participant;
drawing a first virtual eye gaze box around eyes of the geometric proxy and a second virtual eye gaze box around eyes of the different geometric proxy;
aligning the first virtual eye gaze box to be level with the second virtual eye gaze box; and
displaying, in the virtual viewpoint, the eyes of the different geometric proxy based on the alignment of the first virtual eye gaze box and the second virtual eye gaze box.
2. The system of claim 1 , wherein the operations further include:
keeping the virtual viewpoint level with an eye gaze of the local participant using a face tracking technique.
3. The system of claim 1 , wherein the virtual viewpoint is created using a virtual camera that is a composition of a plurality of images from a plurality of cameras that acquire images of the particular participant.
4. The system of claim 1 , wherein the operations further include:
determining that there are at least two participants to the online communication and; and
the drawing of the first virtual eye gaze box and the second virtual eye gaze box is in response to the determination.
5. The system of claim 1 , wherein the operations further include:
changing the virtual viewpoint for the particular participant based on a face tracking technique that tracks a face of the particular participant.
6. The system of claim 1 , wherein the geometric proxy for the particular participant is constructed from a point cloud created from a plurality of depth points representing depth information corresponding to the particular participant.
7. The system of claim 1 , wherein the operations further include:
adding motion parallax to the virtual viewpoint using head tracking of the particular participant, wherein the motion parallax is dependent on an eye gaze of the particular participant.
8. A method comprising
generating, by one or more processors, a scene geometry including a geometric proxy for a particular participant and a different geometric proxy for a different participant in an online communication, the scene geometry visually simulating an in-person conversation;
causing, by the one or more processors, a virtual viewpoint to be displayed for the particular participant that includes the different geometric proxy for the different participant along with a background that is part of the scene geometry, the virtual viewpoint being displayed based on a detected position of the particular participant's face such that a change in the detected position of the particular participant's face causes a change in the virtual viewpoint of the different geometric proxy of the different participant;
drawing, by the one or more processors, a first virtual eye gaze box around eyes of the geometric proxy and a second virtual eye gaze box around eyes of the different geometric proxy;
aligning, by the one or more processors, the first virtual eye gaze box to be level with the second virtual eye gaze box; and
displaying, in the virtual viewpoint, the eyes of the different geometric proxy based on the alignment of the first virtual eye gaze box and the second virtual eye gaze box.
9. The method of claim 8 , further comprising:
keeping the virtual viewpoint level with an eye gaze of the local participant using a face tracking technique.
10. The method of claim 8 wherein the virtual viewpoint is created using a virtual camera that is a composition of a plurality of images from a plurality of cameras that acquire images of the particular participant.
11. The method of claim 8 , further comprising:
determining that there are at least two participants to the online communication; and wherein
the drawing of the first virtual eye gaze box and the second virtual eye gaze box is in response to the determination.
12. The method of claim 8 , further comprising:
changing the virtual viewpoint for the particular participant based on a face tracking technique that tracks a face of the particular participant.
13. The method of claim 8 , wherein the geometric proxy for the particular participant is constructed from a point cloud created from a plurality of depth points representing depth information corresponding to the particular participant.
14. The method of claim 8 , further comprising:
adding motion parallax to the virtual viewpoint using head tracking of the particular participant, wherein the motion parallax is dependent on an eye gaze of the particular participant.
15. A computer-readable storage media storing computer-executable instructions that, when executed by one or more processors, cause a system to perform a plurality of operations comprising:
generating a scene geometry including a geometric proxy for a particular participant and a different geometric proxy for a different participant in an online communication, the scene geometry visually simulating an in-person conversation;
causing a virtual viewpoint to be displayed for the particular participant that includes the different geometric proxy for the different participant along with a background that is part of the scene geometry, the virtual viewpoint being displayed based on a detected position of the particular participant's face such that a change in the detected position of the particular participant's face causes a change in the virtual viewpoint of the different geometric proxy of the different participant;
drawing a first virtual eye gaze box around eyes of the geometric proxy and a second virtual eye gaze box around eyes of the different geometric proxy;
aligning the first virtual eye gaze box to be level with the second virtual eye gaze box; and
displaying, in the virtual viewpoint, the eyes of the different geometric proxy based on the alignment of the first virtual eye gaze box and the second virtual eye gaze box.
16. The computer-readable storage media of claim 15 , wherein the plurality of operations further comprise:
keeping the virtual viewpoint level with an eye gaze of the local participant using a face tracking technique.
17. The computer-readable storage media of claim 15 , wherein the virtual viewpoint is created using a virtual camera that is a composition of a plurality of images from a plurality of cameras that acquire images of the particular participant.
18. The computer-readable storage media of claim 15 , wherein the plurality of operations further comprise:
determining that there are at least two participants to the online communication and; and
wherein the drawing of the first virtual eye gaze box and the second virtual eye gaze box is in response to the determination.
19. The computer-readable storage media of claim 15 , wherein the geometric proxy for the particular participant is constructed from a point cloud created from a plurality of depth points representing depth information corresponding to the particular participant.
20. The computer-readable storage media of claim 15 , wherein the plurality of operations further comprise:
adding motion parallax to the virtual viewpoint using head tracking of the particular participant, wherein the motion parallax is dependent on an eye gaze of the particular participant.