Systems and methods for multi-user video communication with engagement detection and adjustable fidelity
In one embodiment, a method includes maintaining a video communication between client devices with each being associated with a respective video stream, which is associated with a respective match scale measured based on a height of frames of the video stream and a depth of subjects within the frames, determining a respective scaling factor and cropping for each video stream, wherein the respective scaling factor is calculated based on the match scale associated with the corresponding video stream and a target match scale determined from the match scales associated with all video streams, and wherein the respective cropping is determined based on a uniformity of positions of the subjects within the frames across all video streams, generating a merged video stream from each video stream based on the respective scaling factor and cropping, and sending instructions for presenting the merged video stream to one or more of the client devices.
1. A method comprising, by an electronic device:
maintaining a video communication between two or more client devices, wherein each client device is associated with a respective video stream in the video communication, and wherein each video stream is associated with a respective match scale measured based on a height of frames of the video stream and a depth of subjects within the frames;
determining, for each of the video streams, a respective scaling factor and a respective cropping, wherein the respective scaling factor is calculated based on the match scale associated with the corresponding video stream and a target match scale determined from the match scales associated with the video streams associated with the two or more client devices, and wherein the respective cropping is determined based on a uniformity of positions of the subjects within the frames across the video streams associated with the two or more client devices;
generating, based on the respective scaling factor and cropping of each video stream, a merged video stream from each of the video streams for the video communication; and
sending, to one or more of the client devices, instructions for presenting the merged video stream.
2. The method of claim 1 , further comprising:
determining, for each of the video streams, a respective fidelity based on one or more of a date and time associated with the corresponding video stream, a calendar of a participant associated with the corresponding video stream, a command from the participant associated with the corresponding video stream, or a degree of engagement of the participant associated with the video stream, wherein generating the merged video stream is further based on the respective fidelity associated with each of the video streams.
3. The method of claim 1 , further comprising:
determining, for each of the video streams, a respective degree of engagement of a participant associated with the corresponding video stream based on one or more of the participant's presence, the participant's location, the participant's body direction, the participant's head direction, the participant's pose, or the participant's voice, wherein generating the merged video stream is further based on the respective degree of engagement of the participant associated with each of the video streams.
4. The method of claim 1 , further comprising:
determining, for each of the video streams, a respective adjustment of perspective associated with the corresponding video stream to center the subjects within the frames of the corresponding video stream, wherein generating the merged video stream is further based on the respective adjustment of perspective associated with each of the video streams.
5. The method of claim 1 , further comprising:
determining, for each of the video streams, a respective adjustment of color and lighting associated with the corresponding video stream based on a consistency of visual characteristics across the video streams, wherein generating the merged video stream is further based on the respective adjustment of color and lighting associated with each of the video streams.
6. The method of claim 1 , further comprising:
identifying borders between any two of the video streams, wherein generating the merged video stream comprises blurring or blending the identified borders.
7. The method of claim 1 , further comprising:
determining, for each of the video streams, a respective adjustment of space allocation associated with the corresponding video stream based on one or more of a count of participants present in the corresponding video stream, a degree of engagement of a participant in the corresponding video stream, or an activity level of a participant in the corresponding video stream, wherein generating the merged video stream is further based on the respective adjustment of space allocation associated with each of the video streams.
8. An electronic device comprising:
one or more displays;
one or more non-transitory computer-readable storage media including instructions; and
one or more processors coupled to the storage media, the one or more processors configured to execute the instructions to:
maintain a video communication between two or more client devices, wherein each client device is associated with a respective video stream in the video communication, and wherein each video stream is associated with a respective match scale measured based on a height of frames of the video stream and a depth of subjects within the frames;
determine, for each of the video streams, a respective scaling factor and a respective cropping, wherein the respective scaling factor is calculated based on the match scale associated with the corresponding video stream and a target match scale determined from the match scales associated with the video streams associated with the two or more client devices, and wherein the respective cropping is determined based on a uniformity of positions of the subjects within the frames across the video streams associated with the two or more client devices;
generate, based on the respective scaling factor and cropping of each video stream, a merged video stream from each of the video streams for the video communication; and
send, to one or more of the client devices, instructions for presenting the merged video stream.
9. The electronic device of claim 8 , wherein the processors are further configured to execute the instructions to:
determine, for each of the video streams, a respective fidelity based on one or more of a date and time associated with the corresponding video stream, a calendar of a participant associated with the corresponding video stream, a command from the participant associated with the corresponding video stream, or a degree of engagement of the participant associated with the video stream, wherein generating the merged video stream is further based on the respective fidelity associated with each of the video streams.
10. The electronic device of claim 8 , wherein the processors are further configured to execute the instructions to:
determine, for each of the video streams, a respective degree of engagement of a participant associated with the corresponding video stream based on one or more of the participant's presence, the participant's location, the participant's body direction, the participant's head direction, the participant's pose, or the participant's voice, wherein generating the merged video stream is further based on the respective degree of engagement of the participant associated with each of the video streams.
11. The electronic device of claim 8 , wherein the processors are further configured to execute the instructions to:
determine, for each of the video streams, a respective adjustment of perspective associated with the corresponding video stream to center the subjects within the frames of the corresponding video stream, wherein generating the merged video stream is further based on the respective adjustment of perspective associated with each of the video streams.
12. The electronic device of claim 8 , wherein the processors are further configured to execute the instructions to:
determine, for each of the video streams, a respective adjustment of color and lighting associated with the corresponding video stream based on a consistency of visual characteristics across the video streams, wherein generating the merged video stream is further based on the respective adjustment of color and lighting associated with each of the video streams.
13. The electronic device of claim 8 , wherein the processors are further configured to execute the instructions to:
identify borders between any two of the video streams, wherein generating the merged video stream comprises blurring or blending the identified borders.
14. The electronic device of claim 8 , wherein the processors are further configured to execute the instructions to:
determine, for each of the video streams, a respective adjustment of space allocation associated with the corresponding video stream based on one or more of a count of participants present in the corresponding video stream, a degree of engagement of a participant in the corresponding video stream, or an activity level of a participant in the corresponding video stream, wherein generating the merged video stream is further based on the respective adjustment of space allocation associated with each of the video streams.
15. A computer-readable non-transitory storage media comprising instructions executable by a processor to:
maintain a video communication between two or more client devices, wherein each client device is associated with a respective video stream in the video communication, and wherein each video stream is associated with a respective match scale measured based on a height of frames of the video stream and a depth of subjects within the frames;
determine, for each of the video streams, a respective scaling factor and a respective cropping, wherein the respective scaling factor is calculated based on the match scale associated with the corresponding video stream and a target match scale determined from the match scales associated with the video streams associated with the two or more client devices, and wherein the respective cropping is determined based on a uniformity of positions of the subjects within the frames across the video streams associated with the two or more client devices;
generate, based on the respective scaling factor and cropping of each video stream, a merged video stream from each of the video streams for the video communication; and
send, to one or more of the client devices, instructions for presenting the merged video stream.
16. The media of claim 15 , wherein the instructions are further executable by the processor to:
determine, for each of the video streams, a respective fidelity based on one or more of a date and time associated with the corresponding video stream, a calendar of a participant associated with the corresponding video stream, a command from the participant associated with the corresponding video stream, or a degree of engagement of the participant associated with the video stream, wherein generating the merged video stream is further based on the respective fidelity associated with each of the video streams.
17. The media of claim 15 , wherein the instructions are further executable by the processor to:
determine, for each of the video streams, a respective degree of engagement of a participant associated with the corresponding video stream based on one or more of the participant's presence, the participant's location, the participant's body direction, the participant's head direction, the participant's pose, or the participant's voice, wherein generating the merged video stream is further based on the respective degree of engagement of the participant associated with each of the video streams.
18. The media of claim 15 , wherein the instructions are further executable by the processor to:
determine, for each of the video streams, a respective adjustment of perspective associated with the corresponding video stream to center the subjects within the frames of the corresponding video stream, wherein generating the merged video stream is further based on the respective adjustment of perspective associated with each of the video streams.
19. The media of claim 15 , wherein the instructions are further executable by the processor to:
determine, for each of the video streams, a respective adjustment of color and lighting associated with the corresponding video stream based on a consistency of visual characteristics across the video streams, wherein generating the merged video stream is further based on the respective adjustment of color and lighting associated with each of the video streams.
20. The media of claim 15 , wherein the instructions are further executable by the processor to:
identify borders between any two of the video streams, wherein generating the merged video stream comprises blurring or blending the identified borders.
21. A method comprising, by an electronic device:
maintaining a video communication between two or more client devices, wherein each client device is associated with a respective video stream in the video communication; and
automatically adjusting, for each of the video streams, a respective amount of applied obfuscation to the video stream based on a degree of engagement, with the video communication, of a participant shown in the video stream, by (1) increasing the amount of applied obfuscation as the engagement of the participant with the video communication decreases and (2) decreasing the amount of applied obfuscation as the engagement of the participant with the video communication increases.
22. The method of claim 21 , further comprising:
determining, for each of the video streams, a respective degree of engagement of a participant associated with the corresponding video stream based on one or more of the participant's presence, the participant's location, the participant's body direction, the participant's head direction, the participant's pose, or the participant's voice.