System and method for replacing user media streams with animated avatars in live videoconferences
Example implementations provide the representation a remote user in a video-mediated meeting when the user webcam feed is not available or not used, such as if they are attending to a meeting via a wearable device without a camera, or are on-the-go and prefer not to display their webcam feed for privacy or bandwidth-related reasons. In such cases, the system will infer when the user is active in the meeting and allow users to select to display an animated set of keyframes (from past meetings or representing computer-based activity) as a proxy for the user representation. Example implementations may facilitate a richer representation of a meeting participant (as opposed to a static picture or no information) and may lead to enhanced social dynamics within the meeting.
1. An apparatus associated with a user in a video conference, comprising:
a memory configured to manage an association between a plurality of animations and a plurality of events, the plurality of events comprising one or more user activities and an interruption in the video stream, wherein each event of the plurality of events is associated with an animation from the plurality of animations, wherein each of the plurality of animations comprises a plurality of keyframes selected from video frames involving the user;
a camera; and
a processor, configured to:
for a detection of one or more events from video of the camera that is provided to a server configured to manage the video conference, provide instructions to the server to replace, in the video conference, the video from the apparatus with an animation selected from the plurality of animations in the memory that corresponds to the detected one or more events from the video of the camera.
2. The apparatus of claim 1 , wherein the detection of the one or more events comprises a detection of one or more persons other than the user in the video.
3. The apparatus of claim 1 , wherein the processor is configured to conduct detection of the one or more events based on feature extraction of one or more facial features of the user, and wherein the processor is configured to transmit the instructions to the server upon failure of detection of the one or more facial features of the user.
4. The apparatus of claim 1 , wherein the detection of the one or more events comprises a detection of a stoppage of the video.
5. The apparatus of claim 1 , wherein the video frames associated with the user are generated from the video of the camera during the video conference.
6. The apparatus of claim 1 , wherein the apparatus comprises one or more sensors, and wherein the processor is configured to, for the camera being disabled and for a detection of one or more events from the one or more sensors, transmit the instructions to the server to replace, in the video conference, the video from the apparatus with the animation.
7. A non-transitory computer readable medium storing instructions for an apparatus associated with a user in a video conference, the instructions comprising:
managing an association between a plurality of animations and a plurality of events, the plurality of events comprising one or more user activities and an interruption in the video stream, wherein each event of the plurality of events is associated with an animation from the plurality of animations, wherein each of the plurality of animations comprises a plurality of keyframes selected from video frames involving the user, and
for a detection of one or more events from video of a camera that is provided to a server configured to manage the video conference, providing instructions to the server to replace, in the video conference, the video from the apparatus with an animation selected from the plurality of animations that corresponds to the detected one or more events from the video of the camera.
8. The non-transitory computer readable medium of claim 7 , wherein the detection of the one or more events comprises a detection of one or more persons other than the user in the video.
9. The non-transitory computer readable medium of claim 7 , wherein the detection of the one or more events is based on feature extraction of one or more facial features of the user, and wherein the transmitting the instructions to the server is conducted upon failure of detection of the one or more facial features of the user.
10. The non-transitory computer readable medium of claim 7 , wherein the detection of the one or more events comprises a detection of a stoppage of the video.
11. The non-transitory computer readable medium of claim 7 , wherein the video frames associated with the user are generated from the video of the camera during the video conference.
12. The non-transitory computer readable medium of claim 7 , wherein the instructions further comprise, for the camera being disabled and for a detection of one or more events from one or more sensors, transmitting the instructions to the server to replace, in the video conference, the video from the apparatus with the animation.
13. A method for an apparatus associated with a user in a video conference, the method comprising:
managing an association between a plurality of animations and a plurality of events, the plurality of events comprising one or more user activities and an interruption in the video stream, wherein each event of the plurality of events is associated with an animation from the plurality of animations, wherein each of the plurality of animations comprising a plurality of keyframes selected from video frames involving the user, and
for a detection of one or more events from video of a camera that is provided to a server configured to manage the video conference, providing instructions to the server to replace, in the video conference, the video from the apparatus with an animation selected from the plurality of animations that corresponds to the detected one or more events from the video of the camera.
14. The method of claim 13 , wherein the detection of the one or more events comprises a detection of one or more persons other than the user in the video.
15. The method of claim 13 , wherein the detection of the one or more events is based on feature extraction of one or more facial features of the user, and wherein the transmitting the instructions to the server is conducted upon failure of detection of the one or more facial features of the user.
16. The method of claim 13 , wherein the detection of the one or more events comprises a detection of a stoppage of the video.
17. The method of claim 13 , wherein the video frames having the user are generated from the video of the camera during the video conference.
18. The method of claim 13 , further comprising, for the camera being disabled and for a detection of one or more events from one or more sensors, transmitting the instructions to the server to replace, in the video conference, the video from the apparatus with the animation.
19. The method of claim 13 , wherein the one or more sensors comprises at least one of a wearable device and a mobile device, wherein for the camera being disabled and for the at least one of the wearable device and the mobile device indicative of the user being active, transmitting the instructions to the server to replace, in the video conference, the video from the apparatus with an animation.
20. The method of claim 13 , wherein the plurality of keyframes of the animation are user selected from previous videoconference recordings involving the user.