Systems and methods to optimize video streaming using digital avatars
A digital avatar system can process video streams and generate synthetic video with a digital avatar. The digital avatar provides the appearance of a participant from the video stream talking and one or more of performing various behaviors or actions consistent with the participant's behavior when they are live streamed. A digital avatar system can detect triggering events during a live stream and automatically switch to an avatar mode.
1. A method comprising:
causing a presentation, at a first computing device, of a first video feed and audio data of a participant in a video streaming session, wherein the first video feed and the audio data is provided using sensors connected to a second computing device;
detecting a triggering event;
responsive to the triggering event, generating a second video feed depicting a digital avatar corresponding to the participant by:
retrieving an image associated with the participant from the first video feed;
generating an initialized avatar image using the image;
processing the initialized avatar image and the audio data, the processing including an animation of the initialized avatar image that is synchronized with the audio data; and
causing a presentation, using the first computing device, of the second video feed.
2. The method according to claim 1 , wherein detecting the triggering event includes using a Large Language Model.
3. The method according to claim 1 , wherein retrieving the image associated with the participant is further comprised of selecting an image from a frame of the first video feed of the participant.
4. The method according to claim 1 , wherein detecting the triggering event includes at least one of: detecting movement of the participant within the first video feed, determining and monitoring a relative position of the participant within a frame of the first video feed, detecting using object recognition, an object within the first video feed, and receiving a signal from an IoT device.
5. The method according to claim 1 , wherein the animation of the second video feed includes a realistic rendering of the participant.
6. The method according to claim 1 , wherein the method provides a seamless video feed of the participant in the video streaming session by dynamically switching between a recorded video feed of the participant and a synthetic video feed of the participant during the video streaming session.
7. The method according to claim 1 , wherein
detecting, after causing the presentation of the second video feed, a second triggering event; and
responsive to the second triggering event, causing a presentation, using the first computing device, of the first video feed of the participant.
8. The method according to claim 1 , wherein generating the second video feed further comprises:
generating animated facial data, using a first neural network, from the audio data; and
generating the second video feed, using a second neural network, from the animated facial data and from the selected image of the participant.
9. The method according to claim 1 , wherein displaying the second video feed includes replacing the first video feed with the second video feed.
10. A method comprising:
receiving, at a first computing device, a first video feed and audio data of a participant in a video streaming session, wherein the first video feed is provided using sensors connected to a second computing device;
detecting a triggering event;
responsive to the triggering event, generating a second video feed depicting a digital avatar corresponding to the participant by:
retrieving an image associated with the participant from the first video feed;
generating an initialized avatar image using the image;
processing the initialized avatar image and the audio data, the processing including an animation of the initialized avatar image that is synchronized with the audio data; and
sending the second video feed to a third computing device.
11. The method according to claim 10 , wherein detecting the triggering event includes using a Large Language Model.
12. The method according to claim 10 , wherein retrieving the image associated with the participant is further comprised of selecting an image from a frame of the first video feed of the participant.
13. The method according to claim 10 , wherein detecting the triggering event includes at least one of: detecting low lighting conditions in the first video feed, detecting a low battery condition for one of the first computing device or the second computing device, detecting that no video data is being received from the second computing device, and detecting a low bandwidth condition for the first computing device or the second computing device.
14. The method according to claim 10 , wherein generating the second video feed further comprises:
generating animated facial data, using a first neural network, from the audio data; and
generating the digital avatar video, using a second neural network, from the animated facial data and from the selected image of the participant.
15. The method according to claim 10 , wherein generating the second video feed further comprises:
retrieving a set of behavior data; and
processing the initialized avatar image using the set of behavior data.
16. The method according to claim 10 , wherein the method provides a seamless video feed of the participant in the video streaming session by dynamically switching between a recorded video feed of the participant and a synthetic video feed of the participant during the video streaming session.
17. The method according to claim 10 , wherein detecting the triggering event includes detecting that the participant is in a sensitive geographic location.
18. A system, comprising:
one or more processors to:
cause a presentation, at a first computing device, of a first video feed and audio data of a participant in a video streaming session, wherein the first video feed and the audio data is provided using sensors connected to a second computing device;
detect a triggering event;
responsive to the triggering event, generate a second video feed depicting a digital avatar corresponding to the participant by:
retrieving an image associated with the participant from the first video feed;
generating an initialized avatar image using the image;
processing the initialized avatar image and the audio data, the processing including an animation of the initialized avatar image that is synchronized with the audio data; and
cause a presentation, using the first computing device, of the second video feed.
19. The system according to claim 18 , wherein the system provides a seamless video feed of the participant in the video streaming session by dynamically switching between a recorded video feed of the participant and a synthetic video feed of the participant during the video streaming session.
20. The system according to claim 18 , wherein the one or more processors further:
select an image of the participant;
generate animated facial data, using a first neural network, from the audio data; and
generate the second video feed, using a second neural network, from the animated facial data and from the selected image of the participant.