System and method for talking avatar
Aspects of this disclosure provide techniques for generating a viseme and corresponding intensity pair. In some embodiments, the method includes generating, by a server, a viseme and corresponding intensity pair based at least on one of a clean vocal track or corresponding transcription. The method may include generating, by the server, a compressed audio file based at least on one of the viseme, the corresponding intensity, music, or visual offset. The method may further include generating, by the server or a client end application, a buffer of raw pulse-code modulated (PCM) data based on decoding at least a part of the compressed audio file, where the viseme is scheduled to align with a corresponding phoneme.
1. A method for generating a viseme and corresponding intensity pair, comprising:
generating, by a server, a viseme and corresponding intensity pair based at least on one of a clean vocal track or a corresponding transcription;
generating, by the server, a compressed audio file based on one or more of:
the viseme,
the corresponding intensity pair,
music data, or
visual offset data;
converting, by the server, the compressed audio file into an audio format that maintains low-latency for a real-time interactive communication by at least one digital character;
determining, by the server or a client system, a buffer indicating a decoding of at least a part of the compressed audio file based on one or more of:
a hardware configuration of the client system, or
an audio latency associated with executing the at least one digital character on the client system;
generating, by the server or the client system, the buffer comprising raw pulse-code modulated (PCM) data; and
initiating, by the server or the client system, the real-time interactive communication by the at least one digital character such that the viseme associated with the compressed audio file drives one or more of a mouth shape or a facial expression of the at least one digital character during the interactive communication, wherein:
the visual offset data is used to compensate a blend between two or more mouth shapes associated with the digital character, and
the viseme is scheduled to align with a corresponding phoneme based on at least one of:
a size of a decoder audio buffer for the compressed audio file,
a size of a processing buffer, or
a latency between transferring the decoder audio buffer to the client system and a sound being heard.
2. The method of claim 1 , wherein the visual offset data is used to execute one of delaying or advancing where the viseme occurs within the compressed audio file.
3. The method of claim 2 , wherein:
delaying where the viseme occurs is based on rapid switches between shapes associated with a mouth of the at least one digital character; and
advancing where the viseme occurs is based on longer blending between shapes associated with the mouth of the at least one digital character.
4. The method of claim 1 , wherein the size of the decoder audio buffer determines latency data associated with decoding the compressed audio file.
5. The method of claim 1 , wherein the at least one digital character is a subject-matter expert associated with an educational system.
6. The method of claim 1 , wherein the viseme and the corresponding intensity pair are stored in an intermediary file.
7. The method of claim 6 , wherein the intermediary file has an associated timing data indicating where the viseme and the corresponding pair occur within the intermediary file.
8. The method of claim 1 , wherein the viseme drives the mouth shape or the facial expression of the at least one digital character in a language learning application.
9. The method of claim 1 , wherein the viseme is a facial image or a facial expression that characterizes a particular sound.
10. The method of claim 1 , wherein generating the compressed audio file comprises combining at least the viseme, the music data, and the visual offset data to generate the compressed audio file.
11. The method of claim 1 , wherein converting the compressed audio file into an audio format that minimizes latency for a real-time interactive communication by at least one digital character comprises converting the compressed audio file into a compressed Opus file.
12. The method of claim 11 , wherein data associated with the viseme is embedded within the compressed Opus file as a tag.
13. A system for generating a viseme and corresponding intensity pair, the system comprising:
one or more computer processors, and
memory storing instructions which are executable by the one or more computer processors to:
generating a viseme and corresponding intensity pair based at least on one of a clean vocal track or a corresponding transcription;
generating a compressed audio file based on one or more of:
the viseme,
the corresponding intensity pair,
music data, or
visual offset data;
converting the compressed audio file into an audio format that-minimizes latency maintains low-latency for a real-time interactive communication by at least one digital character;
determining a buffer indicating a decoding of at least a part of the compressed audio file based on one or more of:
a hardware configuration of a client system, or
an audio latency associated with executing the at least one digital character on the client system;
generating the buffer comprising raw pulse-code modulated (PCM) data; and
initiating the real-time interactive communication by the at least one digital character such that the viseme associated with the compressed audio file drives one or more of a mouth shape or a facial expression of the at least one digital character during the interactive communication, wherein:
the visual offset data is used to compensate a blend between two or more mouth shapes associated with the digital character, and
the viseme is scheduled to align with a corresponding phoneme based on at least one of:
a size of a decoder audio buffer for the compressed audio file,
a size of a processing buffer, or
a latency between transferring the decoder audio buffer to the client system and a sound being heard.
14. The system of claim 13 , wherein the visual offset data is used to execute one of delaying or advancing where the viseme occurs within the compressed audio file.
15. The system of claim 14 , wherein:
delaying where the viseme occurs is based on rapid switches between shapes associated with a mouth of the at least one digital character; and
advancing where the viseme occurs is based on longer blending between shapes associated with the mouth of the at least one digital character.
16. The system of claim 13 , wherein the size of the decoder audio buffer determines latency data associated with decoding the compressed audio file.
17. The system of claim 13 , wherein the at least one digital character is a subject-matter expert associated with an educational system.
18. The system of claim 13 , wherein the viseme and the corresponding intensity pair are stored in an intermediary file.
19. The system of claim 13 , wherein the viseme drives the mouth shape or the facial expression of the at least one digital character in a language learning application.
20. The system of claim 13 , wherein converting the compressed audio file into an audio format that minimizes latency for a real-time interactive communication by at least one digital character comprises converting the compressed audio file into a compressed Opus file.