System and method for talking avatar
Aspects of this disclosure provide techniques for generating a viseme and corresponding intensity pair. In some embodiments, the method includes generating, by a server, a viseme and corresponding intensity pair based at least on one of a clean vocal track or corresponding transcription. The method may include generating, by the server, a compressed audio file based at least on one of the viseme, the corresponding intensity, music, or visual offset. The method may further include generating, by the server or a client end application, a buffer of raw pulse-code modulated (PCM) data based on decoding at least a part of the compressed audio file, where the viseme is scheduled to align with a corresponding phoneme.
1. A method for generating a viseme and corresponding intensity pair, comprising:
generating, by a server, a viseme and corresponding intensity pair based at least on one of a clean vocal track or corresponding transcription;
generating, by the server, a compressed audio file based at least on one of the viseme, the corresponding intensity, music, or visual offset; and
generating, by the server or a client end application, a buffer of raw pulse-code modulated (PCM) data based on decoding at least a part of the compressed audio file,
wherein the viseme is scheduled to align with a corresponding phoneme, and scheduled to coincide when a user hears a sound to allow two or more mouth shapes or facial expressions to be in an expected position using at least one visual offset, the at least one visual offset being used to compensate for longer blending between the two or more mouth shapes, and wherein the scheduling is based at least on one of:
a size of a decoder audio buffer for the compressed audio file,
a size of a processing buffer, or
a latency between transferring the decoder audio buffer to the client end application and the sound being heard.
2. The method of claim 1 further comprising storing the viseme and corresponding intensity pair in an intermediary file.
3. The method of claim 2 , wherein the viseme and corresponding intensity pair is stored in the intermediary file with specific timings for the viseme and corresponding intensity.
4. The method of claim 1 , wherein the frequency of decoding the compressed audio file is based on the size of the audio buffer.
5. The method of claim 1 further comprising feeding, by the server, the PCM data to a user equipment.
6. The method of claim 1 further comprising transmitting, by the server, the PCM data to a user equipment upon request.
7. The method of claim 1 further comprising scheduling the phoneme to align with at least one of a corresponding mouth shape or facial expression of an animated character.
8. A method for generating a viseme event and corresponding intensity pair, comprising:
generating, by a server, a viseme and corresponding intensity pair based at least on one of a clean vocal track or corresponding transcription;
generating, by the server, a compressed audio file based at least on one of the viseme, the corresponding intensity, music, or visual offset; and
inserting, by the server, a viseme generator based at least on one of a processing buffer or the compressed audio file,
wherein the viseme is scheduled to align with a corresponding phoneme, and scheduled to coincide when a user hears a sound to allow two or more mouth shapes or facial expressions to be in an expected position using at least one visual offset, the at least one visual offset being used to compensate for longer blending between the two or more mouth shapes, and wherein the scheduling is based at least on one of:
a size of a decoder audio buffer for the compressed audio file,
a size of the processing buffer, or
a latency between transferring the decoder audio buffer to a client end application and the sound being heard.
9. The method of claim 8 , wherein the scheduling of the viseme is based on the size of the processing buffer.
10. The method of claim 8 further comprising scheduling the phoneme to align with at least one of a corresponding mouth shape or facial expression of an animated character.
11. The method of claim 10 , wherein the corresponding mouth shape or facial expression of the animated character is created based on the viseme.
12. The method of claim 8 , wherein the frequency of decoding the compressed audio file is based on the size of the processing buffer.
13. The method of claim 8 further comprising feeding, by the server, the compressed audio file to a user equipment.
14. The method of claim 8 further comprising transmitting, by the server, the compressed audio file to a user equipment upon request.
15. A system for generating a viseme event and corresponding intensity pair, comprising:
a processor; and
a non-transitory computer readable storage medium storing programming for execution by the processor, the programming including instructions to:
generate a viseme and corresponding intensity pair based at least on one of a clean vocal track or corresponding transcription;
generate a compressed audio file based at least on one of the viseme, the corresponding intensity, music, or visual offset; and
generate a buffer of raw pulse-code modulated (PCM) data based on decoding at least a part of the compressed audio file,
wherein the viseme is scheduled to align with a corresponding phoneme, and scheduled to coincide when a user hears a sound to allow two or more mouth shapes or facial expressions to be in an expected position using at least one visual offset, the at least one visual offset being used to compensate for longer blending between the two or more mouth shapes, and wherein the scheduling is based at least on one of:
a size of a decoder audio buffer for the compressed audio file,
a size of a processing buffer, or
a latency between transferring the decoder audio buffer to a client end application and the sound being heard.
16. The system of claim 15 , wherein the programming includes further instructions to store the viseme and corresponding intensity pair in an intermediary file.
17. The system of claim 16 , wherein the viseme and corresponding intensity pair is stored in the intermediary file with specific timings for the viseme and intensity.
18. The system of claim 15 , wherein the frequency of decoding the compressed audio file is based on the size of the audio buffer.
19. The system of claim 15 , wherein the programming includes further instructions to feed the PCM data to a user equipment.
20. The system of claim 15 , wherein the programming includes further instructions to transmit the PCM data to a user equipment upon request.