Translating a media asset with vocal characteristics of a speaker
Systems and methods are described herein for generating alternate audio for a media stream. The media system receives media that is requested by the user. The media comprises a video and audio. The audio includes words spoken in a first language. The media system stores the received media in a buffer as it is received. The media system separates the audio from the buffered media and determines an emotional state expressed by spoken words of the first language. The media system translates the words spoken in the first language into words spoken in a second language. Using the translated words of the second language, the media system synthesizes speech having the emotional state previously determined. The media system then retrieves the video of the received media from the buffer and synchronizes the synthesized speech with the video to generate the media content in a second language.
1. A method comprising:
accessing a media asset, the media asset featuring a speaker that utters a plurality of spoken words;
determining an identity of the speaker in the media asset based on metadata associated with the media asset;
identifying vocal characteristics of the identified speaker by:
searching, based on the identity, for another media asset featuring the speaker;
extracting a voice sample featuring the speaker from the another media asset;
identifying the vocal characteristics based on the voice sample from the another media asset;
determining non-linguistic characteristics of the plurality of spoken words;
determining an emotional state expressed in the media asset featuring the speaker that utters the plurality of spoken words based on the non-linguistic characteristics; and
generating a translation of the plurality of spoken words of the media asset featuring the speaker that utters the plurality of spoken words using the identified vocal characteristics, and the determined emotional state.
2. The method of claim 1 , comprising:
accessing audio from the media asset; and
generating based on the accesed audio a converted audio using the identified vocal characteristics, and the determined emotional state.
3. The method of claim 1 , comprising:
translating the plurality of spoken words from a first language into a second language;
transcribing the plurality of spoken words in the first language;
translating the transcribed words of the first language into translated words of the second language; and
synthesizing a speech from the translated words of the second language.
4. The method of claim 3 , comprising:
accessing, from a language translation database, rules of grammar of the first language;
determining, using the rules of grammar of the first language, parts of speech of the translated words of the first language;
accessing, from the language translation database, rules of grammar of the second language; and
translating the translated words of the first language into the translated words of the second language based on the parts of speech of the translated words of the first language and the rules of grammar of the second language.
5. The method of claim 1 , comprising:
determining whether the plurality of spoken words contains emotion identifiers;
in response to determining that the plurality of spoken words contains emotion identifiers:
comparing the emotion identifier to a plurality of emotional identifiers stored in a database; and
determining the emotional state expressed in the media asset based on the comparing.
6. The method of claim 1 , comprising:
comparing the non-linguistic characteristics to a particular set of characteristics stored in a language conversion database; and
determining the emotional state expressed in the media asset based on the comparing.
7. The method of claim 2 , comprising:
accessing video from the received media asset; and
generating for output the accessed video and the converted audio, wherein the converted audio is output with the video instead of the plurality of spoken words,
wherein generating the converted audio comprises:
determining, based on the non-linguistic characteristics, a gender of the speaker that produces the plurality of spoken words;
accessing, from a language conversion database, second non-linguistic characteristics based on the determined gender of the speaker; and
synthesizing a speech using the second non-linguistic characteristics.
8. The method of claim 1 , comprising:
determining, based on the non-linguistic characteristics, an ethnicity of the speaker that produces the plurality of spoken words;
accessing, from a language conversion database, second non-linguistic characteristics based on the determined ethnicity of the speaker; and
synthesizing a speech using the second non-linguistic characteristics.
9. The method of claim 1 , comprising:
determining an identity of the speaker that produces the plurality of spoken words;
accessing, from a language conversion database, a plurality of voice samples of the speaker;
calculating a vocal fingerprint of the speaker using the plurality of voice samples; and
synthesizing a speech based on the calculated vocal fingerprint.
10. The method of claim 1 , comprising:
receiving an input to generate alternate audio for the media asset stream; and
generating the translation in response to the received user input.
11. A system comprising circuitry configured to:
access a media asset, the media asset featuring a speaker that utters a plurality of spoken words;
determine an identity of the speaker in the media asset based on metadata associated with the media asset;
identify vocal characteristics of the identified speaker by:
search, based on the identity, for another media asset featuring the speaker;
extract a voice sample featuring the speaker from the another media asset featuring the speaker; and
identify the vocal characteristics based on the voice sample from the another media asset featuring the speaker;
determine non-linguistic characteristics of the plurality of spoken words;
determine an emotional state expressed in the media asset featuring the speaker that utters the plurality of spoken words based on linguistic characteristics; and
generate a translation of the plurality of spoken words of the media asset featuring the speaker that utters the plurality of spoken words using the identified vocal characteristics, and the determined emotional state.
12. The system of claim 11 , wherein the circuitry is configured to:
access audio from the media asset: and
generate based on the accessed audio a converted audio using the identified vocal characteristics, and the determined emotional state.
13. The system of claim 12 , wherein the circuitry is configured to:
translate the plurality of spoken words from a first language into a second language;
transcribe the plurality of spoken words in the first language;
translate the transcribed words of the first language into translated words of the second language; and
synthesizing a speech from the translated words of the second language.
14. The system of claim 13 , wherein the circuitry is configured to:
access, from a language translation database, rules of grammar of the first language;
determine, using the rules of grammar of the first language, parts of speech of the translated words of the first language;
access, from the language translation database, rules of grammar of the second language; and
translate the translated words of the first language into the translated words of the second language based on the parts of speech of the translated words of the first language and the rules of grammar of the second language.
15. The system of claim 11 , wherein the circuitry is configured to:
determine whether the plurality of spoken words contains emotion identifiers; and
in response to determining that the plurality of spoken words contains emotion identifiers:
compare the emotion identifier to a plurality of emotional identifiers stored in a database; and
determine the emotional state expressed in the media asset based on the comparing.
16. The system of claim 11 , wherein the circuitry is configured to:
compare the non-linguistic characteristics to a particular set of characteristics stored in a language conversion database; and
determine the emotional state expressed in the media asset based on the comparing.
17. The system of claim 12 , wherein the circuitry is configured to:
access video from the received media asset; and
generate for output the accessed video and the converted audio, wherein the converted audio is output with the video instead of the plurality of spoken words;
wherein the circuitry, when generating the converted audio, is further configured to:
determine, based on the non-linguistic characteristics, a gender of the speaker that produces the plurality of spoken words;
access, from a language conversion database, second non-linguistic characteristics based on the determined gender of the speaker; and
synthesize a speech using the second non-linguistic characteristics.
18. The system of claim 11 , wherein the circuitry is configured to:
determine, based on the non-linguistic characteristics, an ethnicity of the speaker that produces the plurality of spoken words;
access, from a language conversion database, second non-linguistic characteristics based on the determined ethnicity of the speaker; and
synthesize a speech using the second non-linguistic characteristics.
19. The system of claim 11 , wherein the circuitry is configured to:
determine an identity of the speaker that produces the plurality of spoken words;
access, from a language conversion database, a plurality of voice samples of the speaker;
calculate a vocal fingerprint of the speaker using the plurality of voice samples; and
synthesize a speech based on the calculated vocal fingerprint.
20. The system of claim 11 , wherein the circuitry is further configured to:
receive an input to generate alternate audio for the media asset; and
generate the translation in response to the received user input.
21. A non-transitory computer readable medium comprising instructions that when executed cause circuitry to:
access a media asset, the media asset featuring a speaker that produce produces a plurality of spoken words;
determine an identity of the speaker in the media asset based on metadata associated with the media asset;
identify vocal characteristics of the identified speaker by:
search, based on the identity, for another media asset featuring the speaker;
extract a voice sample featuring the speaker from the another media asset featuring the speaker; and
identify the vocal characteristics based on the voice sample from the another media asset featuring the speaker;
determine non-linguistic characteristics of the plurality of spoken words;
determine an emotional state expressed in the media asset featuring the speaker that utters the plurality of spoken words based on a non-linguistic characteristics; and
generate a translation of the plurality of spoken words of the media asset featuring the speaker that utters the plurality of spoken words using the identified vocal characteristics, and the determined emotional state.