Audio and video translator
A system and method for translating audio, and video when desired. The translations include synthetic media and data generated using AI systems. Through unique processors and generators executing a unique sequence of steps, the system and method produces more accurate translations that can account for various speech characteristics (e.g., emotion, pacing, idioms, sarcasm, jokes, tone, phonemes, etc.). These speech characteristics are identified in the input media and synthetically incorporated into the translated outputs to mirror the characteristics in the input media. Some embodiments further include systems and methods that manipulate the input video such that the speakers' faces and/or lips appear as if they are natively speaking the generated audio.
1. A system configured to generate a synthetic media with translated speech corresponding to an input media file, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the system to:
digitally acquire the input media file, wherein the input media file includes input audio in a first input language;
segment the input audio into one or more vocal segments, wherein each vocal segment includes a speaker identification to identify the speaker of each vocal segment;
for each vocal segment:
acquire an input transcription, wherein the input transcription includes text corresponding to the words spoken in each vocal segment;
acquire input meta information, the meta information including emotion data and tone data, wherein emotion data corresponds to one or more detectable emotions from a list of predetermined emotions;
input the input meta information and input transcription into a transcription and meta translation generator, wherein the transcription and meta translation generator is a generative adversarial network generator; and
translate, by the transcription and meta translation generator, the input transcription and input meta information into a first output language that is distinct from the first input language, wherein translating is based at least on the emotion data.
2. The system of claim 1 , wherein the input media file is in a computer-readable format.
3. The system of claim 1 , wherein the instructions included in the memory further cause the system to partition one vocal stream from another, reduce background noise, or enhance a quality of the vocal streams.
4. The system of claim 1 , wherein the instructions included in the memory further cause the system to capture lip movement tracking data from the input media file.
5. The system of claim 1 , wherein the instructions included in the memory further cause the system to identify pacing information for each word or phoneme in each vocal segment.
6. The system of claim 5 , wherein segmenting the input audio and identifying pacing information is performed by a speaker diarization processor configured to receive the input media file.
7. The system of claim 1 , wherein the text of the input transcription is formatted according to the international phonetics alphabet.
8. The system of claim 1 , wherein the input transcription further includes sentiment analysis and tracking data corresponding to anatomical landmarks for the speaker for each vocal segment.
9. The system of claim 1 , wherein acquiring the input transcription of the input audio includes providing the input audio to an artificial intelligence (AI) generator configured to convert the input audio into text.
10. The system of claim 1 , wherein acquiring input meta information includes providing the input audio and the input transcription to an AI meta information processor configured to identify meta information.
11. The system of claim 1 , wherein the instructions included in the memory further cause the system to provide the translated transcription and meta information to the audio translation generator which is configured to generate translated audio.
12. The system of claim 11 , wherein the instructions included in the memory further cause the system to stitch the translated audio for each vocal segment back into a single audio file.
13. The system of claim 11 , wherein the input media file includes input video and the instructions included in the memory further cause the system to provide translated audio and the input video to a video sync generator and generating, by the video sync generator, a synced video in which the translated audio syncs with the input video.
14. A non-transitory computer-readable medium for generating a synthetic media with translated speech corresponding to an input media file, comprising instructions stored thereon, that when executed on at least one processor, cause the at least one processor to:
acquire the input media file, wherein the input media file includes input video and input audio in a first input language;
segment the input audio into one or more vocal segments, wherein each vocal segment includes a speaker identification to identify the speaker of each vocal segment;
for each vocal segment:
acquire an input transcription, wherein the input transcription includes text corresponding to the words spoken in each vocal segment;
acquire input meta information, the meta information including emotion data and tone data, wherein emotion data corresponds to one or more detectable emotions from a list of predetermined emotions;
translate the input transcription and input meta information into a first output language that is distinct from the first input language; and
generate translated audio by providing the translated input transcription and meta information to an audio translation generator configured to generate the translated audio, wherein the audio translation generator is a generative adversarial network generator.
15. The non-transitory computer-readable medium of claim 14 , wherein the input media file is in a computer-readable format.
16. The non-transitory computer-readable medium of claim 14 , wherein the instructions further cause the at least one processor to capture lip movement tracking data from the input video.
17. The non-transitory computer-readable medium of claim 14 , wherein the input transcription is formatted according to the international phonetics alphabet.
18. The non-transitory computer-readable medium of claim 14 , wherein the instructions further cause the at least one processor to identify pacing information for each word or phoneme in each vocal segment.
19. The non-transitory computer-readable medium of claim 14 , wherein the instructions further cause the at least one processor to stitch the translated audio for each vocal segment back into a single audio file.
20. The non-transitory computer-readable medium of claim 14 , wherein the instructions further cause the at least one processor to:
provide the translated audio and the input video to a video sync generator; and
generate, by the video sync generator, a synced video in which the translated audio syncs with the input video to create the synthetic media.