Translation training with cross-lingual multi-media support
An improved lecture support system integrates multi-media presentation materials with spoken content so that the listener can follow with both the speech and the supporting materials that accompany the presentation to provide additional understanding. Computer-based systems and methods are disclosed for translation of a spoken presentation (e.g., a lecture, a video) along with the accompanying presentation materials. The content of the presentation materials can be used to improve presentation translation, as it extracts supportive material from the presentation materials as they relate to the speech.
1. A computer-implemented method comprising:
receiving an original presentation material in a first language;
identifying an unknown word in the original presentation materials, the unknown word being a word that a translator is insufficiently trained to recognize;
receiving a speech in the first language, the speech presenting the original presentation materials;
synchronizing the speech and the original presentation material;
using the speech that includes a pronunciation of the unknown word to further train the translator; and
translating the original presentation material to a second language using the further trained translator.
2. The computer-implemented method of claim 1 , further comprising:
generating a transcript of the speech; and
translating the transcript of the speech into the second language.
3. The computer-implemented method of claim 2 , wherein the transcript is generated based on both the speech and the original presentation material.
4. The computer-implemented method of claim 2 , wherein generating the transcript of the speech comprises:
extracting at least one formatting feature from the original presentation materials, and
modifying the transcript by importing the at least one formatting feature extracted from the original presentation materials.
5. The computer-implemented method of claim 1 , wherein the unknown word is part of a named entity.
6. The computer-implemented method of claim 1 , wherein the unknown word is a word that is new to a dictionary associated with the translator.
7. The computer-implemented method of claim 1 , wherein synchronizing the speech and the original presentation material comprises:
generating a first marker that corresponds to a specific portion of the speech; and
identifying a corresponding portion in the original presentation material, the corresponding portion corresponds to the first marker.
8. The computer-implemented method of claim 7 , wherein the first marker is a timestamp that corresponds to a time the speech transition from a first presentation slide to a second presentation slide.
9. The computer-implemented method of claim 1 , using the speech that includes the pronunciation of the unknown word to further train the translator comprises:
performing a search of the unknown word to identify similar materials that have text that includes the unknown word; and
generating a language model probability for the unknown word in the text.
10. The computer-implemented method of claim 1 , wherein the translator comprises an automatic speech recognition model and a machine translation model.
11. A non-transitory computer readable medium configured to store computer code comprising instructions, the instructions, when executed by one or more processors, cause the one or more processors to perform steps comprising:
receiving an original presentation material in a first language;
identifying an unknown word in the original presentation materials, the unknown word being a word that a translator is insufficiently trained to recognize;
receiving a speech in the first language, the speech presenting the original presentation materials;
synchronizing the speech and the original presentation material;
using the speech that includes a pronunciation of the unknown word to further train the translator; and
translating the original presentation material to a second language using the further trained translator.
12. The non-transitory computer readable medium of claim 11 , wherein the steps further comprise:
generating a transcript of the speech; and
translating the transcript of the speech into the second language.
13. The non-transitory computer readable medium of claim 12 , wherein the transcript is generated based on both the speech and the original presentation material.
14. The non-transitory computer readable medium of claim 12 , wherein generating the transcript of the speech comprises:
extracting at least one formatting feature from the original presentation materials, and
modifying the transcript by importing the at least one formatting feature extracted from the original presentation materials.
15. The non-transitory computer readable medium of claim 11 , wherein the unknown word is part of a named entity.
16. The non-transitory computer readable medium of claim 11 , wherein the unknown word is a word that is new to a dictionary associated with the translator.
17. The non-transitory computer readable medium of claim 11 , wherein synchronizing the speech and the original presentation material comprises:
generating a first marker that corresponds to a specific portion of the speech; and
identifying a corresponding portion in the original presentation material, the corresponding portion corresponds to the first marker.
18. The non-transitory computer readable medium of claim 17 , wherein the first marker is a timestamp that corresponds to a time the speech transition from a first presentation slide to a second presentation slide.
19. The non-transitory computer readable medium of claim 11 , using the speech that includes the pronunciation of the unknown word to further train the translator comprises:
performing a search of the unknown word to identify similar materials that have text that includes the unknown word; and
generating a language model probability for the unknown word in the text.
20. The non-transitory computer readable medium of claim 11 , wherein the translator comprises an automatic speech recognition model and a machine translation model.