Real-time automatic online voice translation system and method for telephone conversations

1 . An automatic online voice translation system, comprising:

a virtual translator;

a Session Initiation Protocol (SIP) gateway; and

a storage;

wherein the SIP gateway receives a call from a caller through a SIP trunk connection;

wherein the SIP gateway streams the caller's voice directly to an agent and streams the agent's voice directly to the caller when the virtual translator is not activated, without interruption of the voice stream;

wherein the SIP gateway continuously collects and stores, in the storage, a conversation segment of the caller prior to activation of the virtual translator;

wherein the virtual translator is selectively activated by the agent via clicking on a button or dialing a pre-determined number of combinations;

wherein, upon activation, the SIP gateway creates audio files based on (i) received voice streams of both the caller and the agent and the stored conversation segment collected prior to activation, and sends the created audio files to the virtual translator for real-time speech recognition, language detection, text translation, and text-to-speech processing.

2 . The automatic online voice translation system according to claim 1 , wherein the virtual translator further comprises:

a voice activity detection component, wherein the voice activity detection component detects timings in the audio files in which there is a person speaking, allowing the virtual translator to process only corresponding portions of the audio files;

a speech recognition and language detection component, wherein the speech recognition and language detection component transcribes spoken words in the audio files into text in real time;

a text translation component, wherein the text translation component translates the transcribed text in caller's language into text in agent's language, and translates the transcribed text in the agent's language into text in caller's language; and

a text-to-speech engine, wherein the text-to-speech engine synthesizes the translated text into a translated audible speech in real-time.

3 . The automatic online voice translation system according to claim 2 , wherein the virtual translator further captures emotional tones or sentiments from the caller's voice; the emotional tones or sentiments from the caller's voice and the translated transcribed text from the text translation component are transmitted directly from the virtual translator to the agent, and the translated audible speech from the text-to-speech engine is routed from the virtual translator to the agent through the SIP gateway, enabling the agent to simultaneously listen to the translated audible speech and view the emotional tones or sentiments from the caller's voice and the translated transcribed text.

4 . The automatic online voice translation system according to claim 1 , wherein the SIP gateway creates the audio files based on the received voice streams of both the caller and the agent and the speech of the caller stored in the storage.

5 . The automatic online voice translation system according to claim 1 , wherein a real-time connection of one of WebSocket or high performance remote procedure call (gRPC) for bidirectional communication is established between the SIP gateway and the virtual translator.

6 . The automatic online voice translation system according to claim 1 , wherein the virtual translator is activated by the agent via the agent clicking on a button or dialing a pre-determined number of combinations.

7 . An automatic online voice translation system, comprising:

a virtual translator;

a contact center platform;

a Session Initiation Protocol (SIP) gateway; and

a storage;

wherein the contact center platform receives a call from a caller and processes the call based on at least one of call recording, Interactive Voice Response (IVR) service, or conferencing service;

wherein the contact center platform routes the call to the SIP gateway for connection with an agent;

wherein the SIP gateway streams the caller's voice and the agent's voice, in real-time through the contact center platform when the virtual translator is not activated;

wherein the SIP gateway continuously collects and stores in the storage, a conversation segment of the caller before activation of the virtual translator;

wherein the virtual translator is activated by the agent via clicking on a button or dialing a pre-determined number of combinations:

wherein, upon activation, the SIP gateway creates audio files including both (i) real-time voice streams and ii) the stored conversation segment collected before activation, and transmits the audio files to the virtual translator for real-time speech recognition, language detection, text translation, and text-to-speech processing.

8 . The automatic online voice translation system according to claim 7 , wherein the virtual translator further comprises:

a voice activity detection component, wherein the voice activity detection component detects timings in the audio files in which there is a person speaking, allowing the virtual translator to process only corresponding portions of the audio files;

a speech recognition and language detection component, wherein the speech recognition and language detection component transcribes spoken words in the audio files into text in real time;

a text translation component, wherein the text translation component translates the transcribed text in caller's language into text in agent's language, and translates the transcribed text in the agent's language into text in caller's language; and

a text-to-speech engine, wherein the text-to-speech engine synthesizes the translated text into a translated audible speech in real-time.

9 . The automatic online voice translation system according to claim 8 , wherein the virtual translator further captures emotional tones or sentiments from the caller's voice; the emotional tones or sentiments from the caller's voice and the translated transcribed text from the text translation component are transmitted directly from the virtual translator to the agent, and the translated audible speech from the text-to-speech engine is routed from the virtual translator to the agent through the SIP gateway, enabling the agent to simultaneously listen to the translated audible speech and view the emotional tones or sentiments from the caller's voice and the translated transcribed text.

10 . The automatic online voice translation system according to claim 7 , wherein the SIP gateway creates the audio files based on the received voice streams of both the caller and the agent and the speech of the caller stored in the storage.

11 . The automatic online voice translation system according to claim 7 , wherein a real-time connection of one of WebSocket or high performance remote procedure call (gRPC) for bidirectional communication is established between the SIP gateway and the virtual translator.

12 . The automatic online voice translation system according to claim 7 , wherein the virtual translator is activated by the agent via the agent clicking on a button or dialing a pre-determined number of combinations.

13 . An automatic online voice translation method, comprising:

receiving, by a Session Initiation Protocol (SIP) gateway, a voice stream from a caller;

streaming the voice stream directly to an agent while a virtual translator is not activated;

continuously collecting and storing, in a storage, a conversation segment of the caller prior to activation of the virtual translator;

activating the virtual translator by the agent via clicking on a button or dialing a pre-determined number of combinations;

creating, by the SIP gateway, audio files including both the stored conversation segment and real-time voice streams;

transmitting the audio files to the virtual translator;

performing speech;

recognition, language

detection, text translation, and text-to-speech synthesis in real time; and

transmitting translated audible speech to the agent.

14 . The automatic online voice translation method according to claim 13 , further comprising:

receiving the spoken languages from a caller seeking service from an agent; and

transmitting the second speech to the agent.

15 . The automatic online voice translation method according to claim 14 , further comprising:

activating the virtual translator the agent via the agent clicking on a button or dialing a pre-determined number of combinations.

16 . The automatic online voice translation method according to claim 14 , further comprising:

receiving, by the virtual translator, response voice streams from the agent;

converting the response voice streams from the agent to a second text;

translating the second text into a second translated text to a caller's language;

converting the second translated text to a third speech; and

transmitting the third speech to the caller.

17 . The automatic online voice translation method according to claim 14 , further comprising:

capturing, by the virtual translator, emotional tones or sentiments from the caller's voice;

transmitting, by the virtual translator, an indicator indicating the emotional tones or sentiments from the caller's voice to the agent.

18 . The automatic online voice translation method according to claim 14 , further comprising:

receiving, by the agent, the indicator indicating the emotional tones or sentiments and the second speech simultaneously.

19 . The automatic online voice translation method according to claim 14 , wherein the transmitting of the second speech to the agent further comprising:

transmitting, by the virtual translator, the second speech to the SIP gateway; and

transmitting, by the SIP gateway, the second speech to the agent.

20 . The automatic online voice translation method according to claim 14 , wherein the transmitting of the second speech to the agent further comprising:

transmitting, by the virtual translator, the second speech to the SIP gateway;

transmitting, by the SIP gateway, the second speech to a contact center platform; and

transmitting, by the contact center platform, the second speech to the agent.