Speech translation using latency based filler generation

1 . A method, performed at a computer system comprising a processor and a non-transitory computer readable medium, comprising:

receiving, from a first client device over a network, a streaming input audio signal in a first language;

converting the streaming input audio signal to a streaming machine-encoded representation;

dynamically translating the streaming machine-encoded representation of speech information that is in the first language to a machine-encoded representation corresponding to a second language using a translation model that translates a sequence of translatable units of varying lengths in the first language to respective translated units in the second language, wherein at each translation step, the translation model:

determines when a dynamic threshold is reached indicating that sufficient speech information is received to form a translatable unit, and

responsive to reaching the dynamic threshold for the translatable unit, translating the translatable unit to the second language to form a translated unit;

synthesizing and outputting, in an output audio stream provided over a network to a second client device, a most recent translated unit;

responsive to a delay between outputting the most recent translated unit and availability of a forthcoming next translated unit exceeding a threshold time value, selecting filler content that is in the second language to at least partially fill the delay;

synthesizing and outputting to the second client device over the network, the filler content in the output audio stream; and

upon availability of the next translated unit, synthesizing and outputting to the second client device over the network, the next translated unit in the output audio stream.

2 . The method of claim 1 , wherein determining when the dynamic threshold is reached is based in part on a target accuracy for translating the translatable unit and grammar differences between the first language and the second language.

3 . The method of claim 2 , wherein the translation model is trained to dynamically adjust the dynamic threshold to minimize an amount of speech information that forms a translatable unit while maintaining the target accuracy.

4 . The method of claim 1 ,

wherein selecting the filler content comprises:

determining speech characteristics of the speech information, and

selecting the filler content based in part on the speech characteristics.

5 . The method of claim 1 , wherein the streaming input audio signal has corresponding video content, the method further comprising:

synchronizing the output audio stream with corresponding video frames of the video content to form translated video content, and

outputting the translated video content to a client device.

6 . The method of claim 1 , wherein selecting the filler content comprises:

predicting the delay between outputting the most recent translated unit and the availability of the next translated unit, and

applying the predicted delay to a filler word selection model that outputs the filler content.

7 . The method of claim 1 , wherein synthesizing and outputting the most recent translated unit comprises:

adjusting a pronunciation time for the most recent translated unit in the output audio stream based in part on the delay.

8 . A non-transitory computer-readable storage medium comprising stored instructions, the instructions when executed by a processor of a device, cause the device to:

receive, from a first client device over a network, a streaming input audio signal in a first language;

convert the streaming input audio signal into a streaming machine-encoded representation;

dynamically translate the streaming machine-encoded representation of speech information that is in the first language to a machine-encoded representation corresponding to a second language using a translation model that translates a sequence of translatable units of varying lengths in the first language to respective translated units in the second language, wherein at each translation step, the translation model is configured to:

determine when a dynamic threshold is reached indicating that sufficient speech information is received to form a translatable unit, and

responsive to reaching the dynamic threshold for the translatable unit, translate the translatable unit to the second language to form a translated unit;

synthesize from the streaming machine-encoded representation, and output in an output audio stream provided over a network to a second client device, a most recent translated unit;

responsive to a delay between outputting the most recent translated unit and availability of a next translated unit exceeding a threshold time value, select filler content to at least partially fill the delay;

synthesize and output to the second client device over the network, the filler content in the output audio stream; and

upon availability of the next translated unit, synthesize and output to the second client device over the network, the next translated unit in the output audio stream.

9 . The non-transitory computer-readable storage medium of claim 8 , wherein determining when the dynamic threshold is based in part on a target accuracy for translating the translatable unit and grammar differences between the first language and the second language.

10 . The non-transitory computer-readable storage medium of claim 9 , wherein the translation model is trained to dynamically adjust the dynamic threshold to minimize an amount of speech information that forms a translatable unit while maintaining the target accuracy.

11 . The non-transitory computer-readable storage medium of claim 8 , wherein selecting the filler content comprises:

determining speech characteristics of the speech information, and

selecting the filler content based in part on the speech characteristics.

12 . The non-transitory computer-readable storage medium of claim 8 , wherein the streaming input audio signal has corresponding video content, the non-transitory computer-readable storage medium further comprising stored instructions that when executed cause the device to:

synchronize the output audio stream with corresponding video frames of the video content to form translated video content, and

output the translated video content to a client device.

13 . The non-transitory computer-readable storage medium of claim 8 , where the stored instructions to select the filler content further comprise stored instruction that when executed cause the device to:

predict the delay between outputting the most recent translated unit and the availability of the next translated unit, and

apply the predicted delay to a filler word selection model that is configured to output the filler content.

14 . The non-transitory computer-readable storage medium of claim 8 , wherein the stored instructions to synthesize and output the most recent translated unit further comprise stored instructions that when executed cause the device to:

adjust a pronunciation time for the most recent translated unit in the output audio stream based in part on the delay.

15 . A computer system comprising:

a processor; and

a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the processor, cause the processor to:

receive a streaming input audio signal in a first language;

convert the streaming input audio signal to a streaming machine-encoded representation;

dynamically translate the streaming machine-encoded representation of speech information that is in the first language to a machine-encoded representation corresponding to a second language using a translation model that translates a sequence of translatable units of varying lengths in the first language to respective translated units in the second language, wherein at each translation step, the translation model is configured to:

determine when a dynamic threshold is reached indicating that sufficient speech information is received to form a translatable unit, and

responsive to reaching the dynamic threshold for the translatable unit, translate the translatable unit to the second language to form a translated unit;

synthesize and output, in an output audio stream provided over a network to a second client device, a most recent translated unit;

responsive to a delay between outputting the most recent translated unit and availability of a next translated unit exceeding a threshold time value, select filler content to at least partially fill the delay;

synthesize and output to the second client device over the network, the filler content in the output audio stream; and

upon availability of the next translated unit, synthesize and output to the second client device over the network, the next translated unit in the output audio stream.

16 . The computer system of claim 15 , wherein the translation model is trained to dynamically adjust the dynamic threshold to minimize an amount of speech information that forms a translatable unit while maintaining a target accuracy.

17 . The computer system of claim 15 , wherein selecting the filler content comprises:

determining speech characteristics of the speech information, and

selecting the filler content based in part on the speech characteristics.