Method and server for a text-to-speech processing
Methods and servers for processing a textual input for generating an audio output are disclosed. The audio output is a sequence of waveform segments generated in real-time by a trained Convolutional Neural Network. The method includes, at a given iteration, generating a given waveform segment which includes storing first tensor data computed by a first hidden layer during the given iteration, and where the first tensor data has tensor-chunk data. The tensor-chunk data is used during the given iteration for generating the given waveform segment and is to be used during a next iteration for generating a next waveform segment. The method includes, at the next iteration, generating the next waveform segment, which comprises storing second tensor data computed by the first hidden layer during the next iteration. The second tensor data excludes redundant tensor-chunk data that is identical to the tensor-chunk data from the first tensor data.
1 . A method of performing text-to-speech (TTS) processing of a textual input for generating an audio output, the audio output to be provided to a user of an electronic device and comprising a plurality of sequential waveform segments, the electronic device being communicatively coupled to a server,
a given one of the plurality of sequential waveform segments to be generated in real-time during a respective in-use iteration of a trained Convolutional Neural Network (CNN),
the CNN comprising at least a first hidden layer and a second hidden layer, the second hidden layer being sequential to the first hidden layer, the second hidden layer being configured to generate tensor data based on tensor data generated by the first hidden layer during a respective in-use iteration of the CNN,
the method executable by the server, the method comprising:
generating, by the server, CNN input data based on the textual input,
the CNN input data being associated with respective timestamps, the timestamps to be used for determining iteration-specific input data for a respective in-use iteration of the CNN;
at a given in-use iteration of the CNN:
generating, by the server employing the CNN, a given waveform segment of the audio output based on first iteration-specific data available for the given in-use iteration, the generating comprising:
storing, in a memory, first tensor data computed by the first hidden layer during the given in-use iteration, the first tensor data having tensor-chunk data,
the tensor-chunk data (i) being used during the given in-use iteration for generating the given waveform segment of the audio output and (ii) to be used during a next in-use iteration of the CNN for generating a next waveform segment of the audio output,
the tensor-chunk data being identifiable by a start-position of the tensor-chunk data in the memory and an end-position of the tensor-chunk data in the memory;
at the next in-use iteration of the CNN:
generating, by the server employing the CNN, the next waveform segment of the audio output based on second iteration-specific data available for the next in-use iteration and a portion of the first iteration-specific data, the generating comprising:
storing, in the memory, second tensor data computed by the first hidden layer during the next in-use iteration,
the second tensor data being stored in the memory sequentially to the first tensor data of the first hidden layer from the given in-use iteration,
the second tensor data excluding a redundant tensor-chunk data, the redundant tensor-chunk data being identical to the tensor-chunk data from the first tensor data,
the second tensor data being identifiable by a start-position of the second tensor data in the memory and an end-position of the second tensor data in the memory,
the start-position of the second tensor data corresponding to the end-position of the tensor- chunk data in the memory;
applying, by the server, a convolution filter of the second hidden layer on (i) the tensor-chunk data from the first tensor data and (ii) the second tensor data stored in the memory,
thereby generating third tensor data by the second hidden layer to be stored in the memory, the third tensor data being used for generating the next waveform segment of the audio output; and
storing, by the server, the third tensor data in the memory,
wherein the memory comprises a layer-dedicated memory space for a respective hidden layer of the CNN,
the layer-dedicated memory space being defined by boundaries in the memory,
a first boundary being indicative of a start-position of the respective layer-dedicated memory space in the memory and a second boundary being indicative of an end- position of the respective layer-dedicated memory space in the memory,
the layer-dedicated memory space for a respective hidden layer being configured to store tensor data generated by the respective hidden layer during in-use iterations of the CNN,
wherein:
a pair of flags associated with a first layer-dedicated memory space for the first hidden layer are used when storing data generated by the first hidden layer,
the pair of flags being within the boundaries of the first layer-dedicated memory space,
a pair of other flags associated with a second layer-dedicated memory space for the second hidden layer are used for storing data generated by the second hidden layer,
the pair of other flags being within the boundaries of the second layer-dedicated memory space, and
wherein the pair of flags comprises a first flag and a second flag and the pair of other flags comprise a third flag and a fourth flag,
after the given in-use iteration:
the first flag being located a first position corresponding to the start-position of the tensor- chunk data in the memory and the second flag being located at a second position corresponding to the end-position of the tensor-chunk data in the memory,
the third flag being located at a third position in the second layer-dedicated memory space and the fourth flag being located at a fourth position in the second layer-dedicated memory space,
during the next in-use iteration:
the storing the second tensor data computed by the first hidden layer comprises:
moving the second flag to a new second position in the memory, the second tensor data being stored in the memory between the second position and the new second position,
the tensor-chunk data and the second tensor data being stored sequentially in the memory between the first position of the first flag corresponding to the start-position of the tensor-chunk data and the new second position of the second flag,
the applying the convolutional filter on (i) the tensor-chunk data from the first tensor data and (ii) the second tensor data comprises:
applying the convolutional filter on data stored in the memory between the first position of the first flag and the new second position of the second flag,
the storing the third tensor data in the memory comprises:
moving the fourth flag to a new fourth position in the memory, the third tensor data being stored in the memory between the fourth position and the new fourth position.
2 . The method of claim 1 , wherein the CNN input data is in a form of a spectrogram.
3 . The method of claim 2 , wherein the CNN input data is generated by an other trained model, the other trained model configured to transform textual input into a spectrogram output.
4 . The method of claim 1 , wherein the memory is from a GPU-type memory device.
5 . The method of claim 1 , wherein the first hidden layer has an activation function, the tensor-chunk data from the first tensor data generated during the given in-use iteration being an output of the activation function.
6 . The method of claim 1 , wherein a size of the first tensor data in the memory depends on an amount of iteration-specific data available for the given in-use iteration.
7 . The method of claim 1 , wherein a size of the second tensor data in the memory depends on an amount of the second iteration-specific data available for the next in-use iteration.
8 . The method of claim 1 , wherein a size of the tensor-chunk data in the memory has been determined based on hyper parameters of the convolutional filter of the second hidden layer.
9 . The method of claim 8 , wherein the hyper parameters include at least some of: a kernel size parameter, a stride parameter, and a dilation parameter.