Streaming vocoder
A method includes receiving a current spectrogram frame and reconstructing a phase of the current spectrogram frame by, for each corresponding committed spectrogram frame in a sequence of M number of committed spectrogram frames preceding the current spectrogram frame, obtaining a value of a committed phase of the corresponding committed spectrogram frame and estimating the phase of the current spectrogram frame based on a magnitude of the current spectrogram frame and the value of the committed phase of each corresponding committed spectrogram frame in the sequence of M number of committed spectrogram frames preceding the current spectrogram frame. The method also includes synthesizing, for the current spectrogram frame, a new time-domain audio waveform frame based on the estimated phase of the current spectrogram frame.
1 . A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
receiving a current spectrogram frame;
reconstructing a phase of the current spectrogram frame by:
for each corresponding committed spectrogram frame in a sequence of M number of committed spectrogram frames preceding the current spectrogram frame, obtaining a value of a committed phase of the corresponding committed spectrogram frame; and
estimating the phase of the current spectrogram frame by performing one or more iterations within a sliding window that contains the current spectrogram frame, wherein performing each iteration of the one or more iterations within the sliding window comprises:
estimating an uncommitted phase of the current spectrogram frame based on a sequence of N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame; and
updating a complex-valued spectrogram representation within the sliding window by combining the value of the committed phase of each corresponding committed spectrogram frame in the sequence of M number of committed spectrogram frames preceding the current spectrogram frame, the estimated uncommitted phase, and a magnitude of the current spectrogram frame; and
for the current spectrogram frame, synthesizing a new time-domain audio waveform frame based on the estimated phase of the current spectrogram frame.
2 . The method of claim 1 , wherein:
the current spectrogram frame comprises a log-magnitude spectrogram frame output from a speech conversion model; and
prior to reconstructing the phase of the current spectrogram frame, the phase of the current spectrogram frame is initialized with a value equal to zero.
3 . The method of claim 1 , wherein the M number of committed spectrogram frames preceding the current spectrogram frame is equal to one.
4 . The method of claim 1 , wherein the M number of committed spectrogram frames preceding the current spectrogram frame is at least two.
5 . The method of claim 1 , wherein estimating the uncommitted phase of the current spectrogram frame based on the N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame comprises:
for each corresponding uncommitted spectrogram frame in the sequence of N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame, obtaining a value of an uncommitted phase of the corresponding uncommitted spectrogram frame; and
estimating the uncommitted phase of the current spectrogram frame is based on the value of the uncommitted phase of each corresponding uncommitted spectrogram frame in the sequence of N number of committed spectrogram frames within the sliding window that are subsequent to the current spectrogram frame.
6 . The method of claim 1 , wherein the N number of uncommitted spectrogram frames and the M number of committed spectrogram frames are equal.
7 . The method of claim 1 , wherein the N number of uncommitted spectrogram frames and the M number of committed spectrogram frames are different.
8 . The method of claim 1 , wherein the N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame is equal to one.
9 . The method of claim 1 , wherein the N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame is at least two.
10 . The method of claim 1 , wherein the current spectrogram frame is in a Short-time Fourier transform (STFT) domain when reconstructing the phase of the current spectrogram frame.
11 . The method of claim 10 , wherein synthesizing the new time-domain audio waveform frame based on the estimated phase of the current spectrogram frame comprises running a streaming inverse STFT on an output frame corresponding to the current spectrogram frame, the output frame extracted using the estimated phase of the current spectrogram frame.
12 . The method of claim 1 , wherein the operations further comprise, after reconstructing the phase of the current spectrogram frame, designating the current spectrogram frame as a committed frame and storing the estimated phase of the current spectrogram frame as a committed phase.
13 . The method of claim 1 , wherein the data processing hardware resides on a user computing device or a server.
14 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a current spectrogram frame;
reconstructing a phase of the current spectrogram frame by:
for each corresponding committed spectrogram frame in a sequence of M number of committed spectrogram frames preceding the current spectrogram frame, obtaining a value of a committed phase of the corresponding committed spectrogram frame; and
estimating the phase of the current spectrogram frame by performing one or more iterations within a sliding window that contains the current spectrogram frame, wherein performing each iteration of the one or more iterations within the sliding window comprises:
estimating an uncommitted phase of the current spectrogram frame based on a sequence of N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame; and
updating a complex-valued spectrogram representation within the sliding window by combining the value of the committed phase of each corresponding committed spectrogram frame in the sequence of M number of committed spectrogram frames preceding the current spectrogram frame, the estimated uncommitted phase, and a magnitude of the current spectrogram frame; and
for the current spectrogram frame, synthesizing a new time-domain audio waveform frame based on the estimated phase of the current spectrogram frame.
15 . The system of claim 14 , wherein:
the current spectrogram frame comprises a log-magnitude spectrogram frame output from a speech conversion model; and
prior to reconstructing the phase of the current spectrogram frame, the phase of the current spectrogram frame is initialized with a value equal to zero.
16 . The system of claim 14 , wherein the M number of committed spectrogram frames preceding the current spectrogram frame is equal to one.
17 . The system of claim 14 , wherein the M number of committed spectrogram frames preceding the current spectrogram frame is at least two.
18 . The system of claim 14 , wherein estimating the uncommitted phase of the current spectrogram frame based on the N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame comprises:
for each corresponding uncommitted spectrogram frame in the sequence of N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame, obtaining a value of an uncommitted phase of the corresponding uncommitted spectrogram frame; and
estimating the uncommitted phase of the current spectrogram frame is further-based on the value of the uncommitted phase of each corresponding uncommitted spectrogram frame in the sequence of N number of committed spectrogram frames within the sliding window that are subsequent to the current spectrogram frame.
19 . The system of claim 14 , wherein the N number of uncommitted spectrogram frames and the M number of committed spectrogram frames are equal.
20 . The system of claim 14 , wherein the N number of uncommitted spectrogram frames and the M number of committed spectrogram frames are different.
21 . The system of claim 14 , wherein the N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame is equal to one.
22 . The system of claim 14 , wherein the N number of uncommitted spectrogram frames within the sliding window that are subsequent to the current spectrogram frame is at least two.
23 . The system of claim 14 , wherein the current spectrogram frame is in a Short-time Fourier transform (STFT) domain when reconstructing the phase of the current spectrogram frame.
24 . The system of claim 23 , wherein synthesizing the new time-domain audio waveform frame based on the estimated phase of the current spectrogram frame comprises running a streaming inverse STFT on an output frame corresponding to the current spectrogram frame, the output frame extracted using the estimated phase of the current spectrogram frame.
25 . The system of claim 14 , wherein the operations further comprise, after reconstructing the phase of the current spectrogram frame, designating the current spectrogram frame as a committed frame and storing the estimated phase of the current spectrogram frame as a committed phase.
26 . The system of claim 14 , wherein the data processing hardware resides on a user computing device or a server.