IP Library › Granted Patent US 12,190,896
Granted Patent B2
US 12,190,896 · App. 17/856,292 · Granted Jan 7, 2025

Generating audio waveforms using encoder and decoder neural networks

Inventors: Yunpeng Li (Zurich, CH); Marco Tagliasacchi (Kilchberg, CH); Dominik Roblek (Meilen, CH); Félix de Chaumont Quitry (Zurich, CH); Beat Gfeller (Dubendorf, CH); Hannah Raphaelle Muckenhirn (Zurich, CH); Victor Ungureanu (Thalwil, CH); Oleg Rybakov (Redmond, WA); Karolis Misiunas (Zurich, CH); Zalán Borsos (Zurich, CH)
Assignee: Google LLC
G10L19/022G06N3/045
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,190,896
App. No.
17/856,292
Granted
Jan 7, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing an input audio waveform using a generator neural network to generate an output audio waveform. In one aspect, a method comprises: receiving an input audio waveform; processing the input audio waveform using an encoder neural network to generate a set of feature vectors representing the input audio waveform; and processing the set of feature vectors representing the input audio waveform using a decoder neural network to generate an output audio waveform that comprises a respective output audio sample for each of a plurality of output time steps.

Claims (92)

1. A method performed by one or more computers, the method comprising:

receiving an input audio waveform that comprises a respective input audio sample for each of a plurality of input time steps;

processing the input audio waveform using an encoder neural network to generate a set of feature vectors representing the input audio waveform,

wherein the encoder neural network comprises a sequence of encoder blocks that are each configured to:

process a respective set of input feature vectors in accordance with a set of encoder block parameters to generate a set of output feature vectors, comprising down-sampling the set of input feature vectors; and

processing the set of feature vectors representing the input audio waveform using a decoder neural network to generate an output audio waveform that comprises a respective output audio sample for each of a plurality of output time steps,

wherein the decoder neural network comprises a sequence of decoder blocks that are each configured to:

process a respective set of input feature vectors in accordance with a set of decoder block parameters to generate a set of output feature vectors, comprising up-sampling the set of input feature vectors;

wherein the output audio waveform represents a version of the input audio waveform that has been filtered to include only audio from a target audio source; and

wherein the encoder neural network, the decoder neural network, or both additionally process a conditioning vector representing the target audio source.

2. The method of claim 1 , wherein each encoder block in the sequence of encoder blocks down-samples the set of input feature vectors to the encoder block using a respective strided convolution operation.

3. The method of claim 2 , wherein for each encoder block in the sequence of encoder blocks, the strided convolution operation is a one-dimensional or two-dimensional strided convolution operation.

4. The method of claim 1 , wherein for each encoder block in the sequence of encoder blocks, a dimensionality of the output feature vectors generated by the encoder block is higher than a dimensionality of the input feature vectors processed by the encoder block.

5. The method of claim 1 , wherein each decoder block in the sequence of decoder blocks up-samples the set of input feature vectors to the decoder block using a respective strided transposed convolution operation.

6. The method of claim 5 , wherein for each decoder block in the sequence of decoder blocks, the strided transposed convolution operation is a one-dimensional or two-dimensional strided transposed convolution operation.

7. The method of claim 1 , wherein for each decoder block in the sequence of decoder blocks, a dimensionality of the output feature vectors generated by the decoder block is lower than a dimensionality of the input feature vectors processed by the decoder block.

8. The method of claim 1 , wherein for each encoder block that is after a first encoder block in the sequence of encoder blocks, the set of input feature vectors to the encoder block comprises a set of output feature vectors generated by a preceding encoder block in the sequence of encoder blocks.

9. The method of claim 1 , wherein for each decoder block that is after a first decoder block in the sequence of decoder blocks, the set of input feature vectors to the decoder block comprises: (i) a set of output feature vectors of a corresponding encoder block, and (ii) a set of output feature vectors generated by a preceding decoder block in the sequence of decoder blocks.

10. The method of claim 1 , wherein the encoder neural network comprises a transform layer prior to the sequence of encoder blocks, wherein the transform layer maps the input audio waveform to an alternative representation in an alternative domain.

11. The method of claim 10 , wherein the transform layer maps the input audio waveform to an alternative representation in a time-frequency domain.

12. The method of claim 11 , wherein the transform layer implements a Fourier transform operation.

13. The method of claim 10 , wherein the decoder neural network comprises an inverse transform layer after the sequence of decoder blocks, wherein the inverse transform layer maps a representation of the output audio waveform in the alternative domain to a representation of the audio waveform in a time domain.

14. The method of claim 13 , wherein the inverse transform layer implements an inverse Fourier transform operation.

15. The method of claim 1 , wherein the encoder neural network and the decoder neural network are jointly trained, and the training comprises:

obtaining a plurality of training examples that each include: (i) a respective input audio waveform, and (ii) a corresponding target audio waveform;

processing the respective input audio waveform from each training example using the encoder neural network followed by the decoder neural network to generate an output audio waveform that is an estimate of the corresponding target audio waveform;

determining gradients of an objective function that depends on the respective output waveform and respective target waveform for each training example; and

using the gradients of the objective function to update a set of encoder neural network parameters and a set of decoder neural network parameters.

16. The method of claim 15 , wherein the training further comprises, for each training example:

processing data derived from the output audio waveform using a discriminator neural network to generate a set of one or more discriminator scores, wherein each discriminator score characterizes an estimated likelihood that the output audio waveform is an audio waveform that was generated using the encoder neural network and the decoder neural network;

wherein the objective function comprises an adversarial loss that depends on the discriminator scores generated by the discriminator neural network.

17. The method of claim 16 , wherein the data derived from the output audio waveform comprises the output audio waveform, a down-sampled version of the output audio waveform, or a Fourier-transformed version of the output audio waveform.

18. The method of claim 16 , wherein the training further comprises, for each training example:

generating a respective set of discriminator scores using each of a plurality of discriminator neural networks, wherein each discriminator neural network processes a respective version of the output audio waveform that has been down-sampled by a respective factor;

wherein the adversarial loss depends on the discriminator scores generated by the plurality of discriminator neural networks.

19. The method of claim 16 , wherein the discriminator neural network is trained to generate discriminator scores that distinguish between: (i) output audio waveforms generated using the encoder neural network and the decoder neural network, and (ii) target audio waveforms from training examples.

20. The method of claim 16 , wherein the discriminator neural network is a convolutional neural network, and wherein a number of discriminator scores in the set of discriminator scores generated by the discriminator neural network is proportional to a length of the output audio waveform.

21. The method of claim 16 , wherein the objective function comprises a reconstruction loss that, for each training example, measures an error between: (i) the output audio waveform, and (ii) the corresponding target audio waveform.

22. The method of claim 21 , wherein for each training example, the reconstruction loss measures a discrepancy between: (i) one or more intermediate output generated by the discriminator neural network by processing the output audio waveform, and (ii) one or more intermediate outputs generated by the discriminator neural network by processing the corresponding target audio waveform.

23. The method of claim 21 , wherein the objective function comprises a linear combination of the adversarial loss and the reconstruction loss.

24. The method of claim 1 , wherein each encoder block and each decoder block comprise one or more respective dilated convolutional neural network layers.

25. The method of claim 1 , wherein each encoder block and each decoder block comprise one or more respective residual blocks.

26. The method of claim 1 , wherein the output audio waveform is an enhanced version of the input audio waveform.

27. The method of claim 26 , wherein the output audio waveform is a de-noised version of the input audio waveform.

28. The method of claim 27 , wherein the encoder neural network processes both: (i) the input audio waveform, and (ii) corresponding accelerometer data.

29. The method of claim 28 , wherein the accelerometer data comprises an accelerometer waveform that is captured synchronously with the input audio waveform.

30. The method of claim 29 , wherein the accelerometer data comprises an accelerometer waveform captured by a bone conductance accelerometer mounted on a headphone.

31. The method of claim 26 , wherein the output audio waveform represents a version of the input audio waveform captured at a higher sampling frequency than the input audio waveform.

32. The method of claim 31 , wherein the encoder neural network and the decoder neural network are trained on input audio waveforms having a range of sampling frequencies.

33. The method of claim 31 , wherein the input audio waveform is generated as an output of a text-to-speech model that processes an input sequence of text to generate an audio waveform that is a verbalization of the input sequence of text.

34. The method of claim 1 , wherein the input audio waveform is captured using a target microphone, and the output audio waveform represents a version of the input audio waveform captured using a source microphone.

35. The method of claim 34 , further comprising using the input audio waveform and the corresponding output audio waveform to optimize parameters of a target microphone model that defines a mapping from audio waveforms captured using the source microphone to audio waveforms captured using the target microphone.

36. The method of claim 35 , further comprising:

processing a plurality of source audio waveforms corresponding to the source microphone using the optimized target microphone model to generate a plurality of target audio waveforms corresponding to the target microphone; and

training a machine learning model to perform an audio processing task using the target audio waveforms corresponding to the target microphone.

37. The method of claim 1 , wherein the input audio waveform is a speech waveform corresponding to a verbalization by a person, and wherein the output audio waveform represents an anonymized version of the verbalization that would be generated by a text-to-speech model.

38. The method of claim 1 , wherein the target audio source corresponds to a particular speaker.

39. The method of claim 1 , wherein the target audio source corresponds to a particular musical instrument.

40. The method of claim 1 , wherein the conditioning vector representing the target audio source is generated based on a sample audio waveform from the target audio source, wherein generating the conditioning vector comprises:

processing the sample audio waveform from the target audio source using a conditioning neural network to generate a plurality of feature vectors representing the sample audio waveform; and

generating the conditioning vector using the plurality of feature vectors representing the sample audio waveform.

41. The method of claim 40 , wherein the conditioning neural network comprises a sequence of encoder blocks that are each configured to:

process a respective set of input feature vectors in accordance with a set of encoder block parameters to generate a set of output feature vectors, comprising down-sampling the set of input feature vectors.

42. The method of claim 40 , wherein generating the conditioning vector using the plurality of feature vectors representing the sample audio waveform comprises:

pooling the plurality of feature vectors representing the sample audio waveform to generate a pooled feature vector;

determining a respective score for each of the plurality of feature vectors representing the sample audio waveform based on a similarity between: (i) the pooled feature vector, and (ii) the feature vector representing the sample audio waveform; and

generating the conditioning vector by combining the plurality of feature vectors representing the sample audio waveform using the scores.

43. The method of claim 1 , wherein the input audio waveform is a speech or music waveform.

44. The method of claim 1 , wherein the encoder neural network and the decoder neural network are implemented on a mobile device.

45. The method of claim 1 , wherein the encoder neural network and the decoder neural network are implemented on a personal digital assistant device.

46. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving an input audio waveform that comprises a respective input audio sample for each of a plurality of input time steps;

processing the input audio waveform using an encoder neural network to generate a set of feature vectors representing the input audio waveform,

wherein the encoder neural network comprises a sequence of encoder blocks that are each configured to:

process a respective set of input feature vectors in accordance with a set of encoder block parameters to generate a set of output feature vectors, comprising down-sampling the set of input feature vectors; and

processing the set of feature vectors representing the input audio waveform using a decoder neural network to generate an output audio waveform that comprises a respective output audio sample for each of a plurality of output time steps,

wherein the decoder neural network comprises a sequence of decoder blocks that are each configured to:

process a respective set of input feature vectors in accordance with a set of decoder block parameters to generate a set of output feature vectors, comprising up-sampling the set of input feature vectors;

wherein the output audio waveform represents a version of the input audio waveform that has been filtered to include only audio from a target audio source; and

wherein the encoder neural network, the decoder neural network, or both additionally process a conditioning vector representing the target audio source.

47. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving an input audio waveform that comprises a respective input audio sample for each of a plurality of input time steps;

processing the input audio waveform using an encoder neural network to generate a set of feature vectors representing the input audio waveform,

wherein the encoder neural network comprises a sequence of encoder blocks that are each configured to:

process a respective set of input feature vectors in accordance with a set of encoder block parameters to generate a set of output feature vectors, comprising down-sampling the set of input feature vectors; and

processing the set of feature vectors representing the input audio waveform using a decoder neural network to generate an output audio waveform that comprises a respective output audio sample for each of a plurality of output time steps,

wherein the decoder neural network comprises a sequence of decoder blocks that are each configured to:

process a respective set of input feature vectors in accordance with a set of decoder block parameters to generate a set of output feature vectors, comprising up-sampling the set of input feature vectors;

wherein the output audio waveform represents a version of the input audio waveform that has been filtered to include only audio from a target audio source; and

wherein the encoder neural network, the decoder neural network, or both additionally process a conditioning vector representing the target audio source.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE THE SPELLING OF THE 4TH AND 10TH INVENTORS NAME PREVIOUSLY RECORDED AT REEL: 060548 FRAME: 0460. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Jul 25, 2022
From: LI, YUNPENG; TAGLIASACCHI, MARCO; ROBLEK, DOMINIK; DE CHAUMONT QUITRY, FELIX; GFELLER, BEAT; MUCKENHIRN, HANNAH RAPHAELLE; UNGUREANU, VICTOR; RYBAKOV, OLEG; MISIUNAS, KAROLIS; BORSOS, ZALAN
To: GOOGLE LLC
Reel/Frame 060920/0523 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 19, 2022
From: LI, YUNPENG; TAGLIASACCHI, MARCO; ROBLEK, DOMINIK; DE CHAUMONT QUITRY, F?LIX; GFELLER, BEAT; MUCKENHIRN, HANNAH RAPHAELLE; UNGUREANU, VICTOR; RYBAKOV, OLEG; MISIUNAS, KAROLIS; BORSOS, ZAL?N
To: GOOGLE LLC
Reel/Frame 060548/0460 →
Continuity (2)
Provisional Application 63218141 · Jul 2, 2021
Related Publication 20230013370A1 · Jan 19, 2023
References Cited (150)
US 10090001B2 · Theverapperuma et al. · 2018 [cited by applicant]
US 10977555B2 · Jansson · 2021 [cited by examiner]
US 11727926B1 · Liu · 2023 [cited by examiner]
US 11790926B2 · Lee · 2023 [cited by examiner]
US 11854564B1 · Chatlani · 2023 [cited by examiner]
US 20100172519A1 · Kimura · 2010 [cited by examiner]
US 20180329897A1 · Kalchbrenner · 2018 [cited by examiner]
US 20200365166A1 · Zhang · 2020 [cited by examiner]
US 20210117733A1 · Mahto · 2021 [cited by examiner]
US 20210193159A1 · Pearson · 2021 [cited by examiner]
US 20210366461A1 · Ahmed · 2021 [cited by examiner]
US 20220270590A1 · Beaufays · 2022 [cited by examiner]
US 20220277721A1 · Zhang · 2022 [cited by examiner]
US 20220343898A1 · Fu · 2022 [cited by examiner]
US 20230094630A1 · Zhang · 2023 [cited by examiner]
US 20230186937A1 · Uhlich · 2023 [cited by examiner]
US 20230229892A1 · Biswas · 2023 [cited by examiner]
US 20230274754A1 · Ahn · 2023 [cited by examiner]
US 20230326476A1 · Andreev · 2023 [cited by examiner]
Liu, Hung-Ping, Yu Tsao, and Chiou-Shann Fuh. “Bone-conducted speech enhancement using deep denoising autoencoder.” Speech Communication 104 (2018): 106-112. (Year: 2018). [cited by examiner]
Gfeller, Beat, Dominik Roblek, and Marco Tagliasacchi. “One-shot conditional audio filtering of arbitrary sounds.” ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE… [cited by examiner]
Pandey, Ashutosh, and DeLiang Wang. “Densely connected neural network with dilated convolutions for real-time speech enhancement in the time domain.” ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech a… [cited by examiner]
Kaneko, Takuhiro, et al. “Generative adversarial network-based postfilter for statistical parametric speech synthesis.” 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017.… [cited by examiner]
Li, Yunpeng, et al. “Real-time speech frequency bandwidth extension.” ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021. (Year: 2021). [cited by examiner]
Yu, Cheng, et al. “Time-domain multi-modal bone/air conducted speech enhancement.” IEEE Signal Processing Letters 27 (2020): 1035-1039. (Year: 2020). [cited by examiner]
[No Author Listed] [online], “anker 01210 3067,” May 10, 2020, retrieved on Jan. 23, 2024, <https://www.youtube.com/watch?v=ediYpruLT3Y&list=PLa5CkN3odpnxi3WqMH4MgVk7XUjCP99d3&index=2>, 53 pages [Video Submission]. [cited by applicant]
[No Author Listed] [online], “columbia d14543 bx559,” May 10, 2020, retrieved on Jan. 23, 2024, <https://www.youtube.com/watch?v=jaCyLksGQJQ&list=PLa5CkN3odpnxi3WqMH4MgVk7XUjCP99d3&index=3>, 53 pages[Video Submission]. [cited by applicant]
[No Author Listed] [online], “Freddie Meter,” Available on or before Nov. 14, 2019, Internet Archive: Wayback Machine URL <https://web.archive.org/web/20190501000000*/https://freddiemeter.withyoutube.com/>, retrieved on… [cited by applicant]
[No Author Listed] [online], “zonophone x28011 10409u,” May 10, 2020, retrieved on Jan. 23, 2024, <https://www.youtube.com/watch?v=TPWi2344sEQ&list=PLa5CkN3odpnxi3WqMH4MgVk7XUjCP99d3&index=1>, 45 pages [Video Submission… [cited by applicant]
[No Author Listed], “Method for the subjective assessment of intermediate quality level of audio systems,” ITU-Recommendation BS.1534-3, Oct. 2015, 36 pages. [cited by applicant]
Abdulatif et el., “AeGAN: Time-Frequency Speech Denoising via Generative Adversarial Networks,” CoRR, Submitted on Jun. 6, 2020, arXiv:1910.12620v3, 5 pages. [cited by applicant]
Abel et al., “Artificial Bandwidth Extension Using Deep Neural Networks for Spectral Envelope Estimation,” International Workshop on Acoustic Signal Enhancement (IWAENC) 2016, 2016, 5 pages. [cited by applicant]
Acero et al., “Environmental Robustness in Automatic Speech Recognition,” International Conference on Acoustics, Speech, and Signal Processing, 1990, 4 pages. [cited by applicant]
Acero, “Acoustical and Environmental Robustness in Automatic Speech Recognition,” A Dissertation for the degree of Doctor of Philosophy in Electrical Engineering, Carnegie Mellon University, Department of Electrical and… [cited by applicant]
Anand et al., “Spearphone: A Speech Privacy Exploit via Accelerometer-Sensed Reverberations from Smartphone Loudspeakers,” CoRR, Submitted on Jul. 12, 2019, arXiv:1907.05972v1, 16 pages. [cited by applicant]
Ariav et al., “An End-to-End Multimodal Voice Activity Detection Using Wavenet Encoder and Residual Networks,” IEEE Journal of Selected Topics in Signal Processing, May 2019, 13(2):265-274. [cited by applicant]
Armanious et al., “MedGAN: Medical Image Translation using GANs,” CORR, Submitted on Jun. 17, 2018, arXiv:1806.06397v1, 17 pages. [cited by applicant]
Attias et al., “Speech Denoising and Dereverberation Using Probabilistic Models,” Advances in Neural Information Processing Systems 13, 2001, 7 pages. [cited by applicant]
Ba et al., “Layer Normalization,” CoRR, Submitted on Jul. 21, 2016, arXiv:1607.06450v1, 14 pages. [cited by applicant]
Bassiou et al., “Greek Folk Music Denoising Under a Symmetric α-Stable Noise Assumption,” 2014 10th International Conference on Heterogeneous Networking for Quality, Reliability, Security, and Electronics Engineers, 201… [cited by applicant]
Berouti et al., “Enhancement of speech corrupted by acoustic noise,” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1979, 4:208-211. [cited by applicant]
Birnbaum et al., “Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulation,” 33rd Conference on Neural Information Processing Systems, 2019, 12 pages. [cited by applicant]
Biswas et al., “Audio Codec Enhancement with Generative Adversarial Networks,” CoRR, Submitted on Jan. 27, 2020, arXiv:2001.09653v1, 5 pages. [cited by applicant]
Blog.google [online], “Song stuck in your head? Just hum to search,” Oct. 15, 2020, retrieved on Dec. 18, 2023, retrieved from URL <https://blog.google/products/search/hum-to-search/>, 5 pages. [cited by applicant]
Blog.research.google [online], “Improving Speech Representations and Personalized Models Using Self-Supervision,” Jun. 18, 2020, retrieved on Dec. 8, 2023, retrieved from URL <https://blog.research.google/2020/06/improv… [cited by applicant]
Borsos et al., “MicAugment: One-shot Microphone Style Transfer,” CoRR, Submitted on Oct. 19, 2020, arXiv:2010.09658v1, 5 pages. [cited by applicant]
Clevert et al., “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs),” CoRR, Submitted on Feb. 22, 2016, arXiv:1511.07289v5, 14 pages. [cited by applicant]
Datashare.ed.ac.uk [online], “CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92),” Nov. 13, 2019, retrieved on Dec. 22, 2023, retrieved from URL <https://datashare.ed.ac.uk/hand… [cited by applicant]
Datashare.ed.ac.uk [online], “Superseded CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit,” Apr. 4, 2017, retrieved on Dec. 22, 2023, retrieved from URL <https://datashare.ed.ac.uk/handle/10… [cited by applicant]
Deng et al., “Exploiting time-frequency patterns with LSTM-RNNs for low-bitrate audio restoration,” Neural Computing and Applications, 2020, 32(4):1095-1107. [cited by applicant]
Distill.pub [online], “Deconvolution and Checkerboard Artifacts,” Oct. 17, 2016, retrieved on Dec. 21, 2023, retrieved from URL <https://distill.pub/2016/deconv-checkerboard/>, 10 pages. [cited by applicant]
Donahue et al., “Exploring speech enhancement with generative adversarial networks for robust speech recognition,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5024-5028. [cited by applicant]
Dong et al., “Image Super-Resolution Using Deep Convolutional Networks,” CoRR, Submitted on Jul. 31, 2015, arXiv:1501.00092v3, 14 pages. [cited by applicant]
Engel et al., “DDSP: Differentiable Digital Signal Processing,” CoRR, Submitted on Jan. 14, 2020, arXiv:2001.04643v1, 19 pages. [cited by applicant]
Ephraim et al., “Speech Enhancement Using a Minimum Mean-Square Error Log-Spectral Amplitude Estimator,” IEEE Transactions on Acoustics, Speech, and Signal Processing, Apr. 1985, 33(2):443-445. [cited by applicant]
Ephrat et al., “Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation,” ACM Trans. Graph., Aug. 2018, 37(4):1-11. [cited by applicant]
Eskimez et al., “Adversarial Training for Speech Super-Resolution,” IEEE Journal of Selected Topics in Signal Processing, 2019, 13(2):1-12. [cited by applicant]
Feng et al., “Speech feature denoising and dereverberation via deep autoencoders for noisy reverberant speech recognition,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 4, 2014,… [cited by applicant]
Fevotte et al., “Sparse Linear Regression with Structured Priors and Application to Denoising of Musical Audio,” IEEE Transactions on Audio, Speech, and Language Processing, Jan. 2008, 16(1):174-185. [cited by applicant]
Fonseca et al., “FSD50K: An Open Dataset of Human-Labeled Sound Events,” CoRR, Submitted on Oct. 1, 2020, arXiv:2010.00475v1, 24 pages. [cited by applicant]
Font et al., “Freesound Technical Demo,” ACM International Conference on Multimedia (MM'13), Oct. 21-25, 2013, pp. 411-412. [cited by applicant]
Garofolo et al., “TIMIT Acoustic-phonetic Continuous Speech Corpus,” NIST Interagency/Internal Report (NISTIR)—4930, Feb. 1993, 94 pages. [cited by applicant]
Gemmeke et al., “Audio Set: An ontology and human-labeled dataset for audio events,” Proc. IEEE ICASSP, 2017, 5 pages. [cited by applicant]
Germain et al., “Speech Denoising with Deep Feature Looses,” Interspeech 2019, Sep. 15-19, 2019, pp. 2723-2727. [cited by applicant]
Gfeller et al., “One-shot conditional audio filtering of arbitrary sounds,” CoRR, Submitted on Nov. 4, 2020, arXiv:2011.02421v1, 5 pages. [cited by applicant]
Gfeller et al., “SPICE: Self-supervised Pitch Estimation,” CoRR, Submitted on Sep. 4, 2020, arXiv:1910.11664v2, 10 pages. [cited by applicant]
Github.com [online], “google-research/ seanet,” Available on or before Oct. 28, 2020, via Internet Archive: Wayback Machine URL <https://web.archive.org/web/20200615000000*/https://github.com/google-research/seanet>, re… [cited by applicant]
Github.com [online], “Streaming Aware neural network models,” Available on or before May 29, 2020, via Internet Archive: Wayback Machine URL <https://web.archive.org/web/20201001000000*/https://github.com/google-researc… [cited by applicant]
Google-research.github.io [online], “Audio samples for “MicAugment: One-shot Microphone Style Transfer”,” Available on or before Nov. 1, 2020, via Internet Archive: Wayback Machine URL <https://web.archive.org/web/20200… [cited by applicant]
Google-research.github.io [online], “Audio samples for the paper “One-shot conditional audio filtering of arbitrary sounds”,” Available on or before Jan. 26, 2021, via Internet Archive: Wayback Machine URL <https://web.… [cited by applicant]
Google-research.github.io [online], “Audio samples from “SEANet: A Multi-modal Speech Enhancement Network”,” Available on or before Oct. 22, 2020, via Internet Archive: Wayback Machine URL <https://web.archive.org/web/2… [cited by applicant]
Grais et al., “Single channel speech music separation using nonnegative matrix factorization and spectral masks,” 17th International Conference on Digital Signal Processing (DSP), 2011, 6 pages. [cited by applicant]
Graves et al., “Neural Turing Machines,” CoRR, Submitted on Dec. 10, 2014, arXiv:1410.5401v2, 26 pages. [cited by applicant]
Gritsenko et al., “A Spectral Energy Distance for Parallel Speech Synthesis,” CoRR, Submitted on Oct. 23, 2020, arXiv:2008.01160v2, 19 pages. [cited by applicant]
Gupta et al., “Speech bandwidth extension with WaveNet,” CoRR, Submitted on Jul. 5, 2019, arXiv:1907.04927v1, 4 pages. [cited by applicant]
He et al., “Identity Mappings in Deep Residual Networks,” CoRR, Submitted on Jul. 25, 2016, arXiv:1603.05027v3, 15 pages. [cited by applicant]
Hershey et al., “CNN Architectures for Large-Scale Audio Classification,” CoRR, Submitted on Jan. 10, 2017, arXiv:1609.09430v2, 5 pages. [cited by applicant]
Hershey et al., “Model-Based Fusion of Bone and Air Sensors for Speech Enhancement and Robust Speech Recognition,” ISCA Tutorial and Research Workshop on Statistical and Perceptual Audio Processing ICC Jeju, Korea, Oct.… [cited by applicant]
Hou et al., “Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks,” IEEE Transactions on Emerging Topics in Computational Intelligence, 2018, 11 pages. [cited by applicant]
Huang et al., “Deep learning for monaural speech separation,” 2014 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2014, 5 pages. [cited by applicant]
Kamath et al., “A multi-band spectral subtraction method for enhancing speech corrupted by colored noise,” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May 13, 2002, 4 pages. [cited by applicant]
Karras et al., “Progressive Growing of GANs for Improved Quality, Stability, and Variation,” International Conference on Learning Representations, 2018, 26 pages. [cited by applicant]
Kavalerov et al., “Universal sound separation,” IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), Oct. 20-23, 2019, 5 pages. [cited by applicant]
Kilgour et al., “Frechet Audio Distance: A Reference-free Metric for Evaluating Music Enhancement Algorithms,” Interspeech, Sep. 15-19, 2019, pp. 2350-2354. [cited by applicant]
Kim et al., “Bandwidth Extension on Raw Audio via Generative Adversarial Networks,” CoRR, Submitted on Mar. 21, 2019, arXiv:1903.09027v1, 10 pages. [cited by applicant]
Kingma et al., “Adam: A Method for Stochastic Optimization,” CoRR, Submitted on Jan. 30, 2017, arXiv:1412.6980v9, 15 pages. [cited by applicant]
Kolbæk et al., “Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks,” CoRR, Submitted on Jul. 11, 2017, arXiv:1703.06284v2, 12 pages. [cited by applicant]
Kong et al., “Source separation with weakly labelled data: An approach to computational auditory scene analysis,” CoRR, Submitted on Feb. 6, 2020, arXiv:2002.02065v1, 5 pages. [cited by applicant]
Kotropoulos et al., “Mobile Phone Identification Using Recorded Speech Signals,” Proceedings of the 19th International Conference on Digital Signal Processing, Aug. 20-23, 2014, pp. 586-591. [cited by applicant]
Kraetzer et al., “Digital Audio Forensics: A First Practical Evaluation on Microphone and Environment Classification,” Workshop on Multimedia & Security, Sep. 20-21, 2007, 11 pages. [cited by applicant]
Krizhevsky et al., “ImageNet Classification with Deep Convolutional Neural Networks,” Advances in Neural Information Processing Systems 25, 2012, 9 pages. [cited by applicant]
Kuleshov et al., “Audio Super-Resolution using Neural Nets,” CoRR, Submitted on Aug. 2, 2017, arXiv:1708.00853v1, 8 pages. [cited by applicant]
Kumar et al., “MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis,” 33rd Conference on Neural Information Processing Systems, 2019, 12 pages. [cited by applicant]
Li et al., “Learning to Denoise Historical Music,” CoRR, Submitted on Aug. 5, 2020, arXiv:2008.02027v1, 8 pages. [cited by applicant]
Li et al., “Real-time Speech Frequency Bandwidth Extension,” CoRR, Submitted on Oct. 21, 2020, arXiv:2010.10677v1, 5 pages. [cited by applicant]
Lim et al., “Time-Frequency Networks for Audio Super-Resolution,” 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, 5 pages. [cited by applicant]
Liu et al., “Bone-conducted speech enhancement using deep denoising autoencoder,” Speech Communication, Jul. 2, 2018, 104:106-112. [cited by applicant]
Loizou, “Speech Enhancement Based on Perceptually Motivated Bayesian Estimators of the Magnitude Spectrum,” IEEE Transactions on Speech and Audio Processing, Sep. 2005, 13(5):857-869. [cited by applicant]
Luo et al., “TaSNet: Time-domain audio separation network for real-time, single-channel speech separation,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 696-700. [cited by applicant]
Maas et al., “Rectifier Nonlinearities Improve Neural Network Acoustic Models,” Proceedings of the 30th International Conference on Machine Learning, 2013, 6 pages. [cited by applicant]
Maaten et al., “Visualizing Data using t-SNE,” Journal of Machine Learning Research, Nov. 2008, 9(86):2579-2605. [cited by applicant]
Mach, “Denoising phonogram cylinders recordings using Structured Sparsity,” 2015 7th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), 2015, pp. 314-319. [cited by applicant]
Madry et al., “Towards Deep Learning Models Resistant to Adversarial Attacks,” CoRR, Submitted on Sep. 4, 2019, arXiv:1706.06083v4, 28 pages. [cited by applicant]
Mathur et al., “Mic2Mic: Using Cycle-Consistent Generative Adversarial Networks to Overcome Microphone Variability in Speech Systems,” CoRR, Submitted on Mar. 27, 2020, arXiv:2003.12425v1, 12 pages. [cited by applicant]
Mathur et al., “Using Deep Data Augmentation Training to Address Software and Hardware Heterogeneities in Wearable and Smartphone Sensing Devices,” 2018 17th ACM/IEEE International Conference on Information Processing i… [cited by applicant]
Michalevsky et al., “Gyrophone: Recognizing Speech From Gyroscope Signals,” Proceedings of the 23rd USENIX conference on Security Symposium, 2014, pp. 1053-1067. [cited by applicant]
Michelashvili et al., “Audio Denoising with Deep Network Priors,” CoRR, Submitted on Nov. 12, 2019, arXiv:1904.07612v2, 5 pages. [cited by applicant]
Micirp.blogspot.com [online], “Microphone Impulse Response Project,” available on or before Aug. 24, 2016, via Internet Archive: Wayback Machine URL <https://web.archive.org/web/20160515000000*/https://micirp.blogspot.c… [cited by applicant]
Ochiai et al., “Multimodal Speakerbeam: Single channel target speech extraction with audio-visual speaker clues,” Interspeech 2019, Sep. 15-19, 2019, pp. 2718-2722. [cited by applicant]
Panayotov et al., “Librispeech: An ASR corpus based on public domain audio books,” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015, 5 pages. [cited by applicant]
Park et al., “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,” CoRR, Submitted on Dec. 3, 2019, arXiv:1904.08779v3, 6 pages. [cited by applicant]
Pascual et al., “SEGAN: Speech Enhancement Generative Adversarial Network,” Interspeech, Aug. 20-24, 2017, pp. 3642-3646. [cited by applicant]
Perez et al., “FiLM: Visual Reasoning with a General Conditioning Layer,” CoRR, Submitted on Dec. 18, 2017, arXiv:1709.07871v2, 13 pages. [cited by applicant]
Perraudin et al., “Inpainting of long audio segments with similarity graphs,” CoRR, Submitted on Feb. 23, 2018, arXiv:1607.06667v4, 13 pages. [cited by applicant]
Pishdadian et al., “Finding Strength in Weakness: Learning to Separate Sounds with Weak Supervision,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, Sep. 4, 2020, 16 pages. [cited by applicant]
Publicdomainproject.org [online], “Public Domain Project,” available on or before May 1, 2010, via Internet Archive: Wayback Machine URL <https://web.archive.org/web/20100601000000*/https://en.publicdomainproject.org/in… [cited by applicant]
Quitry et al., “Learning audio representations via phase prediction,” CoRR, Submitted on Oct. 25, 2019, arXiv:1910.11910, 5 pages. [cited by applicant]
Reddy et al., “Soft Mask Methods for Single-Channel Speaker Separation,” IEEE Transactions on Audio, Speech, and Language Processing, Aug. 2007, 15(6):1766-1776. [cited by applicant]
Rethage et al., “A Wavenet for Speech Denoising,” CoRR, Submitted on Jan. 31, 2018, arXiv:1706.07162, 11 pages. [cited by applicant]
Ronneberger et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” Medical Image Computing and Computer-Assisted Intervention—MICCAI2015, 2015, pp. 234-241. [cited by applicant]
Roux et al., “SDR—half-baked or well done?,” CoRR, Submitted on Nov. 6, 2018, arXiv:1811.02508v1, 5 pages. [cited by applicant]
Roux et al., “The Phasebook: Building Complex Masks via Discrete Representations for Source Separation,” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Mar. 29, 2019, 6 pages. [cited by applicant]
Rybakov et al., “Streaming keyword spotting on mobile devices,” CoRR, Submitted on Jul. 29, 2020, arXiv:2005.06720v2, 5 pages. [cited by applicant]
Salimans et al., “Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks,” 30th Conference on Neural Information Processing Systems, 2016, 9 pages. [cited by applicant]
Scalart et al., “Speech enhancement based on a priori signal to noise estimation,” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1996, 2:629-632. [cited by applicant]
Shor et al., “Towards Learning a Universal Non-Semantic Representation of Speech,” CoRR, Submitted on Aug. 6, 2020, arXiv:2002.12764v6, 5 pages. [cited by applicant]
Slizovskaia et al., “Conditioned Source Separation for Music Instrument Performances,” CoRR, Submitted on Apr. 8, 2020, arXiv:2004.03873v1, 13 pages. [cited by applicant]
Slizovskaia et al., “End-to-End Sound Source Separation Conditioned On Instrument Labels,” CoRR, Submitted on May 9, 2019, arXiv:1811.01850v2, 5 pages. [cited by applicant]
Stockham, Jr. et al., “Blind Deconvolution Through Digital Signal Processing,” Proceedings of the IEEE, Apr. 1975, 63(4):678-692. [cited by applicant]
Stoller et al., “Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation,” Proceedings of the 19th ISMIR Conference, Sep. 23-27, 2018, pp. 334-340. [cited by applicant]
Tagliasacchi et al., “Multi-Task Adapters for On-Device Audio Inference,” IEEE Signal Processing Letters, 2020, 5 pages. [cited by applicant]
Tagliasacchi et al., “Pre-Training Audio Representations With Self-Supervision,” IEEE Signal Processing Letters, 2020, 27:600-604. [cited by applicant]
Tagliasacchi et al., “SEANet: A Multi-modal Speech Enhancement Network,” CoRR, Submitted on Oct. 1, 2020, arXiv:2009.02095v2, 5 pages. [cited by applicant]
Tzinis et al., “Improving Universal Sound Separation Using Sound Classification,” CoRR, Submitted on Nov. 18, 2019, arXiv:1911.07951v1, 5 pages. [cited by applicant]
Verma et al., “CNN-based System for Speaker Independent Cell-Phone Identification from Recorded Audio,” CVPR Workshops, 2019, pp. 53-61. [cited by applicant]
Virtanen et al., “SciPy 1.0: fundamental algorithms for scientific computing in Python,” Nature Methods, Mar. 2020, pp. 261-272. [cited by applicant]
Wang et al., “Speech Super-Resolution Using Parallel WaveNet,” 11th International Symposium on Chinese Spoken Language Processing (ISCSLP), 2018, 5 pages. [cited by applicant]
Wang et al., “VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking,” Interspeech, Sep. 15-19, 2019, pp. 2728-2732. [cited by applicant]
Warden, “Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition,” CoRR, Submitted on Apr. 9, 2018, arXiv:1804.03209v1, 11 pages. [cited by applicant]
Weninger et al., “Speech Enhancement with LSTM Recurrent Neural Networks and its Application to Noise-Robust ASR,” Latent Variable Analysis and Signal Separation, Aug. 2015, 8 pages. [cited by applicant]
Wilson et al., “Exploring Tradeoffs in Models for Low-latency Speech Enhancement,” CoRR, Submitted on Nov. 16, 2018, arXiv:1811.07030v1, 5 pages. [cited by applicant]
Wisdom et al., “Differentiable consistency constraints for improved deep speech enhancement,” CoRR, Submitted on Nov. 20, 2018, arXiv:1811.08521v1, 5 pages. [cited by applicant]
Wisdom et al., “Unsupervised Sound Separation Using Mixtures of Mixtures,” CoRR, Submitted on Jun. 23, 2020, arXiv:2006.12701v1, 14 pages. [cited by applicant]
Wolfel, “Enhanced Speech Features by Single-Channel Joint Compensation of Noise and Reverberation,” IEEE Transactions on Audio, Speech, and Language Processing, Feb. 2009, 17(2):312-323. [cited by applicant]
Wu et al., “Group Normalization,” CoRR, Submitted on Jun. 11, 2018, arXiv:1803.08494v3, 10 pages. [cited by applicant]
Xaudia.com [online], “Xaudia—Ribbon microphones, sales, and repairs” available on or before Feb. 7, 2011, via Internet Archive: Wayback Machine URL <https://web.archive.org/web/20110401000000*/https://xaudia.com/>, retr… [cited by applicant]
Xu et al., “Recursive Visual Sound Separation Using Minus-Plus Net,” CoRR, Submitted on Aug. 30, 2019, arXiv:1908.11602v1, 10 pages. [cited by applicant]
Zeghidour et al., “Wavesplit: End-to-End Speech Separation by Speaker Clustering,” CoRR, Submitted on Jul. 2, 2020, arXiv:2002.08933v2, 10 pages. [cited by applicant]
Zhao et al., “Robust Speaker Identification in Noisy and Reverberant Conditions,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, Apr. 2014, 22(4):836-845. [cited by applicant]
Zhu et al., “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223-2232. [cited by applicant]
Cited By (1)
US 12,455,214