Granted Patent B2

US 12,444,429 · App. 18/085,705 · Granted Oct 14, 2025

Inventors: Brian B Monson (Champaign, IL); Rohit M Ananthanarayana (Champaign, IL)

Assignee: The Board of Regents of the University of Illinois

G10L21/0232G10L21/0224G10L21/0272G10L21/0308G10L25/09G10L25/18G10L25/21G10L25/84

Abstract

Claims (74)

1. A non-transitory computer readable medium comprising program instructions executable by at least one processor to cause the at least one processor to perform a method comprising:

obtaining a first audio sample;

determining that a first portion of the first audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds a threshold energy level;

responsive to determining that the first portion contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level, determining a first audio filter based on the first portion of the first audio sample by:

determining a first spectrogram for the first portion; and

performing non-negative matrix factorization to generate a first matrix and a second matrix whose product corresponds to a low-frequency portion of the first spectrogram that is below a threshold frequency, wherein the first matrix is composed of a set of column vectors that span along a frequency dimension of the first spectrogram, and wherein the second matrix is composed of a set of row vectors that span along a time dimension of the first spectrogram;

subsequent to obtaining the first audio sample, obtaining a second audio sample; and

applying the first audio filter to the second audio sample to generate a first audio output by:

determining a second spectrogram for the second audio sample;

applying the first matrix to a low-frequency portion of the second spectrogram that is below the threshold frequency to generate a third spectrogram that represents noise content of the second audio sample; and

using the third spectrogram to remove the noise content from the second audio sample, thereby generating the first audio output.

2. The non-transitory computer readable medium of claim 1 , wherein the method further comprises:

determining a plurality of zero-crossing rates across time for the first audio sample; and

determining a plurality of signal energy levels across time for the first audio sample, wherein determining that the first portion contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level comprises determining (i) that a zero-crossing rate, of the plurality of zero-crossing rates, that corresponds to the first portion exceeds a threshold zero-crossing rate and (ii) that a signal energy level, of the plurality of signal energy levels, that corresponds to the first portion exceeds a threshold signal energy level.

3. The non-transitory computer readable medium of claim 1 , wherein the first audio sample is divided into a plurality of non-overlapping frames, and wherein determining that the first portion contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level comprises:

determining that a contiguous subset of the plurality of non-overlapping frames of the first audio sample all contain frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level, wherein the first portion consists of the contiguous subset of frames of the first audio sample.

4. The non-transitory computer readable medium of claim 3 , wherein each frame of the plurality of non-overlapping frames of the first audio sample has a duration between 15 milliseconds and 50 milliseconds.

5. The non-transitory computer readable medium of claim 1 , wherein the method further comprises:

prior to obtaining the first audio sample, obtaining a third audio sample;

determining that a second portion of the third audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level; and

responsive to determining that the second portion of the third audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level, determining a second audio filter based on the second portion of the third audio sample by:

determining a fourth spectrogram for the second portion; and

performing non-negative matrix factorization to generate a third matrix and a fourth matrix whose product corresponds to a portion of the fourth spectrogram below the threshold frequency, wherein the third matrix is composed of a further set of column vectors that span along a frequency dimension of the fourth spectrogram, and wherein the fourth matrix is composed of a further set of row vectors that span along a time dimension of the fourth spectrogram,

wherein performing non-negative matrix factorization to generate the first matrix and the second matrix comprises using, as an initial estimate of the first matrix, the third matrix.

6. The non-transitory computer readable medium of claim 1 , wherein determining that the first portion contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level comprises:

determining a spectrogram for the first portion; and

determining that a total energy in the spectrogram above 5.6 kilohertz exceeds the threshold energy level.

7. The non-transitory computer readable medium of claim 1 , wherein using the third spectrogram to remove the noise content from the second audio sample comprises:

performing an inverse transform on the third spectrogram to generated a time-domain noise signal; and

subtracting the time-domain noise signal from the second audio sample to generate the first audio output.

8. The non-transitory computer readable medium of claim 1 , wherein the method further comprises:

prior to obtaining the first audio sample, obtaining a third audio sample;

determining that a second portion of the third audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level; and

responsive to determining that the second portion of the third audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level, determining a second audio filter based on the second portion of the third audio sample,

wherein determining the first audio filter based on the first portion comprises:

determining a third audio filter based on the first portion; and

determining the first audio filter as a weighted combination of the second audio filter and the third audio filter.

9. A method comprising:

obtaining a first audio sample;

determining that a first portion of the first audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds a threshold energy level;

responsive to determining that the first portion contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level, determining a first audio filter based on the first portion of the first audio sample by:

determining a first spectrogram for the first portion; and

performing non-negative matrix factorization to generate a first matrix and a second matrix whose product corresponds to a low-frequency portion of the first spectrogram that is below a threshold frequency, wherein the first matrix is composed of a set of column vectors that span along a frequency dimension of the first spectrogram, and wherein the second matrix is composed of a set of row vectors that span along a time dimension of the first spectrogram;

subsequent to obtaining the first audio sample, obtaining a second audio sample; and

applying the first audio filter to the second audio sample to generate a first audio output by:

determining a second spectrogram for the second audio sample;

applying the first matrix to a low-frequency portion of the second spectrogram that is below the threshold frequency to generate a third spectrogram that represents noise content of the second audio sample; and

using the third spectrogram to remove the noise content from the second audio sample, thereby generating the first audio output.

10. The method of claim 9 , further comprising:

determining a plurality of zero-crossing rates across time for the first audio sample; and

determining a plurality of signal energy levels across time for the first audio sample, wherein determining that the first portion contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level comprises determining (i) that a zero-crossing rate, of the plurality of zero-crossing rates, that corresponds to the first portion exceeds a threshold zero-crossing rate and (ii) that a signal energy level, of the plurality of signal energy levels, that corresponds to the first portion exceeds a threshold signal energy level.

11. The method of claim 9 , wherein the first audio sample is divided into a plurality of non-overlapping frames, and wherein determining that the first portion contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level comprises:

determining that a contiguous subset of the plurality of non-overlapping frames of the first audio sample all contain frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level, wherein the first portion consists of the contiguous subset of frames of the first audio sample.

12. The method of claim 11 , wherein each frame of the plurality of non-overlapping frames of the first audio sample has a duration between 15 milliseconds and 50 milliseconds.

13. The method of claim 9 , further comprising:

prior to obtaining the first audio sample, obtaining a third audio sample;

determining that a second portion of the third audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level; and

responsive to determining that the second portion of the third audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level, determining a second audio filter based on the second portion of the third audio sample by:

determining a fourth spectrogram for the second portion; and

performing non-negative matrix factorization to generate a third matrix and a fourth matrix whose product corresponds to a portion of the fourth spectrogram below the threshold frequency, wherein the third matrix is composed of a further set of column vectors that span along a frequency dimension of the fourth spectrogram, and wherein the fourth matrix is composed of a further set of row vectors that span along a time dimension of the fourth spectrogram,

wherein performing non-negative matrix factorization to generate the first matrix and the second matrix comprises using, as an initial estimate of the first matrix, the third matrix.

14. The method of claim 9 , wherein determining that the first portion contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level comprises:

determining a spectrogram for the first portion; and

determining that a total energy in the spectrogram above 5.6 kilohertz exceeds the threshold energy level.

15. The method of claim 9 , wherein using the third spectrogram to remove the noise content from the second audio sample comprises:

performing an inverse transform on the third spectrogram to generated a time-domain noise signal; and

subtracting the time-domain noise signal from the second audio sample to generate the first audio output.

16. The method of claim 9 , further comprising:

prior to obtaining the first audio sample, obtaining a third audio sample;

determining that a second portion of the third audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level; and

responsive to determining that the second portion of the third audio sample contains frequency content at frequencies higher than 5.6 kilohertz that exceeds the threshold energy level, determining a second audio filter based on the second portion of the third audio sample,

wherein determining the first audio filter based on the first portion comprises:

determining a third audio filter based on the first portion; and

determining the first audio filter as a weighted combination of the second audio filter and the third audio filter.

Assignments (1)

Continuity (2)

References Cited (132)