Granted Patent B2

US 12,431,159 · App. 17/848,341 · Granted Sep 30, 2025

Inventors: Emile de la Rey (Wellington, NZ); Paris Smaragdis (Urbana, IL)

Assignee: WingNut Films Productions Limited

G10L25/51G10L15/063G10L21/0272G10L25/30

Abstract

Claims (35)

1. A system comprising:

a memory component storing machine-readable instructions; and

a logic device configured to execute the machine-readable instructions to:

a trained audio source separation model configured to receive an audio input sample comprising a single-track mixture of audio signals generated from a plurality of audio sources and generate a plurality of audio stems, the plurality of audio stems corresponding to one or more audio source of the plurality of audio sources; and

a self-iterative training system configured to perform a plurality of training iterations, a training iteration comprising generating a new audio source separation model based at least in part on a training dataset comprising a subset of the generated plurality of audio stems from the preceding training iteration, wherein a subset of the generated plurality of audio stems comprises one or more of: a part of one stem, multiple parts of one stem, one complete stem, multiple stems, or a combination of one complete stem and one or more parts of another stem, and wherein the new audio source separation model generated in each iteration is increasingly specific to the mixture of audio signals in the audio input sample,

wherein the self-iterative training system is further configured to determine whether the new audio source separation model is increasingly specific to the mixture of audio signals in the audio input sample by calculating a first quality metric associated with the generated plurality of audio stems, the first quality metric providing a first performance measure of the audio source separation model of the prior iteration, calculating a second quality metric associated with the audio stems generated in the present iteration, the second quality metric providing a second performance measure of the new audio source separation model, and wherein the second quality metric is greater than the first quality metric; and

wherein the new audio source separation model is configured to re-process the audio input stream to generate a plurality of enhanced audio stems.

2. The system of claim 1 , wherein the trained audio source separation model comprises a neural network trained to separate one or more audio source signals from the single-track mixture of audio signals.

3. The system of claim 2 , wherein the neural network is configured to perform audio source separation without applying a mask.

4. The system of claim 1 , further comprising a training dataset comprising labeled source audio data and labeled noise audio data, and wherein the trained audio source separation model is initially trained using the training dataset to generate a general source separation model.

5. The system of claim 4 , wherein during each training iteration at least a subset of the plurality of audio stems are culled based on a threshold metric and added to the training dataset from the preceding training iteration to form a culled dynamically evolving dataset, and wherein the culled dynamically evolving dataset is used to train the new audio source separation model.

6. The system of claim 1 , wherein the trained audio source separation model is trained using a training dataset comprising a plurality of datasets, each of the plurality of datasets comprising labeled audio samples configured to train the system to address source separation associated with an identified source.

7. The system of claim 6 , wherein the plurality of datasets comprises a speech training dataset comprising a plurality of labeled speech samples, and/or a non-speech training dataset comprising a plurality of labeled music and/or noise data samples.

8. The system of claim 1 , wherein the self-iterative training system further comprises a self-iterative dataset generation module configured to generate labeled audio samples from the generated plurality of audio stems.

9. The system of claim 1 , wherein the plurality of enhanced audio stems is generated using a hierarchical branching sequence comprising separating a source signal and a remaining complement signal.

10. A method comprising:

receiving an audio input stream comprising a mixture of audio signals generated from a plurality of audio sources;

generating, using a trained audio source separation model configured to receive the audio input stream, a generated plurality of audio stems corresponding to one or more audio sources of the plurality of audio sources;

updating, the trained audio source separation model through a plurality of training iterations, where a training iteration comprises generating a new audio source separation model based at least in part on a subset of the generated plurality of audio stems derived from a preceding training iteration, wherein a subset of the generated plurality of audio stems comprises one or more of: a part of one stem, multiple parts of one stem, one complete stem, multiple stems, or a combination of one complete stem and one or more parts of another stem;

wherein updating the trained audio source separation model further comprises:

calculating a first quality metric associated with the generated plurality of audio stems, the first quality metric providing a first performance measure of the trained audio source separation model; calculating a second quality metric associated with the enhanced audio stems, the second quality metric providing a performance measure of the new audio source separation model; and comparing the second quality metric to the first quality metric to confirm the second quality metric is greater than the first quality metric; and

re-processing the audio input stream using the new audio source separation model to generate a plurality of enhanced audio stems.

11. The method of claim 10 , wherein the audio input stream comprises one or more single-track audio mixtures, and wherein the trained audio source separation model comprises a neural network trained to separate one or more audio source signals from the one or more single-track audio mixtures.

12. The method of claim 11 , wherein the neural network is configured to perform audio source separation without applying a mask.

13. The method of claim 10 , further comprising:

providing a training dataset comprising labeled source audio data and labeled noise audio data; and

training the trained audio source separation model using the training dataset to generate a general source separation model.

14. The method of claim 13 , further comprising:

adding at least a subset of the generated plurality of audio stems to the training dataset to produce a dynamically evolving dataset;

culling the dynamically evolving dataset based on a threshold metric; and

training the new audio source separation model using the culled, dynamically evolving dataset.

15. The method of claim 10 , wherein the trained audio source separation model is trained using training dataset comprising a plurality of datasets, each of the plurality of datasets comprising labeled audio samples configured to train the audio source separation model to address a different source separation problem.

16. The method of claim 15 , wherein the plurality of datasets comprises a speech training dataset comprising a plurality of labeled speech samples, and/or a non-speech training dataset comprising a plurality of labeled music and/or noise data samples.

17. The method of claim 16 , wherein updating the trained audio source separation model further comprises generating labeled audio samples from the generated plurality of audio stems for a self-iterative dataset.

18. The method of claim 10 , further comprising generating the plurality of enhanced audio stems using a hierarchical branching sequence comprising separating a source signal and a remaining complement signal.

Assignments (2)

Continuity (2)

References Cited (28)