Data augmentation system and method for multi-microphone systems
A method, computer program product, and computing system for obtaining one or more speech signals from a first device, thus defining one or more first device speech signals. One or more speech signals may be obtained from a second device, thus defining one or more second device speech signals. One or more acoustic relative transfer functions mapping reverberation from the one or more first device speech signals to the one or more second device speech signals may be generated. One or more augmented second device speech signals may be generated based upon, at least in part, the one or more acoustic relative transfer functions and first device training data.
1 . A computer-implemented method, executed on a computing device, comprising:
obtaining one or more first device speech signals from a first device, wherein a first speech processing machine learning (ML) model was trained to process voice audio recorded by the first device using training data associated with the first device;
obtaining one or more second device speech signals from a second device;
generating augmented training data from the training data associated with the first device by generating one or more acoustic relative transfer functions (RTFs) mapping reverberation from the one or more first device speech signals to the one or more second device speech signals and processing the training data associated with the first device based on the one or more acoustic RTFs to generate the augmented training data, the one or more acoustic RTFs including one or more static acoustic RTFs generated by extracting a single acoustic relative transfer function per segment of the first device speech signals and the second device speech signals and one or more dynamic acoustic RTFs generated by extracting acoustic RTFs from the first device speech signals and the second device speech signals at predefined time increments, wherein the one or more dynamic acoustic RTFs model speaker movements in speech signals captured by the second device; and
training a second speech processing ML model to process voice audio recorded by the second device using the augmented training data.
2 . The computer-implemented method of claim 1 , further comprising:
processing the one or more first device speech signals; and
processing the one or more second device speech signals.
3 . The computer-implemented method of claim 2 , wherein processing the one or more first device speech signals includes:
detecting one or more speech active portions from the one or more first device speech signals;
identifying a speaker associated with the one or more speech active portions from the one or more first device speech signals; and
applying signal filtering to the one or more speech active portions associated with a predefined signal bandwidth to generate one or more first device filtered speech active portions.
4 . The computer-implemented method of claim 2 , wherein processing the one or more second device speech signals includes:
detecting one or more speech active portions from the one or more second device speech signals;
identifying a speaker associated with the one or more speech active portions from the one or more second device speech signals; and
applying signal filtering to the one or more speech active portions associated with a predefined signal bandwidth to generate one or more second device filtered speech active portions.
5 . The computer-implemented method of claim 1 , wherein generating one or more acoustic RTFs includes modeling relationships between characteristics of the one or more first device speech signals and characteristics of the one or more second device speech signals utilizing an adaptive filter.
6 . The computer-implemented method of claim 1 , further comprising:
adding the one or more acoustic RTFs to a codebook of acoustic RTFs.
7 . The computer-implemented method of claim 1 , wherein the first speech processing ML model is trained to process near field microphone system (NFMS) audio recorded by the first device using NFMS training data associated with the first device.
8 . The computer-implemented method of claim 7 , wherein the second ML model is trained to process far field microphone system (FFMS) audio recorded by the second device using the augmented training data.
9 . The computer-implemented method of claim 1 , wherein the second device is a microphone array that includes multiple audio acquisition devices deployed in an acoustic environment, and wherein the first device is a near field microphone system (NFMS) that is independent from the microphone array.
10 . A computer program product residing on a non-transitory computer readable medium having programming instructions stored thereon which, when executed by one or more processors of a system, cause the system to perform the following operations:
obtaining one or more first device speech signals from a first device, wherein a first speech processing machine learning (ML) model was trained to process voice audio recorded by the first device using training data associated with the first device;
obtaining one or more second device speech signals from a second device;
generating augmented training data from the training data associated with the first device by generating one or more acoustic relative transfer functions (RTFs) mapping reverberation from the one or more first device speech signals to the one or more second device speech signals and processing the training data associated with the first device based on the one or more acoustic RTFs to generate the augmented training data, the one or more acoustic RTFs including one or more static acoustic RTFs generated by extracting a single acoustic relative transfer function per segment of the first device speech signals and the second device speech signals and one or more dynamic acoustic RTFs generated by extracting acoustic RTFs from the first device speech signals and the second device speech signals at predefined time increments, wherein the one or more dynamic acoustic RTFs model speaker movements in speech signals captured by the second device; and
training a second speech processing ML model to process voice audio recorded by the second device using the augmented training data.
11 . The computer program product of claim 10 , wherein the operations further comprise:
processing the one or more first device speech signals; and
processing the one or more second device speech signals.
12 . The computer program product of claim 11 , wherein processing the one or more first device speech signals includes:
detecting one or more speech active portions from the one or more first device speech signals;
identifying a speaker associated with the one or more speech active portions from the one or more first device speech signals; and
applying signal filtering to the one or more speech active portions associated with a predefined signal bandwidth to generate one or more first device filtered speech active portions.
13 . The computer program product of claim 11 , wherein processing the one or more second device speech signals includes:
detecting one or more speech active portions from the one or more second device speech signals;
identifying a speaker associated with the one or more speech active portions from the one or more second device speech signals; and
applying signal filtering to the one or more speech active portions associated with a predefined signal bandwidth to generate one or more second device filtered speech active portions.
14 . The computer program product of claim 10 , wherein generating one or more acoustic RTFs includes modeling relationships between characteristics of the one or more first device speech signals and characteristics of the one or more second device speech signals utilizing an adaptive filter.
15 . The computer program product of claim 10 , wherein the operations further comprise:
adding the one or more acoustic RTFs to a codebook of acoustic RTFs.
16 . A computing system comprising:
one or more processors; and
a memory storing programming instructions for execution by the one or more processors, the programming instructions, upon execution by the one more processors, causing the computing system to perform the following operations:
obtaining one or more first device speech signals from a first device, wherein a first speech processing machine learning (ML) model was trained to process voice audio recorded by the first device using training data associated with the first device;
obtaining one or more second device speech signals from a second device;
generating augmented training data from the training data associated with the first device by generating one or more acoustic relative transfer functions (RTFs) mapping reverberation from the one or more first device speech signals to the one or more second device speech signals and processing the training data associated with the first device based on the one or more acoustic RTFs to generate the augmented training data, the one or more acoustic RTFs including one or more static acoustic RTFs generated by extracting a single acoustic relative transfer function per segment of the first device speech signals and the second device speech signals and one or more dynamic acoustic RTFs generated by extracting acoustic RTFs from the first device speech signals and the second device speech signals at predefined time increments, wherein the one or more dynamic acoustic RTFs model speaker movements in speech signals captured by the second device; and
training a second speech processing ML model to process voice audio recorded by the second device using the augmented training data.
17 . The computing system of claim 16 , wherein the programming instructions further cause the computing system to perform the following operations:
process the one or more first device speech signals; and
process the one or more second device speech signals.
18 . The computing system of claim 17 , wherein processing the one or more first device speech signals includes:
detecting one or more speech active portions from the one or more first device speech signals;
identifying a speaker associated with the one or more speech active portions from the one or more first device speech signals; and
applying signal filtering to the one or more speech active portions associated with a predefined signal bandwidth to generate one or more first device filtered speech active portions.
19 . The computing system of claim 16 , wherein processing the one or more second device speech signals includes:
detecting one or more speech active portions from the one or more second device speech signals;
identifying a speaker associated with the one or more speech active portions from the one or more second device speech signals; and
applying signal filtering to the one or more speech active portions associated with a predefined signal bandwidth to generate one or more second device filtered speech active portions.
20 . The computing system of claim 16 , wherein generating one or more acoustic RTFs includes modeling relationships between characteristics of the one or more first device speech signals and characteristics of the one or more second device speech signals utilizing an adaptive filter.