Method for personalization of ASR models
The disclosure generally relate to a method performed by a user device obtaining a pre-trained automatic speech recognition (ASR) model, obtaining a user data from a user database, analysing a distribution of the user data with respect to an acoustic characteristic, determining, using the distribution, whether data augmentation for the acoustic characteristic is to be applied, when it is determined that data augmentation is to be applied, dividing the user data into a training subset and a validation subset, based on an acoustic characteristic being less audible in the training subset than in the validation subset, applying data augmentation to add the acoustic characteristic to the user data in the training subset, and updating the pre-trained ASR model with the augmented training subset to generate a personalised local ML model.
1 . A method, performed by a user device, for generating a local machine learning (ML) model which is personalised to a user of the user device, the method comprising:
obtaining a pre-trained automatic speech recognition (ASR) model;
obtaining a user data from a user database;
analysing a distribution of the user data with respect to an acoustic characteristic;
determining, using the distribution, whether data augmentation for the acoustic characteristic is to be applied;
when it is determined that data augmentation is to be applied, dividing the user data into a training subset and a validation subset, wherein the acoustic characteristic is less audible in the training subset than in the validation subset;
applying data augmentation to add the acoustic characteristic to the training subset; and
updating the pre-trained ASR model with the augmented training subset to generate a personalised local ML model.
2 . The method of claim 1 , wherein the applying of the data augmentation comprises applying the data augmentation to add the acoustic characteristic to the training subset in parallel.
3 . The method of claim 1 ,
wherein the acoustic characteristic is selected from environmental acoustic characteristics, user acoustic characteristics, and semantic and syntactic characteristics,
wherein the environmental acoustic characteristics includes noise, signal to noise ratio and reverberation,
wherein the user acoustic characteristics includes speed and spectral characteristics, and
wherein the semantic and syntactic characteristics includes meaning and structure of a text uttered by the user.
4 . The method of claim 1 , wherein the analysing of the distribution of the user data with respect to an acoustic characteristic comprises plotting a probability distribution P(θ|X) for user dataset X for acoustic characteristic θ.
5 . The method of claim 1 , wherein the determining, using the distribution, whether the data augmentation is to be applied, comprises selecting data augmentation parameters which maximise a probability of generating similar acoustic characteristics in the user data.
6 . The method of claim 1 ,
wherein the determining, using the distribution, whether the data augmentation is to be applied, comprises:
setting personalised settings, for the user, for a data augmentation parameter which corresponds to the acoustic characteristic, and
wherein the personalised settings comprise at least one statistical value calculated from the corresponding distribution.
7 . The method of claim 6 , further comprising:
comparing a value of at least one of the personalised settings determined from the corresponding distribution to a threshold; and
when the value of the at least one of the personalised settings meets the threshold, determining that the data augmentation is to be applied.
8 . The method of claim 7 , further comprising:
generating a signal having an acoustic characteristic which matches values of the at least one of the personalised settings determined from the corresponding distribution; and
adding the generated signal when applying the data augmentation.
9 . The method of claim 1 , further comprising:
obtaining, using at least one user data in the validation subset, a signal which matches the acoustic characteristic of the at least one user data; and
applying the data augmentation to add the obtained signal to user data in the training subset.
10 . The method of claim 9 , further comprising:
generating a noise signal to be applied the data augmentation,
wherein the applying of the data augmentation comprises adding the noise signal.
11 . The method of claim 10 , wherein the generating of the noise signal comprises:
extracting noise frames which are individual audio frames in the validation subset containing noise;
concatenating consecutive noise frames to form multiple noise segments; and
concatenating a subset of the noise segments to form a noise signal which is longer than the training subset.
12 . The method of claim 11 , further comprising:
comparing each of the multiple noise segments to a length threshold; and
rejecting at least one noise segment which are shorter than the length threshold.
13 . The method of claim 10 , further comprising:
retrieving a noise signal which is similar to the generated noise signal from a generic noise dataset; and
adding the retrieved noise signal when applying data augmentation.
14 . The method of claim 10 , wherein the generating of the noise signal to be applied the data augmentation comprises generating the noise signal to be applied by extracting background noise using a speech separation model.
15 . The method of claim 9 , wherein the applying of the data augmentation comprises applying a reverberation signal.
16 . The method of claim 15 , further comprising:
estimating a value of reverberation in the user data of the validation subset;
selecting the reverberation signal in a form of a room impulse response (RIR) with a matching value of reverberation; and
storing the selected reverberation signal in a personal dataset for the user.
17 . The method of claim 1 , further comprising:
obtaining, at the user device, input speech;
processing the input speech using the personalised local ML model to determine a voice command from the user; and
implementing the voice command on the user device to enable voice-based control of the user device.
18 . A user device comprising:
a memory storing one or more instructions; and
at least one processor configured to execute the one or more instructions to:
obtain a pre-trained automatic speech recognition (ASR) model,
obtain a user data from a user database,
analyse a distribution of the user data with respect to an acoustic characteristic,
determine, using the distribution, whether data augmentation for the acoustic characteristic is to be applied,
when it is determined that data augmentation is to be applied, divide the user data into a training subset and a validation subset, based on an acoustic characteristic being less audible in the training subset than in the validation subset,
apply data augmentation to add the acoustic characteristic in the training subset, and
update the pre-trained ASR model with the augmented training subset to generate a personalised local machine learning (ML) model.
19 . The user device of claim 18 , wherein the at least one processor is further configured to execute the one or more instructions to:
obtain, using at least one user data in the validation subset, a signal which matches the acoustic characteristic of the at least one user data, and
apply the data augmentation to add the obtained signal to user data in the training subset.
20 . A non-transitory data carrier carrying code which, when implemented on a processor of a user device, causes the user device to perform operations, the operations comprising:
obtaining a pre-trained automatic speech recognition (ASR) model;
obtaining a user data from a user database;
analysing a distribution of the user data with respect to an acoustic characteristic;
determining, using the distribution, whether data augmentation for the acoustic characteristic is to be applied;
when it is determined that data augmentation is to be applied, dividing the user data into a training subset and a validation subset, wherein the acoustic characteristic is less audible in the training subset than in the validation subset;
applying data augmentation to add the acoustic characteristic to the training subset; and
updating the pre-trained ASR model with the augmented training subset to generate a personalised local machine learning (ML) model.