Method for personalisation of ASR models
The present techniques generally relate to a computer-implemented method for personalising automatic speech recognition (ASR) or other general-purpose models using mixed precision (MP) quantization. Each model is personalised on a user device or server to a target memory budget B using user data.
1 . A computer-implemented method for generating a personalized local machine learning (ML) model data, the method comprising:
obtaining a general ML model in a form of a neural network comprising a plurality of layers with each layer having a set of general weights;
obtaining a memory budget for the local ML model;
obtaining a set of local data samples; and
for each layer in the plurality of layers of the general ML model:
detecting a sensitivity of the layer to quantization by applying the general ML model to the set of local data samples;
determining, using the detected sensitivity of the layer and the obtained memory budget, an optimum bit depth for the corresponding layer in the local ML model, so that the local ML model satisfies the memory budget;
defining at least one scaling factor for the layer to scale the set of general weights to the optimum bit depth; and
generating a set of quantized weights by quantizing the set of general weights for the layer using the at least one scaling factor,
wherein the set of quantized weights for each layer define the personalized local ML model, and
wherein there are multiple optimum bit depths which satisfy a uniformity constraint.
2 . The method of claim 1 , wherein the memory budget for the local ML model is between a tenth to a half of a memory budget required to store the general ML model.
3 . The method of claim 1 , wherein the general ML model is an automatic speech recognition model and the set of local data samples comprise audio data samples.
4 . The method of claim 1 , wherein detecting the sensitivity of the layer to quantization comprises:
detecting activation of each layer; and
calculating, using the detected activation, a statistic which is indicative of the sensitivity of the layer to quantization.
5 . The method of claim 4 , further comprising:
determining an optimum bit depth by:
ranking each layer using the calculated statistic; and
assigning lower optimum bit depths to the lower ranked layers.
6 . The method of any claim 1 , wherein determining an optimum bit depth comprises:
reducing a bit depth of a first layer in the plurality of layers;
computing a size of the local ML model with the reduced bit depth;
comparing the calculated size with the memory budget,
when the calculated size exceeds the memory budget, reducing a bit depth of a subsequent layer in the plurality of layers; and
repeating the computing, comparing and reducing a bit depth of a subsequent layer until the calculated size meets the memory budget.
7 . The method of claim 6 , wherein the first layer is the lowest ranked layer and a subsequent layer is a layer which is higher in the ranking.
8 . The method of claim 1 , wherein defining at least one scaling factor for the layer comprises:
detecting minimum and maximum values activation of each layer when applying the general ML model to the set of local data samples; and
defining a scaling factor for each layer using a difference between the minimum and maximum values activation of each layer.
9 . The method of claim 8 , wherein the scaling factor S l for each layer is defined by:
S
l
=
(
X
l
M
-
X
l
m
)
/
(
2
b
l
-
1
)
where X l m is the minimum activation value for each layer of the full precision model, X l M is the maximum activation value for each layer of the full precision model and b l is the number of bits in the layer.
10 . The method of claim 1 , wherein there is one scaling factor for each layer and each scaling factor is defined by minimizing a distance between a quantized output from each layer of the local ML model when using a calibration data set and an output from the corresponding layer in the general ML model when using the calibration data set.
11 . The method of claim 1 , wherein there are two scaling factors for each layer and the two scaling factors are defined using two quantization ranges per layer.
12 . The method of claim 11 , further comprising:
defining the two scaling factors by performing a linear search using a calibration data set.
13 . The method of claim 12 , further comprising:
using a Hessian-based calibration optimization in the linear search.
14 . A user device for generating a personalized local machine learning (ML) model, wherein the user device is configured to:
obtain a general ML model in the form of a neural network comprising a plurality of layers with each layer having a set of general weights;
obtain a memory budget for the local ML model;
obtain a set of local data samples; and
for each layer in the plurality of layers of the general ML model:
detect, using a sensitivity module, a sensitivity of the layer to quantization by applying the general ML model to the set of local data samples;
determine, using the detected sensitivity of the layer to quantization and the obtained memory budget, an optimum bit depth for the corresponding layer in the local ML model, so that the local ML model satisfies the memory budget;
define at least one scaling factor for the layer to scale the set of general weights to the optimum bit depth; and
generate, using a quantization module, a set of quantized weights by quantizing the set of general weights for the layer using the at least one scaling factor,
wherein the set of quantized weights for each layer define the personalized local ML model, and
wherein there are multiple optimum bit depths which satisfy a uniformity constraint.
15 . A server for generating personalized local machine learning (ML) model data to be used on a user device, wherein the server is configured to:
train a general ML model in the form of a neural network comprising a plurality of layers with each layer having a set of general weights;
obtain a memory budget for the local ML model to be stored on the user device;
obtain a set of local data samples from the user device; and
for each layer in the plurality of layers of the general ML model:
detect, using a sensitivity module, a sensitivity of the layer to quantization by applying the general ML model to the set of local data samples;
determine, using the detected sensitivity of the layer to quantization and the obtained memory budget, an optimum bit depth for the corresponding layer in the local ML model, so that the local ML model satisfies the memory budget;
define at least one scaling factor for the layer to scale the set of general weights to the optimum bit depth; and
generate, using a quantization module, a set of quantized weights by quantizing the set of general weights for the layer using the at least one scaling factor,
wherein the set of quantized weights for each layer define the personalized local ML model, and
wherein there are multiple optimum bit depths which satisfy a uniformity constraint.