Fast quantised training of trainable modules
A method for training a trainable module that maps input variables onto output variables through an internal processing chain. A learning data set is provided including learning values of the input variables and associated learning values of the output variables. A list of discrete values is provided from which the parameters characterizing the internal processing chain are to be selected, the discrete values being selected such that they can be stored without loss of quality. The learning values are mapped by the trainable module onto assessment values of the output variables. A cost function is evaluated that characterizes deviations of the assessment values of the output variables from the learning values and of at least one parameter of the internal processing chain from at least one discrete value in the list. At least one parameter of the internal processing chain is adjusted to improve the value of the cost function.
1 . A method for training a trainable module that maps one or more input variables onto one or more output variables through an internal processing chain, the internal processing chain being characterized by a set of parameters, the method comprising the following steps:
training the trainable module, the training including:
providing at least one learning data set that includes learning values of the input variables and associated learning values of the output variables;
providing a list of discrete values from which the parameters characterizing the internal processing chain are to be selected during the training, the discrete values being selected such that the discrete values can be stored with a specified number N of bits as a fixed-point number without loss of quality;
mapping, by the trainable module, the learning values contained in the learning data set of the input variables onto assessment values of the output variables;
evaluating a specified cost function that characterizes both a deviation of the assessment values of the output variables from the learning values, contained in the learning data set, of the output variables, and also a deviation of at least one of the parameters of the internal processing chain from at least one of the discrete values in the list; and
adjusting at least one of the parameters of the internal processing chain with a goal of improving a value of the cost function, wherein a value range of the parameters, and/or of a gradient of the cost function, is limited during the adjustment of the at least one of the parameters, using the discrete values.
2 . The method as recited in claim 1 , wherein, during the at least one adjustment of at least one of the parameters:
values of the parameters that are lower than a lowest discrete value of the list are set to the lowest discrete value, and/or
values of the parameters that are higher than a highest discrete value of the list are set to the highest discrete value.
3 . The method as recited in claim 1 , wherein in at least one adjustment of the parameters, a gradient of the cost function expressed in the parameters as variables is ascertained, the gradient being in a direction towards better values of the cost function, and the parameters being modified by a product of the gradient and a step width, in which, in the ascertaining of the gradient:
components of the gradient that relate to those of the parameters that currently have a lowest discrete value of the list are limited to non-negative values, and/or
components of the gradient that relate to those of the parameters that currently have a highest discrete value of the list are limited to non-positive values.
4 . The method as recited in claim 1 , wherein the cost function contains a weighted sum of at least:
a first contribution that characterizes a deviation of the assessment values of the output variables from the learning values, contained in the learning data set, of the output variables, and
a second contribution that characterizes a deviation of at least one parameter of the internal processing chain from at least one discrete value in the list,
wherein, as the training progresses, a weighting of the first contribution is reduced and a weighting of the second contribution is increased.
5 . The method as recited in claim 1 , wherein the number N of bits is between 2 and 7.
6 . The method as recited in claim 1 , wherein the number N of bits is between 2 and 5.
7 . The method as recited in claim 1 , wherein the number N of bits is between 2 and 3.
8 . The method as recited in claim 1 , wherein the list of discrete values is ascertained based on values of the parameters of the internal processing chain that are obtained during a pre-training of the trainable module, the pre-training including the following steps:
mapping, by the trainable module, learning values contained in at least one learning data set, of the input variables onto assessment values of the output variables;
evaluating a specified pre-training cost function that characterizes a deviation of the assessment values of the output variables from the learning values, contained in the learning data set, of the output variables; and
adjusting at least one parameter of the internal processing chain with a goal of improving a value of the pre-training cost function.
9 . The method as recited in claim 8 , wherein a value range is ascertained in which the parameters lie, and discrete values of the list are ascertained as a function of the ascertain value range.
10 . The method as recited in claim 1 , wherein the number N of bits and/or the list of discrete values is adjusted with a goal of improving the value of the cost function.
11 . The method as recited in claim 1 , wherein the discrete values in the list have whole-number values, and discrete values that are numerically adjacent to one another differing in each case by a step width that is a power of two of a non-negative whole number.
12 . The method as recited in claim 1 , wherein the discrete values in the list are distributed uniformly around 0.
13 . The method as recited in claim 1 , wherein each of the parameters is set to that discrete value from the list to which it is closest, and the trainable module is validated after the setting of the parameters to the discrete values, the validating including the following steps:
providing a multiplicity of validation data sets that each include respective validation values of the input variables and associated validation values of the output variables, the validation data sets not being identical with the learning data sets;
for each of the validation data sets, mapping, by the trainable module, the respective validation values of the input variables onto test values of the output variables; and
checking whether deviations of the test values from the validation values of the output variables fulfill a specified criterion.
14 . The method as recited in claim 1 , wherein the list of discrete values includes at least three discrete values.
15 . The method as recited in claim 1 , further comprising outputting the trainable module.
16 . The method as recited in claim 15 , wherein each of the parameters is set to that discrete value from the list to which it is closest, and the output trainable module is validated after the setting of the parameters to the discrete values.
17 . A method, comprising:
training a trainable module that maps one or more input variables onto one or more output variables through an internal processing chain, the internal processing chain being characterized by a set of parameters, the training including:
providing at least one learning data set that includes learning values of the input variables and associated learning values of the output variables,
providing a list of discrete values from which the parameters characterizing the internal processing chain are to be selected during the training, the discrete values being selected such that the discrete values can be stored with a specified number N of bits as a fixed-point number without loss of quality,
mapping, by the trainable module, the learning values contained in the learning data set of the input variables onto assessment values of the output variables,
evaluating a specified cost function that characterizes both a deviation of the assessment values of the output variables from the learning values, contained in the learning data set, of the output variables, and also a deviation of at least one of the parameters of the internal processing chain from at least one of the discrete values in the list, and
adjusting at least one of the parameters of the internal processing chain with a goal of improving a value of the cost function, wherein a value range of the parameters, and/or of a gradient of the cost function, is limited during the adjustment of the at least one of the parameters, using the discrete values;
operating the trainable module by supplying one or more input variables to the trainable module; and
as a function of the output variables supplied by the trainable module, controlling a vehicle or a robot or a quality control system or a system for monitoring an area, based on sensor data.
18 . A non-transitory machine-readable data carrier on which is stored a computer program for training a trainable module that maps one or more input variables onto one or more output variables through an internal processing chain, the internal processing chain being characterized by a set of parameters, the computer program, when executed by one or more computers, causing the one or more computers to perform the following steps:
training the trainable module, the training including:
providing at least one learning data set that includes learning values of the input variables and associated learning values of the output variables;
providing a list of discrete values from which the parameters characterizing the internal processing chain are to be selected during the training, the discrete values being selected such that the discrete values can be stored with a specified number N of bits as a fixed-point number without loss of quality;
mapping, by the trainable module, the learning values contained in the learning data set of the input variables onto assessment values of the output variables;
evaluating a specified cost function that characterizes both a deviation of the assessment values of the output variables from the learning values, contained in the learning data set, of the output variables, and also a deviation of at least one of the parameters of the internal processing chain from at least one of the discrete values in the list; and
adjusting at least one of the parameters of the internal processing chain with a goal of improving a value of the cost function, wherein a value range of the parameters, and/or of a gradient of the cost function, is limited during the adjustment of the at least one of the parameters, using the discrete values.
19 . A computer configured to train a trainable module that maps one or more input variables onto one or more output variables through an internal processing chain, the internal processing chain being characterized by a set of parameters, the computer configured to:
train the trainable module, the training including:
provide at least one learning data set that includes learning values of the input variables and associated learning values of the output variables;
provide a list of discrete values from which the parameters characterizing the internal processing chain are to be selected during the training, the discrete values being selected such that the discrete values can be stored with a specified number N of bits as a fixed-point number without loss of quality;
map, by the trainable module, the learning values contained in the learning data set of the input variables onto assessment values of the output variables;
evaluate a specified cost function that characterizes both a deviation of the assessment values of the output variables from the learning values, contained in the learning data set, of the output variables, and also a deviation of at least one of the parameters of the internal processing chain from at least one of the discrete values in the list; and
adjust at least one of the parameters of the internal processing chain with a goal of improving a value of the cost function, wherein a value range of the parameters, and/or of a gradient of the cost function, is limited during the adjustment of the at least one of the parameters, using the discrete values.