Training network to minimize worst-case error
Some embodiments provide a method for configuring a machine-trained (MT) network that includes multiple configurable weights to train. The method propagates a set of inputs through the MT network to generate a set of output probability distributions. Each input has a corresponding expected output probability distribution. The method calculates a value of a continuously-differentiable loss function that includes a term approximating an extremum function of the difference between the expected output probability distributions and generated set of output probability distributions. The method trains the weights by back-propagating the calculated value of the continuously-differentiable loss function.
1. A method for configuring a machine-trained (MT) network that receives inputs and, for each input, categorizes the input into one of a set of discrete categories, the MT network comprising a plurality of configurable weights to train, the method comprising:
propagating a set of inputs through the MT network to generate a set of output probability distributions, the generated output probability distribution for each input providing a probability of the input being categorized into each of the discrete categories by the MT network, each input having a corresponding expected output probability distribution that indicates the input belongs to a particular category;
calculating a value of a continuously-differentiable loss function comprising a term that uses a natural logarithm of a summation of a plurality of exponential functions to approximate an extremum function of the difference between the expected output probability distributions and generated set of output probability distributions, wherein (i) the index of the summation is the set of discrete categories and (ii) the exponent of the exponential function for each category is a function of generated output probability distributions for inputs belonging to the category; and
training the weights by back-propagating the calculated value of the continuously-differentiable loss function, wherein the term that approximates the extremum function biases the training of the weights towards reducing a difference between the expected output probability distributions and the generated output probability distributions for inputs belonging to a category with a greatest difference.
2. The method of claim 1 , wherein:
the term that uses a natural logarithm of a summation of a plurality of exponential functions approximates a maximum function; and
training the weights comprises minimizing the approximation of the maximum function.
3. The method of claim 1 , wherein:
the term that uses a natural logarithm of a summation of a plurality of exponential functions approximates a minimum function; and
training the weights comprises maximizing the approximation of the minimum function.
4. The method of claim 1 , wherein:
the set of inputs comprises a plurality of inputs for each of the categories; and
for each category of input, the expected output probability distribution for each input is a same expected output probability distribution.
5. The method of claim 4 , wherein the expected output probability distribution for each input is 1 for the category to which the input belongs and 0 for each other category.
6. The method of claim 1 , wherein, for each category, the function of the generated output probability distributions for inputs belonging to the category is an entropy calculation for the generated output probability distributions for inputs belonging to the category.
7. The method of claim 6 , wherein the entropy calculation for the generated output probability distributions for inputs belonging to a particular category comprises a sum over each discrete probability in the generated output probability distributions multiplied by the negative of the base-2 logarithm of the discrete probability.
8. The method of claim 6 , wherein the term that uses a natural logarithm of a summation of a plurality of exponential functions to approximate the extremum function biases the training of the weights towards weight values that minimize the maximum of the entropy calculations for the discrete categories.
9. The method of claim 1 , wherein the MT network comprises input nodes, output nodes, and interior nodes between the input and output nodes, wherein each node produces an output value and each interior node and output node receives as input values a set of output values of input nodes or other interior nodes and applies a set of the configurable weights to each received input value.
10. The method of claim 1 further comprising performing the propagating, calculating, and back-propagating iteratively.
11. The method of claim 1 , wherein training the weights comprises:
back-propagating the calculated value through the MT network to determine, for each weight, a rate of change in the calculated value relative to a rate of change in the weight; and
modifying each particular weight according to the determined rate of change for the particular weight.
12. The method of claim 1 , wherein the MT network is for embedding into a device after training is complete.
13. The method of claim 1 , wherein propagating the set of inputs through the MT network comprises calculating an output value for each interior node and output node, wherein calculating the output value for a particular interior node comprises:
receiving a set of input values from a set of other interior nodes;
calculating a linear summation of each input value multiplied by a corresponding weight value; and
applying a non-linear function to the summation to calculate the output value for the particular node.
14. The method of claim 10 , wherein the propagating, calculating, and back-propagating are performed iteratively until each of the weights have been changed by less than a particular threshold for a particular number of iterations.
15. The method of claim 1 , wherein the inputs are input images and the MT network categorizes each input based on an object shown in the image.
16. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit configures a machine-trained (MT) network that receives inputs and, for each input, categorizes the input into one of a set of discrete categories, the MT network comprising a plurality of configurable weights to train, the program comprising sets of instructions for:
propagating a set of inputs through the MT network to generate a set of output probability distributions, the generated output probability distribution for each input providing a probability of the input being categorized into each of the discrete categories by the MT network, each input having a corresponding expected output probability distribution that indicates the input belongs to a particular category;
calculating a value of a continuously-differentiable loss function comprising a term that uses a natural logarithm of a summation of a plurality of exponential functions to approximate an extremum function of the difference between the expected output probability distributions and generated set of output probability distributions, wherein (i) the index of the summation is the set of discrete categories and (ii) the exponent of the exponential function for each category is a function of generated output probability distributions for inputs belonging to the category; and
training the weights by back-propagating the calculated value of the continuously-differentiable loss function, wherein the term that approximates the extremum function biases the training of the weights towards reducing a difference between the expected output probability distributions and the generated output probability distributions for inputs belonging to a category with a greatest difference.
17. The non-transitory machine-readable medium of claim 16 , wherein:
the set of inputs comprises a plurality of inputs for each of the categories; and
for each category of input, the expected output probability distribution for each input is a same expected output probability distribution.
18. The non-transitory machine-readable medium of claim 17 , wherein the expected output probability distribution for each input is 1 for the category to which the input belongs and 0 for each other category.
19. The non-transitory machine-readable medium of claim 16 , wherein, for each category, the function of the generated output probability distributions for inputs belonging to the category is an entropy calculation for the generated output probability distributions for inputs belonging to the category.
20. The non-transitory machine-readable medium of claim 19 , wherein the entropy calculation for the generated output probability distributions for inputs belonging to a particular category comprises a sum over each discrete probability in the generated output probability distributions multiplied by the negative of the base-2 logarithm of the discrete probability.
21. The non-transitory machine-readable medium of claim 19 , wherein the term that uses a natural logarithm of a summation of a plurality of exponential functions to approximate the extremum function biases the training of the weights towards weight values that minimize the maximum of the entropy calculations for the discrete categories.
22. The non-transitory machine-readable medium of claim 16 , wherein the inputs are input images and the MT network categorizes each input based on an object shown in the image.