Training network to minimize worst case surprise
Some embodiments provide a method for training a machine-trained (MT) network that processes input data using network parameters. The method maps input instances to output values by propagating the instances through the network. The input instances include instances for each of multiple categories. For a particular instance selected as an anchor instance, the method identifies each instance in a different category as a negative instance. The method calculates, for each negative instance of the anchor, a surprise function that probabilistically measures a surprise of finding an output value for an instance in the same category as the anchor that is a greater distance from the output value for the anchor instance than output value for the negative instance. The method calculates a loss function that emphasizes a maximum surprise calculated for the anchor. The method trains the network parameters using the calculated loss function value to minimize the maximum surprise.
1. A method for training a machine-trained (MT) network that processes input data using a plurality of network parameters, the method comprising:
for each input image of a set of input images, propagating the input image through the MT network to generate a corresponding output value indicating a category into which the MT network classifies the input image;
selecting a particular input image as an anchor image;
for each respective input image in a different category from the anchor image:
calculating a distance between the output value for the anchor image and the output value for the respective image in the different category; and
calculating a value for a function that probabilistically measures a likelihood of a distance between the output value for an input image in the same category as the anchor image being greater than the calculated distance between the output value for the anchor image and the output value for the respective image in the different category;
calculating a loss function for the anchor image that emphasizes a maximum value for the function calculated for the anchor image; and
training the network parameters using the calculated loss function to minimize the maximum value for the function.
2. The method of claim 1 , wherein calculating the loss function for the anchor image comprises calculating a loss term as a function of a natural logarithm of a sum of an exponential function of the value calculated for the function for each respective input image in a different category from the anchor image.
3. The method of claim 2 , wherein the loss term is further a function of the maximum value for the function calculated for the anchor image and each of the respective input images in a different category from the anchor image.
4. The method of claim 1 , wherein, for each respective input image, calculating the value for the function for the respective input image comprises calculating a natural logarithm of an inverse of a function of the distance between the output value for the anchor image and the output value for the respective input image.
5. The method of claim 1 , wherein each output value is a point in multiple dimensions, wherein the distance between the output value for the anchor image and the output value for a respective input image in a different category is computed by summing a square of distances between a point representing the output value for the anchor image and a point representing the output value for the respective image in each of the multiple dimensions.
6. The method of claim 1 further comprising calculating a mean of the distances between the output value for the anchor image and the output values for each input image in the same category as the anchor image, wherein the value of the function for each respective input image in a different category from the anchor image is a function of the calculated mean.
7. The method of claim 6 further comprising calculating a standard deviation for the distances between the output value for the anchor image and the output values for each input image in the same category as the anchor image, wherein the value of the function for each respective input image in a different category from the anchor image is a function of the standard deviation.
8. The method of claim 1 , wherein the value of the function for each respective input image in a different category from the anchor image is a function of a distribution of distances between the output value for the anchor image and the output values for each input image in the same category as the anchor image.
9. The method of claim 8 , wherein the distances are normally distributed.
10. The method of claim 1 , wherein the MT network is a neural network comprising input nodes, output nodes, and interior nodes between the input nodes and output nodes, wherein each node produces a node output value and each interior node and output node receives as input values a set of node output values of other nodes.
11. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit trains a machine-trained (MT) network that processes input data using a plurality of network parameters, the program comprising sets of instructions for:
for each input image of a set of input images, propagating the input image through the MT network to generate a corresponding output value indicating a category into which the MT network classifies the input image;
selecting a particular input image as an anchor image;
for each respective input image in a different category from the anchor image:
calculating a distance between the output value for the anchor image and the output value for the respective image in the different category; and
calculating a value for a function that probabilistically measures a likelihood of a distance between the output value for an input image in the same category as the anchor image being greater than the calculated distance between the output value for the anchor image and the output value for the respective image in the different category;
calculating a loss function for the anchor image that emphasizes a maximum value for the function calculated for the anchor image; and
training the network parameters using the calculated loss function to minimize the maximum value for the function.
12. The non-transitory machine-readable medium of claim 11 , wherein the set of instructions for calculating the loss function for the anchor image comprises a set of instructions for calculating a loss term as a function of a natural logarithm of a sum of an exponential function of the value calculated for the function for each respective input image in a different category from the anchor image.
13. The non-transitory machine-readable medium of claim 12 , wherein the loss term is further a function of the maximum value for the function calculated for the anchor image and each of the respective input images in a different category from the anchor image.
14. The non-transitory machine-readable medium of claim 11 , wherein, for each respective input image, the set of instructions for calculating the value for the function for the respective input image comprises a set of instructions for calculating a natural logarithm of an inverse of a function of the distance between the output value for the anchor image and the output value for the respective input image.
15. The non-transitory machine-readable medium of claim 11 , wherein each output value is a point in multiple dimensions, wherein the distance between the output value for the anchor image and the output value for a respective input image in a different category is computed by summing a square of distances between a point representing the output value for the anchor image and a point representing the output value for the respective image in each of the multiple dimensions.
16. The non-transitory machine-readable medium of claim 11 , wherein the program further comprises a set of instructions for calculating a mean of the distances between the output value for the anchor image and the output values for each input image in the same category as the anchor image, wherein the value of the function for each respective input image in a different category from the anchor image is a function of the mean.
17. The non-transitory machine-readable medium of claim 16 , wherein the program further comprises a set of instructions for calculating a standard deviation for the distances between the output value for the anchor image and the output values for each input image in the same category as the anchor image, wherein the value of the function for each respective input image in a different category from the anchor image is a function of the standard deviation.
18. The non-transitory machine-readable medium of claim 11 , wherein the value of the function for each respective input image in a different category from the anchor image is a function of a distribution of distances between the output value for the anchor image and the output values for each input image in the same category as the anchor image.
19. The non-transitory machine-readable medium of claim 18 , wherein the distances are normally distributed.
20. The non-transitory machine-readable medium of claim 11 , wherein the MT network is a neural network comprising input nodes, output nodes, and interior nodes between the input nodes and output nodes, wherein each node produces a node output value and each interior node and output node receives as input values a set of node output values of other nodes.