Neural network loss function that incorporates incorrect category probabilities
Some embodiments provide a method for training a machine-trained (MT) network to classify inputs into multiple categories. The method propagates a set of input training items through the MT network to generate a set of corresponding outputs. Each input training item belongs to a category and the output for each input training item includes, for each category, a computed probability of the input belonging to the category. The method computes a value for a loss function based on the generated outputs. The loss function includes a first term based on the computed probabilities of each input belonging to its category and not based on individual computed probabilities of the inputs belonging to other categories and a second term based on the individual computed probabilities of each input belonging to each of the categories. The method uses the computed value for the loss function to train the MT network.
1. A method for training a machine-trained (MT) network to classify inputs into a plurality of categories, the method comprising:
propagating a set of input training items through the MT network to generate a set of corresponding outputs, wherein (i) each respective input training item belongs to a respective category and (ii) the respective output for each respective input training item comprises, for each category, a computed probability of the input belonging to the category;
computing a value for a loss function based on the generated outputs, the loss function comprising (i) a first term based on the computed probabilities of each respective input belonging to its respective category and not based on individual computed probabilities of the inputs belonging to other categories and (ii) a second term based on the individual computed probabilities of each respective input belonging to each of the categories; and
using the computed value for the loss function to train the MT network.
2. The method of claim 1 , wherein the first term is a cross-entropy term that, for each respective input, is based on a logarithm of the computed probability of the input belonging to its respective category.
3. The method of claim 2 , wherein the loss function further comprises a complementary cross-entropy term that, for each respective input, is based on a logarithm of the computed probability of the input not belonging to its respective category.
4. The method of claim 1 , wherein each of the loss function terms is continuously differentiable.
5. The method of claim 1 , wherein the second term approximates, for each respective input, a maximum of a function applied to each of the individual computed probabilities of the respective input belonging to categories other than its respective category, wherein the function increases as probability increases.
6. The method of claim 1 , wherein the second term approximates, for each respective input, a minimum of a function applied to each of the individual computed probabilities of the respective input belonging to categories other than its respective category, wherein the function increases as probability increases.
7. The method of claim 1 , wherein the second term is a log-sum-exponent term that emphasizes, for each respective input, a largest of the individual computed probabilities of the respective input belonging to categories other than its respective category.
8. The method of claim 1 , wherein the second term is a log-sum-exponent term that emphasizes, for each respective input, a smallest of the individual computed probabilities of the respective input belonging to categories other than its respective category.
9. The method of claim 1 , wherein the second term is a regularizing term that, for each respective input, is smallest when the individual computed probabilities of the respective input belonging to categories other than its respective category are evenly distributed.
10. The method of claim 1 , wherein the input training items are images and the plurality of different categories are types of objects depicted in the images.
11. The method of claim 1 , wherein the input training items are audio recordings and the plurality of different categories are different acoustic scenes in which the audio recordings are captured.
12. The method of claim 1 , wherein the loss function for a particular input is minimized for a given computed probability of the input belonging to its category when the computed probabilities of the input belonging to categories other than its respective category are evenly distributed.
13. The method of claim 1 , wherein using the computed value for the loss function to train the MT network comprises:
backpropagating the computed loss function through the MT network to compute gradients of the loss function with respect to each of a plurality of parameters of the network; and
using the computed gradients to modify the plurality of parameters of the network.
14. A non-transitory machine-readable medium storing a program which when executed by at least one processor trains a machine-trained (MT) network to classify inputs into a plurality of categories, the program comprising sets of instructions for:
propagating a set of input training items through the MT network to generate a set of corresponding outputs, wherein (i) each respective input training item belongs to a respective category and (ii) the respective output for each respective input training item comprises, for each category, a computed probability of the input belonging to the category;
computing a value for a loss function based on the generated outputs, the loss function comprising (i) a first term based on the computed probabilities of each respective input belonging to its respective category and not based on individual computed probabilities of the inputs belonging to other categories and (ii) a second term based on the individual computed probabilities of each respective input belonging to each of the categories; and
using the computed value for the loss function to train the MT network.
15. The non-transitory machine-readable medium of claim 14 , wherein:
the first term is a cross-entropy term that, for each respective input, is based on a logarithm of the computed probability of the input belonging to its respective category; and
the loss function further comprises a complementary cross-entropy term that, for each respective input, is based on a logarithm of the computed probability of the input not belonging to its respective category.
16. The non-transitory machine-readable medium of claim 14 , wherein the second term approximates, for each respective input, a maximum of a function applied to each of the individual computed probabilities of the respective input belonging to categories other than its respective category, wherein the function increases as probability increases.
17. The non-transitory machine-readable medium of claim 14 , wherein the second term approximates, for each respective input, a minimum of a function applied to each of the individual computed probabilities of the respective input belonging to categories other than its respective category, wherein the function increases as probability increases.
18. The non-transitory machine-readable medium of claim 14 , wherein the second term is a log-sum-exponent term that emphasizes, for each respective input, a largest of the individual computed probabilities of the respective input belonging to categories other than its respective category.
19. The non-transitory machine-readable medium of claim 14 , wherein the second term is a log-sum-exponent term that emphasizes, for each respective input, a smallest of the individual computed probabilities of the respective input belonging to categories other than its respective category.
20. The non-transitory machine-readable medium of claim 14 , wherein the second term is a regularizing term that, for each respective input, is smallest when the individual computed probabilities of the respective input belonging to categories other than its respective category are evenly distributed.