Using batches of training items for training a network
Some embodiments provide a method for training a machine-trained (MT) network that processes inputs using network parameters. The method propagates a set of input training items through the MT network to generate a set of output values. The set of input training items comprises multiple training items for each of multiple categories. The method identifies multiple training item groupings in the set of input training items. Each grouping includes at least two training items in a first category and at least one training item in a second category. The method calculates a value of a loss function as a summation of individual loss functions for each of the identified training item groupings. The individual loss function for each particular training item grouping is based on the output values for the training items of the grouping. The method trains the network parameters using the calculated loss function value.
1. A method for training a machine-trained (MT) network that classifies inputs into a plurality of categories, the method comprising:
propagating a plurality of input training items through the MT network to generate a respective output value for each respective input training item, the plurality of input training items comprising input training items for each of the categories;
identifying a plurality of triplets from the plurality of input training items, wherein each respective triplet comprises two input training items in a respective first category and one input training item in a respective second category, wherein a plurality of the input training items belong to at least two different triplets;
calculating a value of a loss function as a summation of respective individual loss functions for each of the respective identified triplets, the respective individual loss function for each respective triplet based on the output values generated for the input training items of the triplet; and
training the MT network using the calculated loss function value.
2. The method of claim 1 , wherein the input items are images and the categories comprise different types of objects found in the images.
3. The method of claim 1 , wherein:
the input items are images and the categories comprise a plurality of different people, wherein each input training item for a particular person comprises a different image of the particular person's face; and
the machine-trained network is for embedding into a device to perform facial recognition.
4. The method of claim 1 , wherein:
the set of input images comprises N i images for each of N p categories; and
a number of identified triplets is of the order (N i ){circumflex over ( )}3*(N p ){circumflex over ( )}2.
5. The method of claim 1 , wherein:
each output value is a vector; and
the respective individual loss function for each respective triplet is a function of the proximity of the vectors for one of the input training items in the first respective category and the input training item in the respective second category to the vector for the other input training item in the first respective category.
6. The method of claim 1 , wherein:
each output value is a vector; and
the respective individual loss function for each respective triplet is a function of the probability of a misclassification of one of the input training items in the respective first category based on the vector of the three input training items of the triplet.
7. The method of claim 1 , wherein:
the MT network comprises input nodes, output nodes, and interior nodes between the input nodes and output nodes;
each node produces a node output value and each interior node and output node receives as node input values a set of node output values of other nodes and applies weights to each received input value; and
training the MT network comprises training the weights.
8. The method of claim 1 , wherein the propagating, identifying, calculating, and training are performed iteratively.
9. The method of claim 1 , wherein training the MT network comprises:
backpropagating the calculated loss function value through the MT network to determine, for each of a set of network parameters, a rate of change in the calculated loss function value relative to a rate of change in the network parameter; and
modifying each network parameter in the set according to the determined rate of change for the network parameter.
10. The method of claim 1 , wherein identifying the plurality of triplets comprises identifying each possible triplet formed by the plurality of input training items.
11. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit trains a machine-trained (MT) network that classifies inputs into a plurality of categories, the program comprising sets of instructions for:
propagating a plurality of input training items through the MT network to generate a respective output value for each respective input training item, the plurality of input training items comprising input training items for each of the categories;
identifying a plurality of triplets from the plurality of input training items, wherein each respective triplet comprises two input training items in a respective first category and one input training item in a respective second category, wherein a plurality of the input training items belong to at least two different triplets;
calculating a value of a loss function as a summation of respective individual loss functions for each of the respective identified triplets, the respective individual loss function for each respective triplet based on the output values generated for the input training items of the triplet; and
training the MT network using the calculated loss function value.
12. The non-transitory machine-readable medium of claim 11 , wherein the input items are images and the categories comprise different types of objects found in the images.
13. The non-transitory machine-readable medium of claim 11 , wherein:
the input items are images and the categories comprise a plurality of different people, wherein each input training item for a particular person comprises a different image of the particular person's face; and
the machine-trained network is for embedding into a device to perform facial recognition.
14. The non-transitory machine-readable medium of claim 11 , wherein:
the set of input images comprises N i images for each of N p categories; and
a number of identified triplets is of the order (N i ){circumflex over ( )}3*(N p ){circumflex over ( )}3.
15. The non-transitory machine-readable medium of claim 11 , wherein:
each output value is a vector; and
the respective individual loss function for each respective triplet is a function of the proximity of the vectors for one of the input training items in the first respective category and the input training item in the respective second category to the vector for the other input training item in the first respective category.
16. The non-transitory machine-readable medium of claim 11 , wherein:
each output value is a vector; and
the respective individual loss function for each respective triplet is a function of the probability of a misclassification of one of the input training items in the respective first category based on the vector of the three input training items of the triplet.
17. The non-transitory machine-readable medium of claim 11 , wherein:
the MT network comprises input nodes, output nodes, and interior nodes between the input nodes and output nodes;
each node produces a node output value and each interior node and output node receives as node input values a set of node output values of other nodes and applies weights to each received input value; and
the set of instructions for training the MT network comprises a set of instructions for training the weights.
18. The non-transitory machine-readable medium of claim 11 , wherein the propagating, identifying, calculating, and training are performed iteratively.
19. The non-transitory machine-readable medium of claim 11 , wherein the set of instructions for training the MT network comprises sets of instructions for:
backpropagating the calculated loss function value through the MT network to determine, for each of a set of network parameters, a rate of change in the calculated loss function value relative to a rate of change in the network parameter; and
modifying each network parameter in the set according to the determined rate of change for the network parameter.
20. The non-transitory machine-readable medium of claim 11 , wherein the set of instructions for identifying the plurality of triplets comprises a set of instructions for identifying each possible triplet formed by the plurality of input training items.