Federated learning with only positive labels
Generally, the present disclosure is directed to systems and methods that perform spreadout regularization to enable learning of a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a limited number of classes (e.g., a single class). Examples of such settings include decentralized training of face recognition models or speaker identification models, where in addition to the user specific facial images and voice samples, the class embeddings for the users also constitute sensitive information that cannot be shared with other users.
1 . A computing system that performs spreadout regularization to prevent model collapse in a federated learning setting with only positive labels, the computing system comprising:
a coordinating computing system comprising one or more processors and one or more memory devices storing instructions that, when executed by the one or more processors, cause the coordinating computing system to perform coordinating operations, the coordinating operations comprising, for at least one of one or more update iterations:
accessing, from one or more databases stored in the one or more memory devices, a current classification matrix that comprises a plurality of class embeddings respectively associated with a plurality of classes;
receiving, via a network interface and respectively from one or more client computing devices, one or more updates respectively to one or more class embeddings of the plurality of class embeddings, wherein each update to one of the class embeddings was generated by a corresponding client computing device based on only positive training examples that have positive labels for a corresponding class, wherein the client computing devices lack access to negative class embeddings for other classes, and wherein each update is transmitted to the coordinating computing system without transmitting the positive training examples themselves, thereby reducing communication of private information;
generating an intermediate classification matrix that comprises a plurality of intermediate class embeddings, wherein generating the intermediate classification matrix comprises applying the one or more updates respectively to the one or more class embeddings of the plurality of class embeddings;
performing a spreadout regularization on at least a subset of the intermediate class embeddings contained in the intermediate classification matrix to obtain an updated classification matrix, wherein performing the spreadout regularization comprises modifying at least one of the class embeddings included in at least the subset of the intermediate class embeddings to increase a cumulative spacing among at least the subset of the intermediate class embeddings, thereby preventing the class embeddings from collapsing to a shared point in an embedding space; and
outputting the updated classification matrix for use in performing classification.
2 . The computing system of claim 1 , wherein:
receiving, via the network interface and respectively from the one or more client computing devices, the one or more updates respectively to the one or more class embeddings comprises receiving, via a network interface and respectively from the one or more client computing devices, one or more replacement class embeddings respectively for the one or more class embeddings; and
applying the one or more updates respectively to the one or more class embeddings comprises respectively replacing the one or more class embeddings with the one or more replacement class embeddings.
3 . The computing system of claim 1 , wherein performing the spreadout regularization on at least the subset of the intermediate class embeddings comprises performing a single unified spreadout regularization on all of the class embeddings contained in the intermediate classification matrix to obtain the updated classification matrix.
4 . The computing system of claim 1 , wherein performing the spreadout regularization on at least the subset of the intermediate class embeddings comprises performing, separately for each of one or more class embeddings, the spreadout regularization relative to only a subset of nearest class embeddings surrounding such class embedding.
5 . The computing system of claim 4 , wherein performing, separately for each of the one or more class embeddings, the spreadout regularization relative to only the subset of nearest class embeddings surrounding such class embedding comprises performing, separately for each of the class embeddings contained in the intermediate classification matrix, the spreadout regularization relative to only the subset of nearest class embeddings surrounding such class embedding.
6 . The computing system of claim 1 , wherein performing the spreadout regularization comprises updating one or more of the class embeddings contained in the classification matrix according to a learning rate and in a direction negative to a gradient of a spreadout regularization function.
7 . The computing system of claim 1 , wherein performing the spreadout regularization comprises evaluating a spreadout regularization function that evaluates the cumulative spacing among at least the subset of the intermediate class embeddings, and wherein a spacing between a pair of class embeddings comprises a square of a maximum of a first value or a second value, wherein the first value equals zero and the second value equals a margin value minus a distance between the pair of class embeddings.
8 . The computing system of claim 1 , wherein performing the spreadout regularization comprises evaluating a spreadout regularization function that evaluates the cumulative spacing among only the subset of the intermediate class embeddings, and wherein a spacing between a pair of class embeddings comprises a negative of a squared distance between the pair of class embeddings.
9 . The computing system of claim 1 , wherein outputting the updated classification matrix for use in performing classification comprises respectively transmitting one or more of the class embeddings from the updated classification matrix to the one or more client computing devices for use at the client computing devices in performing classification of inputs relative to the corresponding class.
10 . The computing system of claim 1 , further comprising:
at least one client computing device of the one or more client computing devices, wherein the at least one client computing device is configured to perform client operations, the client operations comprising, for at least one of the one or more update iterations:
receiving a current version of a first class embedding from the coordinating computing system for a first class of the plurality of classes, wherein the current version of the first class embedding is contained in the updated classification matrix, and wherein at least a majority of training examples accessible by the client computing device comprise positive training examples that have positive labels for the first class;
determining an update to the first class embedding for the first class based on the positive training examples; and
communicating the update to the first class embedding to the coordinating computing system.
11 . The computing system of claim 10 , wherein the client computing device comprises a smartphone.
12 . The computing system of claim 1 , wherein the plurality of classes comprises a plurality of facial recognition classes or speech recognition classes.
13 . The computing system of claim 12 , wherein the positive training examples for each class comprise facial images or speech audio that has a positive label for such class.
14 . A client computing device, comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
one or more positive training examples that have a positive label for a first class of a plurality of different classes; and
instructions that, when executed by the one or more processors, cause the client computing device to perform client operations, the client operations comprising, for at least one of one or more update iterations:
receiving, via a network interface and from a coordinating computing system, a current version of a first class embedding associated with the first class;
determining an update to the first class embedding for the first class based only on the one or more positive training examples, and wherein the client computing device lacks access to negative class embeddings for other classes; and
communicating, via the network interface, the update to the first class embedding to the coordinating computing system for spreadout regularization by the coordinating computing system, wherein the update is communicated to the coordinating computing system without transmitting the positive training examples themselves, thereby reducing communication of private information, wherein the spreadout regularization increases a cumulative spacing among at least a subset of a plurality of class embeddings associated with the plurality of different classes, thereby preventing the class embeddings from collapsing to a shared point in an embedding space, and wherein the first class embedding is included in at least the subset of the plurality of class embeddings.
15 . One or more non-transitory computer-readable media that collectively store class embeddings from an updated class classification matrix having been generated by performance of operations, the operations comprising:
accessing, from one or more databases stored in the one or more memory devices, a current classification matrix that comprises a plurality of class embeddings respectively associated with a plurality of classes;
receiving, via a network interface and respectively from one or more client computing devices, one or more updates respectively to one or more class embeddings of the plurality of class embeddings, wherein each update to one of the class embeddings was generated by a corresponding client computing device based on only positive training examples that have positive labels for a corresponding class, wherein the client computing devices lack access to negative class embeddings for other classes, and wherein each update is transmitted to the coordinating computing system without transmitting the positive training examples themselves, thereby reducing communication of private information;
generating an intermediate classification matrix that comprises a plurality of intermediate class embeddings, wherein generating the intermediate classification matrix comprises applying the one or more updates respectively to the one or more class embeddings of the plurality of class embeddings;
performing a spreadout regularization on at least a subset of the intermediate class embeddings contained in the intermediate classification matrix to obtain the updated classification matrix, wherein performing the spreadout regularization comprises modifying at least one of the class embeddings included in at least the subset of the intermediate class embeddings to increase a cumulative spacing among at least the subset of the intermediate class embeddings, thereby preventing the class embeddings from collapsing to a shared point in an embedding space; and
outputting the updated classification matrix for use in performing classification.
16 . The one or more non-transitory computer-readable media of claim 15 , wherein:
receiving, via the network interface and respectively from the one or more client computing devices, the one or more updates respectively to the one or more class embeddings comprises receiving, via the network interface and respectively from the one or more client computing devices, one or more replacement class embeddings respectively for the one or more class embeddings; and
applying the one or more updates respectively to the one or more class embeddings comprises respectively replacing the one or more class embeddings with the one or more replacement class embeddings.
17 . The one or more non-transitory computer-readable media of claim 15 , wherein performing the spreadout regularization on at least the subset of the intermediate class embeddings comprises performing a single unified spreadout regularization on all of the class embeddings contained in the intermediate classification matrix to obtain the updated classification matrix.
18 . The one or more non-transitory computer-readable media of claim 15 , wherein performing the spreadout regularization on at least the subset of the intermediate class embeddings comprises performing, separately for each of one or more class embeddings, the spreadout regularization relative to only a subset of nearest class embeddings surrounding such class embedding.
19 . The one or more non-transitory computer-readable media of claim 18 , wherein performing, separately for each of the one or more class embeddings, the spreadout regularization relative to only the subset of nearest class embeddings surrounding such class embedding comprises performing, separately for each of the class embeddings contained in the intermediate classification matrix, the spreadout regularization relative to only the subset of nearest class embeddings surrounding such class embedding.
20 . The one or more non-transitory computer-readable media of claim 15 , wherein performing the spreadout regularization comprises updating one or more of the class embeddings contained in the classification matrix according to a learning rate and in a direction negative to a gradient of a spreadout regularization function.