Architecture-agnostic federated learning system

1 . A method for training a hyper network (HN), the method comprising:

sending, by a server to each of a plurality of client devices, a set of parameters of the HN;

receiving, by the server from each of one or more respective client devices of the plurality of client devices, a respective set of locally updated HN parameters; and

generating, by the server and based on the one or more respective sets of locally updated HN parameters received from the one or more respective client devices, a globally updated set of parameters of the HN,

wherein each of the one or more respective client devices generates a respective set of locally updated HN parameters by training a respective local copy of the HN based on a respective local neural network architecture.

2 . The method of claim 1 , wherein the server receives a respective set of locally updated HN parameters from every client device of the plurality of client devices to which the server sent the set of parameters of the HN.

3 . The method of claim 1 , wherein the server receives a first set of locally updated HN parameters from a first client device and a second set of locally updated HN parameters from a second client device.

4 . The method of claim 3 , wherein the first client device generates the first set of locally updated HN parameters based on a first local neural network architecture of the first client device and using first local training data, and

wherein the second client device generates the second set of locally updated HN parameters based on a second neural network architecture of the second client device and using second local training data.

5 . The method of claim 1 , wherein each respective local neural network architecture is represented by a respective descriptor, and wherein the HN processes the respective descriptor to output a respective set of local neural network architecture weights conditioned to the respective local neural network architecture.

6 . The method of claim 5 , wherein each of the one or more respective client devices generates the respective set of locally updated HN parameters by:

training the respective local neural network architecture to provide a set of updated local neural network architecture weights, and

computing the respective set of locally updated HN parameters based on differences between the set of updated local neural network architecture weights and the respective set of local neural network architecture weights output by the HN.

7 . The method of claim 6 , wherein each of the one or more of the respective client devices determines the set of updated local neural network architecture weights using respective local data.

8 . The method of claim 7 , wherein the training the respective local neural network architecture to provide the set of updated local neural network architecture weights comprises performing a predefined number of iterations of a local optimization.

9 . The method of claim 5 , wherein the HN comprises a first sub-network and a second sub-network, wherein the first sub-network processes the respective descriptor in a first stage, and wherein the second sub-network processes output of the first stage to generate the respective set of local neural network architecture weights in a second stage.

10 . The method of claim 9 , wherein the first sub-network is a neural network associated with a first subset of parameters among the set of parameters of the HN, and the second sub-network is a set of multilayer perceptrons (MLPs) associated with a second subset of parameters among the set of parameters of the HN.

11 . The method of claim 5 , wherein the HN is a graph hyper network (GHN) and each respective descriptor by which a respective local neural network architecture is represented comprises a graph that includes a plurality of nodes and at least one directed edge.

12 . The method of claim 1 , wherein the generating, by the server, the globally updated set of parameters of the HN comprises:

averaging, by the server, the one or more respective sets of locally updated HN parameters received from the one or more respective client devices,

wherein the server performs per-element averaging among the one or more respective sets of locally updated HN parameters.

13 . The method of claim 1 , further comprising:

sending, by the server, the globally updated set of parameters of the HN to the one or more respective client devices of the plurality of client devices for a next round of communication.

14 . The method of claim 1 , wherein one of the one or more respective client devices has two or more instances running thereon, each respective instance associated with a local neural network architecture of the one client device, and wherein the local neural network architectures associated with the two or more instances are the same or different.

15 . The method of claim 9 , wherein the HN is a graph hyper network (GHN),

wherein each respective descriptor by which a respective local neural network architecture is represented comprises a graph that includes a plurality of nodes and at least one directed edge,

wherein the HN processes the respective descriptor to output, in the first stage, a set of node features, each node feature corresponding to a parametric layer in the respective local neural network architecture, and to output, in the second stage, a subset of local neural network architecture weights for each parametric layer.

16 . A method for training a hyper network (HN), the method comprising:

receiving, by a client device, a set of parameters of a hyper network (HN) from a server;

generating, by the client device, a set of locally updated HN parameters by training a local copy of the HN based on a local neural network architecture; and

sending, by the client device, the set of locally updated HN parameters to the server,

wherein the server generates a globally updated set of parameters of the HN based on the set of locally updated HN parameters.

17 . The method of claim 16 , wherein the local neural network architecture of the client device is represented by a descriptor, and wherein the local copy of the HN processes the descriptor to output a set of local neural network architecture weights conditioned to the respective local neural network architecture.

18 . The method of claim 17 , wherein the determining, by the client device, the set of locally updated HN parameters based on the local neural network architecture comprises:

generating, by the client device, the set of locally updated HN parameters using local data and the set of parameters of the HN received from the server.

19 . The method of claim 18 , wherein the generating, by the client device, the set of locally updated HN parameters using the local data and the set of parameters of the HN received from the server comprises:

training the local neural network architecture using the local data to provide a set of updated local neural network architecture weights;

and

computing the set of locally updated HN parameters based on differences between the set of updated local neural network architecture weights and the set of local neural network architecture weights output by the local copy of the HN.

20 . The method of claim 19 , wherein the training the local neural network architecture using the local data to provide the set of updated local neural network architecture weights comprises performing a predefined number of iterations of a local optimization.

21 . The method of claim 17 , wherein the local copy of the HN comprises a first sub-network and a second sub-network, wherein the first sub-network processes the descriptor of the local neural network architecture in a first stage, and wherein the second sub-network processes output of the first stage to generate the set of local neural network architecture weights in a second stage.

22 . The method of claim 21 , wherein the first sub-network is a neural network associated with a first subset of parameters among the set of parameters of the HN, and the second sub-network is a set of multilayer perceptrons (MLPs) associated with a second subset of parameters among the set of parameters of the HN.

23 . The method of claim 16 , wherein the server generates the globally updated set of parameters of the HN based on the set of locally updated HN parameters and one or more additional sets of locally updated HN parameters by performing per-element averaging among the set of locally updated HN parameters and the one or more additional sets of locally updated HN parameters, and the method further comprises:

receiving, by the client device, the globally updated set of parameters of the HN for a next round of communication.

24 . A system for training a hyper network (HN), the system comprising:

a server; and

a plurality of client devices;

wherein the server is configured to facilitate:

sending a set of parameters of the HN to each of the plurality of client devices;

receiving, from each of one or more respective client devices of the plurality of client devices, a respective set of locally updated HN parameters; and

generating, based on the one or more respective sets of locally updated HN parameters, a globally updated set of parameters of the HN, and

wherein each respective client device in the at least one of the plurality of client devices is configured to facilitate:

receiving, from the server, the set of parameters of the HN;

generating the respective set of locally updated HN parameters based on a local neural network architecture of the respective client device; and

sending, to the server, the respective set of locally updated HN parameters.

25 . The system of claim 24 , wherein each respective client device is further configured to facilitate determining the respective set of locally updated HN parameters based on the local neural network architecture of the respective client device by:

generating a set of local neural network architecture weights conditioned to the respective local neural network architecture of the respective client device by processing, using the HN, a descriptor representing the local neural network architecture; and

training the local neural network architecture using local data to provide a set of updated local neural network architecture weights,

computing the respective set of locally updated HN parameters based on differences between the set of updated local neural network architecture weights and the set of local neural network architecture weights conditioned to the respective local neural network.