Generative adversarial neural architecture search
A method and system for neural architectural search (NAS) for performing a task. A generative adversarial network comprising a generator and a discriminator receives, from a user device, a query for neural network architecture, the query including a search space. The generator of the generative adversarial network generates a plurality of generated neural network architectures responsive to the received search space. The discriminator of the generative adversarial network selects an optimal neural network architecture from among the plurality of generated neural network architectures. The optimal generated neural network architecture is transmitted to the user device.
1 . A method for neural architectural search (NAS) for performing a task, the method comprising:
receiving, by a generative adversarial network comprising a generator and a discriminator, from a user device, a query for neural network architecture, the query including a search space, wherein the discriminator of the generative adversarial network is trained by minimizing a cross-entropy loss between a truth class and predicted probabilities generated from candidate architectures from a set of true architectures or a set of fake architectures respectively;
generating, via the generator of the generative adversarial network, a plurality of generated neural network architectures responsive to the received search space;
selecting, via the discriminator of the generative adversarial network, an optimal neural network architecture from among the plurality of generated neural network architectures; and
transmitting the optimal generated neural network architecture to the user device.
2 . The method of claim 1 , further comprising:
training the generative adversarial network by:
generating, by the generator component of the generative adversarial network, generated neural network architectures based on training data,
evaluating, by the discriminator of the generative adversarial network, the generated neural network architectures,
ranking, by the discriminator, the generated neural network architectures, and
adjusting one or more parameters of the generator based on the ranking of the generated neural network architectures.
3 . The method of claim 2 , wherein the training data comprises the set of true architectures.
4 . The method of claim 2 , wherein the discriminator applies a Siamese scheme in which a pair of architectures in which one architecture is from the set of true architectures and the other architecture is from either the set of true architectures or the set of fake architectures comprising of generated neural network architectures.
5 . The method of claim 2 , wherein the generator and the discriminator are trained by playing a two-player minimax game whose corresponding parameters, φ t and θ t respectively, are optimized using:
(
θ
t
,
ϕ
t
)
←
arg
min
θ
t
max
ϕ
t
V
(
G
θ
t
,
D
ϕ
t
)
=
𝔼
x
∼
p
𝒥
[
log
D
ϕ
t
(
x
)
]
+
𝔼
x
∼
p
𝒥
(
x
;
ϕ
t
)
[
log
(
1
-
D
ϕ
t
(
x
)
)
]
where G denotes the generator and D denotes the discriminator,
x is a Directed Acyclic Graph (DAG) connecting one or more operations, and
is a distribution of currently top performing architectures .
6 . The method of claim 1 , wherein the generator comprises an encoder and a decoder.
7 . The method of claim 6 , wherein the encoder is a graph neural network (GNN)-based, auto-regressive architecture encoder comprising a recurrent neural network (RNN) and GNN.
8 . The method of claim 6 , wherein the decoder comprises a multi-layer perceptron (MLP) that outputs an operator probability distribution and a Gated Recurrent Unit (GRU) that recursively determines edge connections to previous operators.
9 . The method of claim 1 , wherein the generator represents each architecture as a Directed Acyclic Graph (DAG) of operators and connections therebetween.
10 . The method of claim 1 , wherein the discriminator comprises a GNN followed by a MLP classifier.
11 . A system for neural architectural search (NAS) using a generative adversarial network, comprising:
a processing device; and
a memory coupled to the processing device, the memory storing computer-executable instructions that, in response to execution by the processing device, cause the system to:
receive, by a generator of the generative adversarial network, from a user device, a query for neural network architecture, the query including a search space; and
generate, via the generator of the generative adversarial network, a plurality of generated neural network architectures responsive to the received search space; and
select, via a discriminator of the generative adversarial network, an optimal neural network architecture from among the plurality of generated neural network architectures;
wherein the optimal generated neural network architecture is transmitted to the user device, and
wherein the discriminator of the generative adversarial network is trained by minimizing a cross-entropy loss between a truth class and predicted probabilities generated from candidate architectures from a set of true architectures or a set of fake architectures respectively.
12 . The system of claim 11 , wherein the generative adversarial network is trained by:
generating, by the generator component of the generative adversarial network, generated neural network architectures based on training data,
evaluating, by the discriminator of the generative adversarial network, the generated neural network architectures,
ranking, by the discriminator, the generated neural network architectures, and
adjusting one or more parameters of the generator based on the ranking of the generated neural network architectures.
13 . The system of claim 12 , wherein the discriminator applies a Siamese scheme in which a pair of architectures in which one architecture is from the set of true architectures and the other architecture is from either the set of true architectures or the set of fake architectures comprising of generated neural network architectures.
14 . The system of claim 11 , wherein the generator and the discriminator are trained by playing a two-player minimax game whose corresponding parameters, φ t and θ t , respectively, are optimized using:
(
θ
t
,
ϕ
t
)
←
arg
min
θ
t
max
ϕ
t
V
(
G
θ
t
,
D
ϕ
t
)
=
𝔼
x
∼
p
𝒥
[
log
D
ϕ
t
(
x
)
]
+
𝔼
x
∼
p
𝒥
(
x
;
ϕ
t
)
[
log
(
1
-
D
ϕ
t
(
x
)
)
]
where G denotes the generator and D denotes the discriminator,
x is a Directed Acyclic Graph (DAG) connecting one or more operations, and
is a distribution of currently top performing architectures .
15 . The system of claim 11 , wherein the generator comprises an encoder and a decoder.
16 . The system of claim 15 , wherein the encoder is a graph neural network (GNN)-based, auto-regressive architecture encoder comprising a recurrent neural network (RNN) and GNN.
17 . The system of claim 11 , wherein the decoder comprises a multi-layer perceptron (MLP) that outputs an operator probability distribution and a Gated Recurrent Unit (GRU) that recursively determines edge connections to previous operators.
18 . The system of claim 11 , wherein the generator represents each architecture as a Directed Acyclic Graph (DAG) of operators and connections therebetween.
19 . The system of claim 11 , wherein the discriminator comprises a GNN followed by a MLP classifier.
20 . A non-transitory machine-readable storage medium having tangibly stored thereon executable instructions for execution by one or more processors that, in response to execution by the one or more processors, cause the one or more processors to:
receive, by a generative adversarial network comprising a generator and a discriminator, from a user device, a query for neural network architecture, the query including a search space, wherein the discriminator of the generative adversarial network is trained by minimizing a cross-entropy loss between a truth class and predicted probabilities generated from candidate architectures from a set of true architectures or a set of fake architectures respectively;
generate, via the generator of the generative adversarial network, a plurality of generated neural network architectures responsive to the received search space;
select, via the discriminator of the generative adversarial network, an optimal neural network architecture from among the plurality of generated neural network architectures; and
transmit the optimal generated neural network architecture to the user device.