Apparatus and method for linearly approximating deep neural network model
Provided are an apparatus and method for linearly approximating a deep neural network (DNN) model which is a non-linear function. In general, a DNN model shows good performance in generation or classification tasks. However, the DNN fundamentally has non-linear characteristics, and therefore it is difficult to interpret how a result from inputs given to a black box model has been derived. To solve this problem, linear approximation of a DNN is proposed. The method for linearly approximating a DNN model includes 1) converting a neuron constituting a DNN into a polynomial, and 2) classifying the obtained polynomial as a polynomial of input signals and a polynomial of weights.
1. A method of linearly approximating a model, which is a non-linear function, of a deep neural network (DNN) executing on a processor and comprising an input layer, a hidden layer and an output layer, the method comprising:
receiving, into the processor, an input to a neuron of the DNN;
expanding, by the processor, the input to the neuron of the DNN into a polynomial, wherein inputs to the neuron of the DNN are x1 and x2, an output of the neuron of the DNN is y, and weights are w1 and w2;
approximating, by the processor, the neuron of the DNN with a Taylor series in parallel with the polynomial expansion of the neuron, to obtain a non-linear activation function
y
=
tanh
(
h
)
≈
h
-
1
3
h
3
,
where h is an output of a hidden layer, and h=w 1 x 1 +w 2 x 2 is obtained as a result of the approximating the neuron of the DNN with the Taylor series;
classifying, by the processor, the polynomially expanded input and the Taylor-series approximated neuron as a polynomial of input signals and a polynomial of weights, and acquiring a polynomial
y
=
(
w
1
x
1
+
w
2
x
2
)
-
1
3
(
w
1
x
1
+
w
2
x
2
)
3
=
w
1
x
1
+
w
2
x
2
+
-
w
1
2
w
2
x
1
2
x
2
-
w
1
w
2
2
x
1
x
2
2
-
1
3
w
1
3
x
1
3
-
1
3
w
2
3
x
2
3
;
and
outputting, by the processor, the linearly approximated DNN model for analysis using a linear system interpretation method.
2. A method of linearly approximating a model, which is a non-linear function, of a deep neural network (DNN) executing on a processor and comprising an input layer, a hidden layer and an output layer, the method comprising:
receiving, into the processor, an input to a neuron of the DNN;
expanding, by the processor, the input to the neuron of the DNN, wherein an input to the neuron of the DNN is x and an output of the neuron of the DNN is y, into a polynomially expanded input p(x);
approximating, by the processor, the neuron of the DNN with a Taylor series in parallel with the polynomial expansion of the neuron;
classifying, by the processor, the polynomially expanded input p(x) and the Taylor-series approximated neuron as a polynomial of input signals and a polynomial of weights, and converting the polynomially expanded input p(x) and the Taylor-series approximated neuron into a form of equation y=a·p(x), where p(x) is an n th -order polynomial of input signals, p(x)=(1,x 1 ,x 2 ,x 1 2 ,x 1 x 2 ,x 2 2 ,x 1 3 ,x 1 2 x 2 ,x 2 2 x 1 ,x 2 3 ), and a weight matrix a is a polynomial of a weight matrix W,
a
=
(
0
,
w
1
,
w
2
,
0
,
0
,
0
,
-
1
3
w
1
3
,
-
w
1
w
2
2
,
-
w
1
2
w
2
,
-
1
3
w
2
3
)
;
and
outputting, by the processor, the linearly approximated DNN model for analysis using a linear system interpretation method.
3. The method of claim 1 , further comprising:
converting the polynomial
y
=
(
w
1
x
1
+
w
2
x
2
)
-
1
3
(
w
1
x
1
+
w
2
x
2
)
3
=
w
1
x
1
+
w
2
x
2
+
-
w
1
2
w
2
x
1
2
x
2
-
w
1
w
2
2
x
1
x
2
2
-
1
3
w
1
3
x
1
3
-
1
3
w
2
3
x
2
3
into a form of equation y=a·p(x), where p(x) is an n th -order polynomial of input signals, p(x)=(1,x 1 ,x 2 ,x 1 2 ,x 1 x 2 ,x 2 2 ,x 1 3 ,x 1 2 x 2 ,x 2 2 x 1 ,x 2 3 ), and a weight matrix a is a polynomial of a weight matrix W,
a
=
(
0
,
w
1
,
w
2
,
0
,
0
,
0
,
-
1
3
w
1
3
,
-
w
1
w
2
2
,
-
w
1
2
w
2
,
-
1
3
w
2
3
)
.
4. An apparatus including a deep neural network (DNN) executing on a processor and comprising an input layer, a hidden layer and an output layer, for linearly approximating a DNN model which is a non-linear function, wherein the apparatus is configured to:
receive, into the processor, an input to a neuron of the DNN;
expand, by the processor, the input to the neuron of the DNN into a polynomial, wherein inputs to the neuron of the DNN are x1 and x2, an output of the neuron of the DNN is y, and weights are w1 and w2;
approximate, by the processor, the neuron of the DNN with a Taylor series in parallel with the polynomial expansion of the neuron, to obtain a non-linear activation function
y
=
tanh
(
h
)
≈
h
-
1
3
h
3
,
where h is an output of a hidden layer and h=w 1 x 1 +w 2 x 2 is obtained as a result of approximating the neuron of the DNN with the Taylor series;
classify, by the processor, the polynomially expanded and the Taylor-series approximated neuron as a polynomial of input signals and a polynomial of weights, and acquire a polynomial
y
=
(
w
1
x
1
+
w
2
x
2
)
-
1
3
(
w
1
x
1
+
w
2
x
2
)
3
=
w
1
x
1
+
w
2
x
2
+
-
w
1
2
w
2
x
1
2
x
2
-
w
1
w
2
2
x
1
x
2
2
-
1
3
w
1
3
x
1
3
-
1
3
w
2
3
x
2
3
;
and
output, by the processor, the linearly approximated DNN model for analysis using a linear system.
5. An apparatus including a deep neural network executing on a processor and comprising an input laver, a hidden layer and an output layer, for linearly approximating a DNN model which is a non-linear function, wherein the apparatus is configured to:
receive, into the processor, an input to a neuron of the DNN;
expand, by the processor, the input to the neuron of the DNN, wherein an input to the neuron of the DNN is x and an output of the neuron of the DNN is y, into a polynomially expanded input p(x);
approximate, by the processor, the neuron of the DNN with a Taylor series in parallel with the polynomial expansion of the neuron;
classify, by the processor, the polynomially expanded input p(x) and the Taylor-series approximated neuron as a polynomial of input signals and a polynomial of weights, and convert the polynomially expanded input p(x) and the Taylor-series approximated neuron into a form of equation y=a·p(x), where p(x) is an n th -order polynomial of input signals, p(x)=(1,x 1 ,x 2 ,x 1 2 ,x 1 x 2 ,x 2 2 ,x 1 3 ,x 1 2 x 2 ,x 2 2 x 1 ,x 2 3 ), and a weight matrix a is a polynomial of a weight matrix W,
a
=
(
0
,
w
1
,
w
2
,
0
,
0
,
0
,
-
1
3
w
1
3
,
-
w
1
w
2
2
,
-
w
1
2
w
2
,
-
1
3
w
2
3
)
;
and
output, by the processor, the linearly approximated DNN model for analysis using a linear system.
6. The apparatus of claim 4 , wherein the apparatus is further configured to convert the polynomial
y
=
(
w
1
x
1
+
w
2
x
2
)
-
1
3
(
w
1
x
1
+
w
2
x
2
)
3
=
w
1
x
1
+
w
2
x
2
+
-
w
1
2
w
2
x
1
2
x
2
-
w
1
w
2
2
x
1
x
2
2
-
1
3
w
1
3
x
1
3
-
1
3
w
2
3
x
2
3
into a form of equation y=a·p(x),
where p(x) is an n th -order polynomial of input signals, p(x)=(1,x 1 ,x 2 ,x 1 2 ,x 1 x 2 ,x 2 2 ,x 1 3 ,x 1 2 x 2 ,x 2 2 x 1 ,x 2 3 ), and a weight matrix a is a polynomial of a weight matrix W,
a
=
(
0
,
w
1
,
w
2
,
0
,
0
,
0
,
-
1
3
w
1
3
,
-
w
1
w
2
2
,
-
w
1
2
w
2
,
-
1
3
w
2
3
)
.