Apparatus and method for training neural networks using weight tying
An apparatus comprising circuitry that implements an artificial neural network training algorithm that uses weight tying.
1. An apparatus comprising:
circuitry that implements an artificial neural network training algorithm that uses weight tying, wherein
the circuitry is configured to
compute a weight-tied weight matrix based on an index matrix and based on a value vector;
update the weight tying using a predefined number of iterations of a clustering algorithm to update the index matrix; and
quantize values of the value vector after updating the weight tying, wherein
the circuitry is configured to, in each iteration of the clustering algorithm, update a value vector according to
[
v
(
l
)
]
k
=
1
#
{
I
(
l
)
=
k
}
∑
ij
,
I
(
l
)
=
k
[
W
(
l
)
]
ij
where W (l) is a full-precision weight matrix for layer l of the neural network, I (l) is the index matrix, i and j denote rows and columns, respectively, of W (l) and I (l) , and k denotes a value index.
2. The apparatus of claim 1 , wherein
the predefined number of iterations of the clustering algorithm used to update the weight tying is one.
3. The apparatus of claim 1 , wherein
the circuitry is configured to update, in each iteration of the clustering algorithm, the index matrix according to
[
I
(
l
)
]
ij
=
arg
min
k
=
1
,
…
,
K
(
l
)
❘
"\[LeftBracketingBar]"
[
W
(
l
)
]
ij
-
[
v
(
l
)
]
k
❘
"\[RightBracketingBar]"
where W (l) is a full-precision weight matrix for layer l of the neural network, I (l) is the index matrix, i and j denote rows and columns, respectively, of W (l) and I (l) , and k denotes a value index.
4. The apparatus of claim 1 , wherein
the circuitry is configured to quantize the values of the value vector to the nearest power-of-two.
5. The apparatus of claim 4 , wherein
the circuitry is configured to quantize the values of the value vector according to the quantization scheme:
x
q
=
{
s
·
2
⌊
b
⌋
b
-
⌊
b
⌋
≤
log
2
1.5
s
·
2
⌈
b
⌉
b
-
⌊
b
⌋
>
log
2
1
.
5
where s=sign(x) and h=log 2 |x|, and where x is the value which is to be quantized and x q is the quantized value.
6. The apparatus of claim 1 , wherein
the circuitry is configured to compute the weight-tied weight matrix based on the index matrix and based on the value vector that comprises more than three values.
7. The apparatus of claim 1 , wherein
the circuitry is configured to update full precision weights based on gradients.
8. The apparatus of claim 1 , wherein
the circuitry is configured to compute the gradients based on a cost function and based on the weight-tied weight matrix.
9. The apparatus of claim 7 , wherein
the circuitry is configured to compute the cost function based on a loss function and based on a forward pass function.
10. The apparatus of claim 7 , wherein
the circuitry is configured to compute the gradients based on a backward pass function.
11. The apparatus of claim 1 , wherein
the training algorithm is a stochastic gradient descent training algorithm.
12. The apparatus of claim 1 , wherein
the artificial neural network is a deep convolutional neural network.
13. A method of training an artificial neural network, the method comprising:
performing an artificial neural network training algorithm that uses weight tying;
computing a weight-tied matrix based on an index matrix and based on a value vector;
update the weight tying using a predefined number of iterations of a clustering algorithm to update the index matrix; and
quantizing values of the value vector after updating the weight tying, wherein
updating the weight tying includes, in each iteration of the clustering algorithm, updating a value vector according to
[
v
(
l
)
]
k
=
1
#
{
I
(
l
)
=
k
}
∑
ij
,
I
(
l
)
=
k
[
W
(
l
)
]
ij
where W (l) is a full-precision weight matrix for layer l of the neural network, I (l) is the index matrix, i and j denote rows and columns, respectively, of W (l) and I (l) , and k denotes a value index.
14. A non-transitory computer-readable medium encoded with computer-readable instructions that, when executed by a processor, cause the processor to perform a method comprising:
performing an artificial neural network training algorithm that uses weight tying;
computing a weight-tied matrix based on an index matrix and based on a value vector;
update the weight tying using a predefined number of iterations of a clustering algorithm to update the index matrix; and
quantizing values of the value vector after updating the weight tying, wherein
updating the weight tying includes, in each iteration of the clustering algorithm, updating a value vector according to
[
v
(
l
)
]
k
=
1
#
{
I
(
l
)
=
k
}
∑
ij
,
I
(
l
)
=
k
[
W
(
l
)
]
ij
where W (l) is a full-precision weight matrix for layer l of the neural network, I (l) is the index matrix, i and j denote rows and columns, respectively, of W (l) and I (l) , and k denotes a value index.