Neural network with approximated activation function
A neural network including at least one neural network layer and an activation function connected to an output of the at least one neural network layer. The activation function is implemented as an approximation function of a mathematically defined real valued non-linear activation function, wherein the approximation function allows for integer-only processing of fixed-point representations of input values of the approximation function.
1 . A neural network, comprising:
at least one neural network layer; and
an activation function connected to an output of the at least one neural network layer to provide an entropy model;
wherein the activation function is implemented as an approximation function of a mathematically defined real valued non-linear activation function,
wherein the approximation function allows for integer-only processing of fixed-point representations of input values of the approximation function,
wherein the approximation function comprises a finite number of Taylor series summands which are determined based on expected values of data to be input to the at least one neural network layer or into the approximation function, and an accumulator register size of an accumulator register,
wherein the neural network further comprises the accumulator register for buffering summation results, and
wherein the neural network is configured to:
reconstruct at least a portion of an image by processing a bitstream based on the provided entropy model to obtain a latent tensor representing a component of the image, and processing the latent tensor to obtain a tensor representing the component of the image; or
transform the tensor representing the component of the image into the latent tensor, and processing the latent tensor based on the provided entropy model to generate the bitstream.
2 . The neural network according to claim 1 , wherein the approximation function further comprises at least one of a polynomial function, rational function, a rectified linear activation (ReLU) function, a LeakyReLU function, or a parametric ReLU function.
3 . The neural network according to claim 2 , wherein the approximation function further comprises the polynomial function and a maximum degree of the polynomial function is determined based on at least one of a) the expected values of data to be input to the at least one neural network layer or into the approximation function, b) the accumulator register size of the accumulator register for buffering the summation results, or c) the approximation function, or wherein the approximation function further comprises the rational function and a maximum degrees of polynomials in nominator and denominator of the rational function are determined based on at least one of a) the expected values of data to be input the at least one neural network layer or into the approximation function, b) the accumulator register size of the accumulator register for buffering the summation results, or c) the approximation function.
4 . The neural network according to claim 1 , wherein the mathematically defined real valued non-linear activation function is selected from a group consisting of a Softmax function, a sigmoid function, a hyperbolic tangent function, a Swish function, a Gaussian Error Linear Unit function, or a Scaled Exponential Linear Unit function.
5 . The neural network according to claim 4 , wherein the mathematically defined non-linear activation function is the Softmax function and the approximation function is defined as
f
(
x
)
i
=
Re
LU
(
x
i
)
+
ε
∑
j
=
1
K
(
Re
LU
(
x
j
)
+
ε
)
wherein i denotes an i-th component of an input vector x and a sum runs over all K components of that input vector, and ε denotes a positive constant avoiding division by zero, or wherein the mathematically defined non-linear activation function is the Softmax function and the approximation function is defined as
f
(
x
)
i
=
Re
LU
(
1
+
x
i
)
+
ε
∑
j
=
1
K
(
Re
LU
(
1
+
x
j
)
+
ε
)
wherein i denotes an i-th component of an input vector x and a sum runs over all K components of that input vector, and wherein ε denotes a positive constant avoiding division by zero.
6 . An apparatus for encoding at least a portion of the image, comprising an encoder side of an autoencoder comprising the neural network according to claim 1 , the autoencoder comprising the encoder side and a decoder side, the encoder side being configured to output the bitstream encoding at least the portion of the image.
7 . The apparatus according to claim 6 , wherein the autoencoder is a variational autoencoder, the apparatus comprising one of a) hyperprior of the variational autoencoder comprising the neural network, b) an autoregressive prior of the variational autoencoder comprising the neural network, or c) a combination of the hyperprior and the autoregressive prior of the variational autoencoder at least one of which comprises the neural network.
8 . An apparatus for decoding at least the portion of the image, comprising a decoder side of an autoencoder comprising the neural network according to claim 1 , the autoencoder comprising an encoder side and the decoder side, the decoder side being configured to receive the bitstream encoding at least the portion of the image, reconstruct at least the portion of the image, and output at least the reconstructed portion of the image.
9 . An apparatus for encoding at least the portion of the image, comprising an encoder side of an autoencoder comprising the neural network according to claim 1 , the autoencoder comprising the encoder side and the decoder side, the apparatus further comprising processing circuitry configured for transforming the tensor representing the component of the image into the latent tensor, providing the entropy model using the neural network, and processing the latent tensor based on the provided entropy model to generate the bitstream.
10 . An apparatus for decoding at least the portion of the image, comprising a decoder side of an autoencoder comprising the neural network according to claim 1 , the autoencoder comprising a decoder side and an encoder side, the apparatus further comprising processing circuitry configured for: providing the entropy model using the neural network, processing the bitstream based on the provided entropy model to obtain the latent tensor representing the component of the image, and processing the latent tensor to obtain the tensor representing the component of the image.
11 . The neural network according to claim 1 , wherein s j represents a fractional part of real valued weights, wherein s j of a second scaling factor for a j-th output channel of the at least one neural network layer fulfills a condition defined as:
s
j
<
log
2
[
2
n
-
k
-
1
-
❘
"\[LeftBracketingBar]"
W
j
❘
"\[RightBracketingBar]"
2
-
❘
"\[LeftBracketingBar]"
b
j
❘
"\[RightBracketingBar]"
2
k
-
1
∑
w
i
∈
W
j
❘
"\[LeftBracketingBar]"
w
i
❘
"\[RightBracketingBar]"
]
wherein W j denotes a subset of trainable weights of the at least one neural network layer, |W j | denotes number of elements in subset W j , n denotes a bit size of the accumulator register, k denotes a pre-defined bitdepth of input data and b j denotes a bias value.
12 . The neural network according to claim 11 , wherein s j of the second scaling factor for the j-th output channel of the at least one neural network layer is defined by:
s
j
=
⌊
log
2
[
2
n
-
k
-
1
-
❘
"\[LeftBracketingBar]"
W
j
❘
"\[LeftBracketingBar]"
2
-
❘
"\[LeftBracketingBar]"
b
j
❘
"\[RightBracketingBar]"
2
k
-
1
∑
w
i
∈
W
j
❘
"\[LeftBracketingBar]"
w
i
❘
"\[RightBracketingBar]"
]
⌋
.
⌊
x
⌋
=
floor
(
x
)
13 . A method of operating a neural network comprising at least one neural network layer, the method comprising:
implementing an approximation function of a mathematically defined real valued non-linear activation function as an activation function of the at least one neural network layer, wherein the approximation function allows for integer-only processing of fixed-point representations of input values of the approximation function, and wherein the activation function is connected to an output of the at least one neural network layer to provide an entropy model,
wherein the approximation function comprises a finite Taylor series comprising a finite number of Taylor series summands which are determined based on expected values to be input into the at least one neural network layer or into the approximation function, and an accumulator register size of an accumulator register,
wherein the neural network further comprises the accumulator register for buffering summation results; and
wherein the method further comprises:
reconstructing at least a portion of an image by processing a bitstream based on the provided entropy model to obtain a latent tensor representing a component of the image, and processing the latent tensor to obtain a tensor representing the component of the image, or
transforming the tensor representing the component of the image into the latent tensor, and processing the latent tensor based on the provided entropy model to generate the bitstream.
14 . A method of encoding data, comprising the steps of the method of operating the neural network according to claim 13 and outputting the bistream.
15 . A method of decoding encoded data, comprising the steps of the method of operating the neural network according to claim 13 and outputting the reconstructed portion of the image.
16 . A computer program product comprising a program code stored on a non-transitory medium, wherein the program, when executed on one or more processors, performs the method according to claim 13 .