Systems and methods for normalization in deep learning
A method of training a neural network may be provided. The method may include receiving a raw data set and normalizing the raw data set using a group normalization layer to generate a training data set for training a neural network, wherein the group normalization layer segments the raw data set into a plurality of groups of a predetermined size. The method may also include randomly initializing the neural network by randomly assigning values to a plurality of weights corresponding to a plurality of layers of the neural network. The method may further include training the neural network using the training data set.
1 . A method of training a neural network, the method comprising:
receiving a raw data set;
normalizing the raw data set using a group normalization layer to generate a training data set for training a neural network, wherein the group normalization layer segments the raw data set into a plurality of groups of a predetermined size;
randomly initializing the neural network by randomly assigning values to a plurality of weights corresponding to a plurality of layers of the neural network; and
training the neural network using the training data set.
2 . The method of claim 1 , wherein the group normalization layer comprises an activation based layer.
3 . The method of claim 1 , wherein the group normalization layer comprises a parametric layer.
4 . The method of claim 1 , further comprising:
causing, by the group normalization layer, a linear growth of activation variance during a forward propagation within the neural network.
5 . The method of claim 1 , further comprising:
causing, by a non-linearity in a residual path of the neural network, a linear growth of activation variance during a forward propagation within the neural network.
6 . The method of claim 1 , further comprising:
preventing, by the group normalization layer, a rank collapse within at least one internal layer of the neural network.
7 . The method of claim 1 , further comprising:
avoiding, by the group normalization layer, gradient explosion during a backward propagation within the neural network.
8 . The method of claim 1 , further comprising:
determining the size based on training speed and training stability of the neural network.
9 . The method of claim 1 , wherein the neural network comprises a deep neural network.
10 . The method of claim 1 , wherein the neural network comprises a convolutional neural network.
11 . A system comprising:
a non-transitory storage medium storing computer program instructions; and
one or more processors configured to execute the computer program instructions to cause operations comprising:
receiving a raw data set;
normalizing the raw data set using a group normalization layer to generate a training data set for training a neural network, wherein the group normalization layer segments the raw data set into a plurality of groups of a predetermined size;
randomly initializing the neural network by randomly assigning values to a plurality of weights corresponding to a plurality of layers of the neural network; and
training the neural network using the training data set.
12 . The system of claim 11 , wherein the group normalization layer comprises an activation based layer.
13 . The system of claim 11 , wherein the group normalization layer comprises a parametric layer.
14 . The system of claim 11 , wherein the operations further comprise:
causing, by the group normalization layer, a linear growth of activation variance during a forward propagation within the neural network.
15 . The system of claim 11 , wherein the operations further comprise:
causing, by a non-linearity in a residual path of the neural network, a linear growth of activation variance during a forward propagation within the neural network.
16 . The system of claim 11 , wherein the operations further comprise:
preventing, by the group normalization layer, a rank collapse within at least one internal layer of the neural network.
17 . The system of claim 11 , wherein the operations further comprise:
avoiding, by the group normalization layer, gradient explosion during a backward propagation within the neural network.
18 . The system of claim 11 , wherein the operations further comprise:
determining the size based on training speed and training stability of the neural network.
19 . The system of claim 11 , wherein the neural network comprises a deep neural network.
20 . The system of claim 11 , wherein the neural network comprises a convolutional neural network.