Artificial neural networks generated by low discrepancy sequences
Artificial neural networks (ANNs) are computing systems that imitate a human brain by learning to perform tasks by considering examples. These ANNs are typically created by connecting several layers of neural units using connections, where each neural unit is connected to every other neural unit either directly or indirectly to create fully connected layers within the ANN. However, by representing an artificial neural network utilizing paths from an input of the ANN to an output of the ANN, a complexity of the ANN may be reduced, and the ANN may be trained and implemented in a much faster manner when compared to fully connected layers within the ANN. More specifically, the ANN may be trained sparse from scratch in order to avoid a more expensive procedure of training the ANN and compressing it afterwards.
1 . A method to accelerate training of an ANN, the method comprising:
creating the ANN as a plurality of paths,
where each path of the plurality of paths includes:
(a) a plurality of vertices each representing a neural unit within the ANN, and
(b) a plurality of edges each representing a weighted connection within the ANN, and
where each path of the plurality of paths is assigned a number;
for each path of the plurality of paths, initializing all weighted connections within the path with a same sign determined based on the number assigned to the path, where the sign is determined as either a positive sign or a negative sign; and
training the ANN, including for each weighted connection within the ANN:
keeping the sign of the weighted connection fixed during the training, and
training a magnitude of a value of the weighted connection,
where the training of the ANN is accelerated by training only the magnitude of the value of the weighted connection while keeping the sign of the weighted connection fixed.
2 . The method of claim 1 , wherein at least one layer of the ANN is at least one of a fully connected layer and a 1×1 convolution.
3 . The method of claim 1 , wherein the signs of the values of all weighted connections along a first predetermined number of the plurality of paths are positive, and the signs of the values of all weighted connections along a second predetermined number of the plurality of paths are negative.
4 . The method of claim 1 , wherein the ANN is trained utilizing labeled input training data.
5 . The method of claim 4 , wherein the training is semi-supervised.
6 . The method of claim 4 , wherein the training is unsupervised.
7 . The method of claim 1 , further comprising processing input data by the trained ANN to produce output data, the input data including one or more of image data, textual data, audio data, and video data, random numbers, pseudo-random numbers, and quasi-random numbers, and the output data including one or more of a classification, a categorization, a regression, a function approximation, a probability, and samples of data according to a learned distribution.
8 . The method of claim 1 , where each path of the plurality of paths of the ANN extends from an input layer of the ANN to an output layer of the ANN.
9 . The method of claim 1 , further comprising:
implementing the trained ANN; and
processing input data by the trained ANN to produce output data,
wherein during the processing of the input data to produce the output data, the same sign of all weighted connections within a path of the plurality of paths of the trained ANN is not explicitly considered to improve a performance of the trained ANN.
10 . A method to accelerate training of an ANN, the method comprising:
creating the ANN as a plurality of paths each connecting at least one input of the ANN to at least one output of the ANN,
where each of the plurality of paths are generated utilizing a low discrepancy sequence;
initializing a value of all weighted connections in a path of the plurality of paths with a same sign determined from a dimension of the low discrepancy sequence, where the sign is determined as either a positive sign or a negative sign; and
training the ANN, where a sign of a value of each weighted connection is fixed and only the magnitude of the value of each weighted connection is trained:
keeping the sign of the weighted connection fixed during the training, and
training a magnitude of a value of the weighted connection,
where the training of the ANN is accelerated by training only the magnitude of the value of the weighted connection while keeping the sign of the weighted connection fixed.
11 . The method of claim 10 , wherein dimensions of the low discrepancy sequences that cause coalescing links within the ANN are skipped.
12 . The method of claim 10 , wherein the low discrepancy sequence is at least one of a Sobol′, Niederreiter, and Niederreiter-Xing sequence.
13 . The method of claim 10 , wherein the low discrepancy sequence includes a scrambled low discrepancy sequence.
14 . The method of claim 10 , wherein successive dimensions of the low discrepancy sequence form (0,m,s)-nets in base b.
15 . The method of claim 10 , wherein each of the plurality of paths are sampled by a Cascaded Sobol′ sequence.
16 . The method of claim 10 , wherein connections between two layers of the ANN are created by referencing neurons by a corresponding component of the low discrepancy sequence and referencing inputs in their natural order or a permutation of their natural order.
17 . The method of claim 10 , wherein convolutional layers within the ANN are sampled by selecting one of the input channels of the ANN and one of a plurality of convolutional neurons of the ANN specified by an edge of a path.
18 . The method of claim 10 , wherein the ANN is trained utilizing labeled input training data.
19 . The method of claim 18 , wherein the training is semi-supervised.
20 . The method of claim 18 , wherein the training is unsupervised.
21 . The method of claim 10 , further comprising processing input data by the trained ANN to produce output data, the input data including one or more of image data, textual data, audio data, and video data, random numbers, pseudo-random numbers, and quasi-random numbers, and the output data including one or more of a classification, a categorization, a regression, a function approximation, a probability, and samples of data according to a learned distribution.
22 . The method of claim 10 , further comprising:
implementing the trained ANN; and
processing input data by the trained ANN to produce output data,
wherein during the processing of the input data to produce the output data, the same sign of all weighted connections within a path of the plurality of paths of the trained ANN is not explicitly considered to improve a performance of the trained ANN.