ROBUST ANTI-ADVERSARIAL MACHINE LEARNING
Systems and methods to improve the robustness of a network that has been trained to convergence, particularly with respect to small or imperceptible changes to the input data. Various techniques, which can be utilized either individually or in various combinations, can include adding biases to the input nodes of the network, increasing the minibatch size of the training data, adding special nodes to the network that have activations that do not necessarily change with each data example of the training data, splitting the training data based upon the gradient direction, and making other intentionally adversarial changes to the input of the neural network. In more robust networks, a correct classification is less likely to be disturbed by random or even intentionally adversarial changes in the input values.
1 - 9 . (canceled)
10 . A method for increasing a robustness of a neural network comprising an input layer, a hidden layer, and an output layer, the method comprising:
increasing a minibatch size of a training data set for training the neural network.
11 . The method claim 10 , further comprising:
changing a hyperparameter controlling an activation function of a node to cause the activation function to tend to converge such that the activation function that is more robust against incremental changes in an input to the node.
12 . The method of claim 11 , further comprising:
adding a special node to the neural network, the special node comprising a non-monotonic activation function.
13 . The method of claim 12 , further comprising:
implementing a softmax gate to select which of a plurality of values should be passed through to a higher level node.
14 . The method of claim 13 , further comprising:
adding a special node trained by one-shot learning.
15 . The method of claim 14 , further comprising:
applying a transformation to the input to make the neural network more robust against adversarial changes.
16 . The method of claim 10 , wherein the minibatch size is increased until the minibatch size is equal to a size of the training data set.
17 . The method of claim 16 , wherein the minibatch size is increased to the size of the training data set over a plurality of iterations.
18 . The method of claim 16 , wherein the minibatch size is increased to the size of the training data set over a single iteration.
19 . The method of claim 10 , further comprising:
utilizing a fixed minibatch size during a normal learning period during training of the neural network;
determining whether the training of the neural network is approaching a stationary point; and
increasing the minibatch size as the training of the neural network approaches a stationary point.
20 . The method of claim 10 , further comprising:
determining whether training of the neural network is in a monotonic learning phase; and
increasing the minibatch size according to whether the training is in the monotonic learning phase.
21 . The method of claim 20 , wherein the minibatch size is increased to a size of the training data set.
22 . The method of claim 10 , wherein the method is executed by a learning coach controlling the neural network.
23 . A method for increasing a robustness of a neural network comprising an input layer, a hidden layer, and an output layer, the method comprising:
changing a hyperparameter controlling an activation function of a node to cause the activation function to tend to converge such that the activation function is more robust against incremental changes in an input to the node.
24 - 29 . (canceled)
30 . The method of claim 23 , wherein the hyperparameter controlling the activation function of the node controls a slope of an asymptote to the activation function.
31 - 38 . (canceled)
39 . A method for increasing a robustness of a neural network comprising an input layer, a hidden layer, and an output layer, the method comprising:
adding a special node to the neural network, the special node comprising a non-monotonic activation function.
40 - 43 . (canceled)
44 . The method of claim 39 , wherein the special node is a member of a set of nodes programmed to function as a softmax gate for a set of nodes of the neural network.
45 . The method of claim 39 , wherein the special node comprises a template node programmed such that a derivative of its activation is close to zero relative to changes in data that are far from a template value of the template node.
46 . The method of claim 39 , wherein the special node is programmed to function as a Gaussian mixture distribution model.
47 - 50 . (canceled)
51 . A method for increasing a robustness of a neural network comprising an input layer, a hidden layer, and an output layer, the method comprising:
implementing a softmax gate to select which of a plurality of input values should be passed through to a higher level node of the neural network.
52 - 53 . (canceled)
54 . The method of claim 51 , wherein the softmax gate comprises a first set of nodes of the neural network whose joint set of activations represent a set of softmax values and wherein the set of softmax values are utilized to gate output values of a second set of nodes of the neural network to the higher level node of the neural network.
55 - 56 . (canceled)
57 . A method for increasing a robustness of a neural network comprising an input layer, a hidden layer, and an output layer, the method comprising:
adding a special node trained by one-shot learning.
58 - 78 . (canceled)
79 . A computer system for increasing a robustness of a neural network comprising an input layer, a hidden layer, and an output layer, the computer system comprising:
one or more processor cores; and
a memory coupled to the one or more processor cores, the memory storing instructions that, when executed by the one or more processor cores, cause the computer system to:
increase a minibatch size of a training data set for training the neural network.
80 - 148 . (canceled)