Optimizing capacity and learning of weighted real-valued logic
Maximum expressivity can be received representing a ratio between maximum and minimum input weights to a neuron of a neural network implementing a weighted real-valued logic gate. Operator arity can be received associated with the neuron. Logical constraints associated with the weighted real-valued logic gate can be determined in terms of weights associated with inputs to the neuron, a threshold-of-truth, and a neuron threshold for activation. The threshold-of-truth can be determined as a parameter used in an activation function of the neuron, based on solving an activation optimization formulated based on the logical constraints, the activation optimization maximizing a product of expressivity representing a distribution width of input weights to the neuron and gradient quality for the neuron given the operator arity and the maximum expressivity. The neural network of logical neurons can be trained using the activation function at the neuron, the activation function using the determined threshold-of-truth.
1. A method of learning of weighted real-valued logic implemented in a neural network of logical neurons, comprising:
receiving maximum expressivity representing a ratio between maximum and minimum input weights to a neuron of a neural network implementing a weighted real-valued logic gate;
receiving operator arity associated with the neuron;
defining logical constraints associated with the weighted real-valued logic gate in terms of weights associated with inputs to the neuron, a threshold-of-truth, and a neuron threshold for activation;
determining the threshold-of-truth as a parameter used in an activation function of the neuron, based on solving an activation optimization formulated based on the logical constraints, the activation optimization maximizing a product of expressivity representing a distribution width of input weights to the neuron and gradient quality for the neuron given the operator arity and the maximum expressivity; and
training the neural network of logical neurons using the activation function at the neuron, the activation function using the determined threshold-of-truth.
2. The method of claim 1 , wherein the activation function includes a sigmoid function.
3. The method of claim 1 , wherein the activation function includes a 3-piece leaky rectified linear unit (ReLU).
4. The method of claim 1 , further including, based on the defined logical constraints, defining a lower bound on the neuron threshold;
defining an upper bound on the neuron threshold;
defining a lower bound on the neuron's input representing a minimum point of an operating range of the neuron;
defining a lower bound on minimum input weight in terms of the expressivity, operator arity and the threshold-of-truth;
defining a logical bandwidth representing a width of the operating range of the activation function of the neuron;
defining an upper bound on the expressivity,
wherein the activation optimization is formulated based on the lower bound on the neuron threshold, the upper bound on the neuron threshold, the lower bound on the neuron's input, the lower bound on minimum input weight, the logical bandwidth and the upper bound on the expressivity.
5. The method of claim 4 , wherein the activation optimization is derived in terms of the expressivity, the threshold-of-truth, the lower bound on the neuron's input and the upper bound of the maximum input expected for the neuron activation.
6. The method of claim 1 , further including continually updating the activation function for the threshold-of-truth value to optimize gradients of the activation function over a logical operating range by forming a piecewise linear function that is false at an operating range minimum, at threshold-of-falsity for a threshold-of-falsity input, at threshold-of-truth for a threshold-of-truth input, and true at the operating range maximum.
7. The method of claim 1 , further including introducing a slack variable associated with the neuron's input to allow the neuron's input to be removed without requiring higher expressivity.
8. The method of claim 1 , wherein sum of the input weights to the neuron is determined given only a minimum weight and the expressivity.
9. The method of claim 1 , wherein the threshold-of-truth is updated based on a current expressivity.
10. The method of claim 1 , further including visualizing properties of the weighted real-valued logic gate.
11. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:
receive maximum expressivity representing a ratio between maximum and minimum input weights to a neuron of a neural network implementing a weighted real-valued logic gate;
receive operator arity associated with the neuron;
define logical constraints associated with the weighted real-valued logic gate in terms of weights associated with inputs to the neuron, a threshold-of-truth, and a neuron threshold for activation;
determine the threshold-of-truth as a parameter used in an activation function of the neuron, based on solving an activation optimization formulated based on the logical constraints, the activation optimization maximizing a product of expressivity representing a distribution width of input weights to the neuron and gradient quality for the neuron given the operator arity and the maximum expressivity; and
train the neural network of logical neurons using the activation function at the neuron, the activation function using the determined threshold-of-truth.
12. The computer program product of claim 11 , wherein the activation function includes a sigmoid function.
13. The computer program product of claim 11 , wherein the activation function includes a 3-piece leaky rectified linear unit (ReLU).
14. The computer program product of claim 11 , wherein the device is caused to, based on the defined logical constraints,
define a lower bound on the neuron threshold;
define an upper bound on the neuron threshold;
define a lower bound on the neuron's input representing a minimum point of an operating range of the neuron;
define a lower bound on minimum input weight in terms of the expressivity, operator arity and the threshold-of-truth;
defining a logical bandwidth representing a width of the operating range of the activation function of the neuron;
define an upper bound on the expressivity,
wherein the activation optimization is formulated based on the lower bound on the neuron threshold, the upper bound on the neuron threshold, the lower bound on the neuron's input, the lower bound on minimum input weight, the logical bandwidth and the upper bound on the expressivity.
15. The computer program product of claim 14 , wherein the activation optimization is derived in terms of the expressivity, the threshold-of-truth, the lower bound on the neuron's input and the upper bound of the maximum input expected for the neuron activation.
16. The computer program product of claim 11 , wherein the device is further caused to continually update the activation function for the threshold-of-truth value to optimize gradients of the activation function over a logical operating range by forming a piecewise linear function that is false at an operating range minimum, at threshold-of-falsity for a threshold-of-falsity input, at threshold-of-truth for a threshold-of-truth input, and true at the operating range maximum.
17. The computer program product of claim 11 , wherein the device is further caused to introduce a slack variable associated with the neuron's input to allow the neuron's input to be removed without requiring higher expressivity.
18. The computer program product of claim 11 , wherein sum of the input weights to the neuron is determined given only a minimum weight and the expressivity.
19. The computer program product of claim 11 , wherein the threshold-of-truth is updated based on a current expressivity.
20. A system comprising:
a hardware processor; and
a memory device coupled with the hardware processor,
the hardware processor configured to at least:
receive maximum expressivity representing a ratio between maximum and minimum input weights to a neuron of a neural network implementing a weighted real-valued logic gate;
receive operator arity associated with the neuron;
define logical constraints associated with the weighted real-valued logic gate in terms of weights associated with inputs to the neuron, a threshold-of-truth, and a neuron threshold for activation;
determine the threshold-of-truth as a parameter used in an activation function of the neuron, based on solving an activation optimization formulated based on the logical constraints, the activation optimization maximizing a product of expressivity representing a distribution width of input weights to the neuron and gradient quality for the neuron given the operator arity and the maximum expressivity; and
train the neural network of logical neurons using the activation function at the neuron, the activation function using the determined threshold-of-truth.