Integer gate logic artificial neural network
Apparatus and method for implementing an Artificial Neural Network (ANN) section which eliminates the need for backpropagation during training. The ANN section has a plurality of integer gate logic (IGL) nodes respectively arranged into an input layer, an output layer, and at least one hidden layer. Each node has multiple inputs and a single output, and uses a non-differentiable activation function to emulate Boolean logic functions (including XOR), near-Boolean functions, and unknown functions, based on one or more selectable weight values. A chain isolation optimization process is used to select and isolate each node during training to assess the impact of the different weight parameters on the output. Enhanced error functions, batch processing scheduling, and random node selection techniques can be used during training. The nodes can be singly connected, or arranged into convolutional filters or localized fully interconnected layers.
1. An apparatus comprising a computer circuit having a memory in which an Artificial Neural Network (ANN) section is stored, the ANN section comprising a plurality of nodes respectively arranged into an input layer, an output layer, and at least one hidden layer interconnected between the respective input and output layers, each node in the ANN section having multiple inputs, a single output, and a non-differentiable activation function configured to emulate one or more Boolean logic functions responsive to a magnitude of the multiple inputs and at least one weight value,
wherein a selected node from the plurality of nodes in the input layer and the at least one hidden layer has the single output thereof connected to a total of one other node in a downstream layer to facilitate training of the at least one weight value of the selected node using a chain isolation optimization process without backpropagation,
wherein a total of one active chain path extends through the ANN from the selected node to each node in the output layer, and
wherein the chain isolation optimization process comprises adjusting the at least one weight value of the selected node and recalculating output values for each downstream node along the active chain path to determine a change in an output error from the output layer.
2. The apparatus of claim 1 , wherein each non-input layer node has a total of two inputs, each of the two inputs connected to a total of one other node in an upstream layer.
3. The apparatus of claim 1 , wherein the selected node is a first selected node in a selected layer having an overall population of nodes, wherein the first selected node is randomly selected from among the overall population of nodes for training using a first pass of the chain isolation optimization process to arrive at a first set of adjusted parameters for the first selected node that provides a first reduced output error value, and wherein a different, second selected node in the selected layer is subsequently randomly selected from among the overall population of nodes for training using a second pass of the chain isolation optimization process to arrive at a second set of adjusted parameters for the second node that provides a second reduced output error value lower than the first reduced output error value.
4. The apparatus of claim 3 , wherein a subset of selected nodes from the selected layer are sequentially randomly selected for training in turn to provide a final reduced output error value for the ANN section, the subset of selected nodes constituting 5% or less of the overall population of nodes in the selected layer.
5. The apparatus of claim 1 , wherein the single output of each node is characterized as Y where Y ranges from a minimum value 0 to a maximum value P where P is a positive integer, and the non-differentiable activation function operates upon a weighted sum (WS) of the multiple inputs and the at least one weight value to generate Y responsive to the following relations:
if WS is less than zero (0), then Y is zero;
if WS is between 0 and P, Y is equal to WS;
if WS is between P and 2P, Y is determined in relation to a difference between WS and P;
if WS is between 2P and 3P, Y is determined in relation to a difference between WS and 2P; and
if WS is greater than 3P, Y is equal to P.
6. The apparatus of claim 1 , wherein each selected node combines input values from the multiple inputs using weight values for each input value, a bias value and a global precision value P selected in relation to a desired precision between a minimum value and a maximum value to generate a weighted sum (WS), and wherein the non-differentiable activation function operates upon the WS to generate an associated output Y by the selected node where Y is a positive integer value from 0 to P.
7. The apparatus of claim 1 , wherein the ANN section further comprises a convolutional filter layer disposed between the input layer and a selected one of the at least one hidden layers.
8. The apparatus of claim 1 , wherein the ANN section further comprises a fully interconnected layer (FIL) disposed between the input layer and a selected one of the at least one hidden layers.
9. The apparatus of claim 1 , wherein the non-differentiable activation function is configured to model each of the following Boolean logic functions responsive to different magnitudes of the at least one weight value: NOR, XA, XB, AND, NOTB, XOR, B, NOTA, A, NXOR, NAND, OR, NXA, NXB, NULL, and ALL.
10. The apparatus of claim 1 , wherein each selected node combines input values from the multiple inputs using weight values for each input value, a bias value and a global precision value selected in relation to a desired precision between a minimum value and a maximum value of the single output for the selected node.
11. The apparatus of claim 10 , wherein the desired precision value is characterized as a positive integer P, each of the input values ranges in magnitude from 0 to P, each of the weight values range in magnitude from −2P to +2P, the bias value ranges in magnitude from −1P to +3P, and the output of the selected node ranges in magnitude from 0 to P.
12. The apparatus of claim 1 , wherein the chain isolation optimization process comprises identifying a selected node, detecting changes in an output error at an output node coupled to the selected node responsive to each of a different combination of parametric values applied to the selected node, and calculating updated output values for each of a sequence of downstream nodes coupled along a chain path from the selected node to the output node using the output from the selected node and previously stored outputs from other nodes in the ANN section.
13. The apparatus of claim 1 , wherein the computer circuit comprises at least one programmable processor, the memory is a computer storage memory utilized by the at least one programmable processor, and the ANN section is realized in software in the computer storage memory.
14. The apparatus of claim 1 , wherein the computer circuit comprises one or more application specific integrated circuit (ASIC) devices.
15. A method comprising:
configuring an Artificial Neural Network (ANN) section as a plurality of nodes respectively arranged into an input layer, an output layer, and at least one hidden layer interconnected between the respective input and output layers, each node in the ANN section having multiple inputs, a single output, and a non-differentiable activation function configured to emulate one or more Boolean logic functions responsive to a magnitude of the multiple inputs and a set of parametric values; and
training the ANN section using a chain isolation optimization process comprising identifying a selected node, detecting changes in an output error at an output node coupled to the selected node responsive to each of a different combination of the parametric values applied to the selected node, and calculating updated output values for each of a sequence of downstream nodes coupled along a chain path from the selected node to the output node using the output from the selected node and previously stored outputs from other nodes in the ANN section,
wherein a total of one active chain path extends through the ANN from each selected node to each node in the output laver, and wherein the chain isolation optimization process further comprises adjusting the at least one weight value of the selected node and recalculating output values for each downstream node along the active chain path to determine a change in an output error from the output layer.
16. The method of claim 15 , wherein the training step comprises successively applying a batch of training data examples to the input layer and determining an output error from the output layer for each of the training data examples responsive to a difference between a desired output and an actual output, the output error determined using cached values from each of the other nodes in the ANN section.
17. The method of claim 15 , wherein the training step further comprises a prior pruning operation upon nodes in the input layer for locations associated with null locations across the training data examples.
18. The method of claim 15 , wherein each non-input layer node has a total of two inputs, each of the two inputs connected to a total of one other node in an upstream layer.
19. The method of claim 15 , wherein the selected node is a first selected node in a selected layer having an overall population of nodes, wherein the first selected node is randomly selected from among the overall population of nodes for training using a first pass of the chain isolation optimization process to arrive at a first set of adjusted parameters for the first selected node that provides a first reduced output error value, and wherein a different, second selected node in the selected layer is subsequently randomly selected from among the overall population of nodes for training using a second pass of the chain isolation optimization process to arrive at a second set of adjusted parameters for the second node that provides a second reduced output error value lower than the first reduced output error value.
20. The method of claim 15 , wherein the non-differentiable activation function has at least one local minimum disposed between an adjacent pair of local maximums.
21. The method of claim 15 , wherein each selected node combines input values from the multiple inputs using weight values for each input value, a bias value and a global precision value P selected in relation to a desired precision between a minimum value and a maximum value to generate a weighted sum (WS), and wherein the non-differentiable activation function operates upon the WS to generate an associated output Y by the selected node, where Y is a positive integer in a range of from 0 to P.
22. The method of claim 21 , wherein the training step comprises generating an output error value for the ANN section in relation to a difference between a predicted output value and a desired output value using an error forcing function that amplifies the output error value responsive to a magnitude of the difference exceeding a predetermined threshold.
23. The method of claim 15 , wherein the set of parametric values for each node are randomized prior to the training step.
24. The method of claim 15 , wherein the training step further comprises flagging examples in the test data set that exhibit misclassification error above a selected threshold during a first pass, and advances both occurrence and frequency of the flagged test values during a subsequent second pass.
25. A method comprising:
configuring an Artificial Neural Network (ANN) section as a plurality of nodes respectively arranged into an input layer, an output layer, and at least one hidden layer interconnected between the respective input and output layers, each node in the ANN section having multiple inputs, a single output, and a non-differentiable activation function configured to emulate one or more Boolean logic functions responsive to a magnitude of the multiple inputs and a set of parametric values; and
training the ANN section using a chain isolation optimization process comprising identifying a selected node, detecting changes in an output error at an output node coupled to the selected node responsive to each of a different combination of the parametric values applied to the selected node, and calculating updated output values for each of a sequence of downstream nodes coupled along a chain path from the selected node to the output node using the output from the selected node and previously stored outputs from other nodes in the ANN section;
wherein each selected node combines input values from the multiple inputs using weight values for each input value, a bias value and a global precision value selected in relation to a desired precision between a minimum value and a maximum value to generate a weighted sum (WS), and wherein the non-differentiable activation function operates upon the WS to generate an associated output by the selected node; and
wherein the global precision value is characterized as a positive integer P, each of the input values ranges in magnitude from 0 to P, each of the weight values range in magnitude from −2P to +2P, the bias value ranges in magnitude from −1P to +3P, and the output of the selected node ranges in magnitude from 0 to P.
26. The method of claim 25 , wherein the training step comprises generating an output error value for the ANN section in relation to a difference between a predicted output value and a desired output value using an error forcing function that amplifies the output error value responsive to a magnitude of the difference exceeding a predetermined threshold.
27. An apparatus comprising a computer circuit having a memory in which an Artificial Neural Network (ANN) section is stored, the ANN section comprising a plurality of nodes respectively arranged into an input layer, an output layer, and at least one hidden layer interconnected between the respective input and output layers, each node in the ANN section having multiple inputs, a single output, and a non-differentiable activation function configured to emulate one or more Boolean logic functions responsive to a magnitude of the multiple inputs and at least one weight value,
wherein a selected node from the plurality of nodes in the input layer and the at least one hidden layer has the single output thereof connected to a total of one other node in a downstream layer to facilitate training of the at least one weight value of the selected node using a chain isolation optimization process without backpropagation,
wherein the chain isolation optimization process comprises detecting changes in an output error at an output node coupled to the selected node responsive to each of a different combination of parametric values applied to the selected node, and calculating updated output values for each of a sequence of downstream nodes coupled along a chain path from the selected node to the output node using the output from the selected node and previously stored outputs from other nodes in the ANN section,
wherein a total of one active chain path extends through the ANN from the selected node to each node in the output layer, and
wherein the chain isolation optimization process comprises adjusting the at least one weight value of the selected node and recalculating output values for each downstream node along the active chain path to determine a change in an output error from the output layer.
28. The apparatus of claim 27 , wherein the ANN section forms a portion of a large language model (LLM), a generative AI system, or an image classification system.
29. The apparatus of claim 27 , wherein at least one non-output layer of the ANN section is characterized as a convolutional layer or a fully interconnected layer, and wherein the nodes in all remaining non-output layers in the ANN section each have a single output connected to a single downstream node.
30. The apparatus of claim 27 , wherein the computer circuit is characterized as at least one graphical processing unit (GPU) having a plurality of programmable processors that operate in parallel.