Modular SoC AI/ML inference engine supporting multiple ML models sharing inference-instance data
An electronic circuit system implementing and executing machine learning inference engines. While ML inference engines are based on (architectures and parameters defined by) configured, trained and tuned machine learning models, our design has the novel ability to support data driven, on-the-fly-reconfigured model runs. Reconfiguration and tuning operations include dynamic computational graph modifications, define-by-run alterations, changes to network depth (number of layers) and width (neurons per layer), and adjustments to weights, biases, plus activation function parameters. Neural networks supported include Feed-Forward, RNN, CNN, and Hopfield architectures, plus Ensemble, Federated, Cooperating, Adversarial, and Swarm collections. Decision Trees and Forests are also supported, as are more esoteric approaches such as ART and KAN. Our invention is capable of running both standalone and cooperatively, the cooperative processing being local and/or remote/cloud based, interfacing with telemetry applications to feed data, and machine learning software to feed new or updated models.
1 . A digital circuit implementing a neural network inference engine comprising:
a plurality of neural networks, with each of the plurality of neural networks having a plurality of layers, said plurality of layers comprising at least a first hidden layer, one or more optional hidden layers, and a final output layer;
wherein input data is split, using the digital circuit, into input sets when required for a machine learning model, the input sets being disjoint or overlapping depending on the machine learning model, and wherein input values of the input sets are fed to one or more of the plurality of neural networks to implement one or more models, and
wherein each of the plurality of neural networks has a distinct final output layer;
a plurality of neurons in each of the plurality of layers, said plurality of neurons comprising a first neuron set, a second neuron set, and one or more additional neuron sets,
wherein the first neuron set is in the first hidden layer and the second neuron set is in a second hidden layer;
a digital data transfer circuit in each of the plurality of neural networks,
wherein a type of the digital data transfer circuit can be independently selected for each of the plurality of neural networks,
wherein the digital data transfer circuit is configured to transfer neuron outputs, known as activations, for any of the plurality of neurons present between and among the plurality of layers;
wherein the digital data transfer circuit is configured to transfer data of the neuron outputs for all of the plurality of neurons in the final output layer of one neural network of the plurality of neural networks directly to an output channel of the digital circuit configured to receive the data, and produce output as a final answer of an inference operation.
2 . The digital circuit of claim 1 wherein each of the plurality of neural networks is configured to be run in parallel or serially.
3 . The digital circuit of claim 1 wherein the plurality of neural networks are based on evaluating a logical or combination logical-and-arithmetic expression with one or more feature values of the input data being processed.
4 . The digital circuit of claim 1 wherein one or more individual neural networks of the plurality of neural networks are substituted by a collection of more than one neural network either in an ensemble group mode or in a recursive application of the network collections.
5 . The digital circuit of claim 4 , wherein the digital data transfer circuit transfers the neuron outputs from a first layer of one neural network of the collection of more than one neural network to a second layer of a same neural network or a different neural network of the collection of more than one neural network, wherein the neuron outputs are transferred either by collecting outputs from a plurality of neurons of the first layer and transmitting the collected outputs in bulk to the second layer, or by transferring outputs from individual neurons of the first layer to neurons of the second layer.
6 . The digital circuit of claim 1 wherein at any of the plurality of layers in any of the plurality of neural networks, a decision is made via evaluating and testing a logical or arithmetic or combination logical-and-arithmetic expressions on one or more calculated single neuron or groups of neuron outputs to a transmitting layer and continue processing from a receiving layer.
7 . The digital circuit of claim 1 wherein the digital transfer circuit is configured to be used as neuron inputs for all of the plurality of neurons in the layer or layers of the plurality of layers that the data is transferred.
8 . The digital circuit of claim 1 wherein the data is transferred sequentially from layer to layer.
9 . The digital circuit of claim 1 wherein the data is transferred non-sequentially from layer to layer.
10 . The digital circuit of claim 1 wherein the neuron outputs are transferred in bulk to a next sequential layer.
11 . The digital circuit of claim 1 wherein one or more additional final neural networks are configured to function as data fusion mechanisms at inference time, merging the final answers transferred from each network using one or more fusion algorithms which were installed on each final network from training operations before these models were loaded on the digital circuit.
12 . A digital circuit implementing an inference engine comprising:
one or more decision trees, with each of the one or more decision trees having a plurality of levels, said plurality of levels comprising at least a first top level, one or more optional intermediate levels, and a final bottom output level,
wherein input data flows into the digital circuit and is split into input sets when required for a machine learning model, the input sets being disjoint or overlapping depending on the machine learning model, and wherein input values of the input sets are sent according to a topology and fed to one or more of the one or more decision trees to implement one or more models, and
wherein each of the one or more decision trees has a distinct final output level;
a plurality of decision-boxes in each of the plurality of levels, said plurality of decision-boxes comprising a first decision-box set, a second decision-box set, and one or more additional decision-box sets,
wherein the first decision-box set is in a first hidden level and the second decision-box set is in a second hidden level;
a digital data transfer circuit in each of the one or more decision trees,
wherein a type of the digital data transfer circuit is independently selected for each of the one or more decision trees,
wherein the digital data transfer circuit is configured to transfer decision-box outputs for any of the plurality of decision-boxes present between and among the plurality of levels; and
wherein the digital data transfer circuit is configured to transfer data of the decision-box outputs for all of the plurality of decision-boxes in the final level of a given network directly to an output channel of the digital data transfer circuit as a final answer of an inference operation.
13 . The digital circuit of claim 12 wherein each of the one or more decision trees is implemented using neural network circuitry.
14 . The digital circuit of claim 12 wherein each decision-box in the one or more decision trees is implemented as a one-decision-box level with an additional logical comparison.
15 . The digital circuit of claim 14 wherein a level-to-level communication mechanism is used for each tree branch connecting two decision-boxes.