Operational neural networks and self-organized operational neural networks with generative neurons
Systems, methods, apparatuses, and computer program products for neural networks. In accordance with some example embodiments, an operational neuron model may comprise an artificial neuron comprising a composite nodal operator, a pool-operator, and an activation function operator. The nodal operator may comprise a linear function or non-linear function. In accordance with certain example embodiments, a generative neuron model may include a composite nodal-operator generated during the training using Taylor polynomial approximation without restrictions. In accordance with various example embodiments, a self-organized operational neural network (Self-ONN) may include one or more layers of generative neurons.
1 . An apparatus comprising:
at least one processor;
at least one memory; and
an operational neuron model configured to perform operations using the at least one processor and at least one memory, comprising:
an artificial neuron comprising a composite nodal operator;
a pool-operator; and
an activation function operator, wherein the composite nodal operator comprises a linear function or non-linear function, wherein the operational neuron model is configured to perform a stochastic gradient-descent training method comprising back propagation, and the composite nodal operator comprises at least one kernel parameter of a distinct frequency.
2 . The operational neuron model of claim 1 , wherein the composite nodal operator comprises at least one of a sinusoid, exponential, Gaussian, and Laplacian function or any other nonlinear function.
3 . The operational neuron model of claim 2 , wherein the operational neuron model reduces to a convolutional neuron model based upon the composite nodal operator comprising the linear function.
4 . An apparatus comprising:
at least one processor;
at least one memory; and
a generative neuron model configured to perform operations using the at least one processor and at least one memory, comprising:
a composite nodal operator generated during training using Taylor polynomial approximation without restrictions, wherein the generative neuron model is configured to perform a stochastic gradient-descent training method comprising back propagation, and the composite nodal operator comprises at least one kernel parameter of a distinct frequency.
5 . The generative neuron model of claim 4 , wherein an order of the polynomial of the Taylor polynomial approximation comprises a network parameter.
6 . The generative neuron model of claim 5 , wherein the generative neuron model reduces to a convolutional neuron of a CNN when the order of the polynomial of the Taylor polynomial equals to 1.
7 . The generative neuron model of claim 4 , wherein the generative neuron model is configured to self-organize the composite nodal operator during training.
8 . An apparatus comprising:
at least one processor;
at least one memory; and
an operational neural network (ONN) configured to perform operations using the at least one processor and at least one memory, comprising:
a neuron model configured to perform at least one linear or non-linear transformation in each of a plurality of neuron layers, wherein the operational neuron model is configured to perform a stochastic gradient-descent training method comprising back propagation, and the neuron model comprises at least one kernel parameter of a distinct frequency.
9 . An apparatus comprising:
at least one processor;
at least one memory; and
an operational neural network configured to perform operations using the at least one processor and at least one memory, comprising one or more layers of operational neurons, wherein the operational neuron model is configured to perform a stochastic gradient-descent training method comprising back propagation, and the operational neuron model comprises at least one kernel parameter of a distinct frequency.
10 . An apparatus comprising:
at least one processor;
at least one memory; and
a self-organized operational neural network (Self-ONN) configured to perform operations using the at least one processor and at least one memory, comprising one or more layers of generative neurons, wherein the Self-ONN is configured to perform a stochastic gradient-descent training method comprising back propagation, and the operational neuron model comprises at least one kernel parameter of a distinct frequency.