Machine learning through multiple layers of novel machine trained processing nodes
Some embodiments of the invention provide efficient, expressive machined-trained networks for performing machine learning. The machine-trained (MT) networks of some embodiments use novel processing nodes with novel activation functions that allow the MT network to efficiently define with fewer processing node layers a complex mathematical expression that solves a particular problem (e.g., face recognition, speech recognition, etc.). In some embodiments, the same activation function (e.g., a cup function) is used for numerous processing nodes of the MT network, but through the machine learning, this activation function is configured differently for different processing nodes so that different nodes can emulate or implement two or more different functions (e.g., two or more Boolean logical operators, such as XOR and AND). The activation function in some embodiments is a periodic function that can be configured to implement different functions (e.g., different sinusoidal functions).
1. For an electronic device, a method comprising:
capturing a set of input data; and
executing a machine-trained (MT) network to process the captured set of input data and generate a set of output data based on the set of input data, wherein executing the MT network comprises executing a plurality of layers of processing nodes, each processing node of a plurality of the processing nodes comprising:
a linear operator that computes an intermediate value by combining a set of input values of the processing node based on a set of weight values of the processing node; and
a nonlinear operator that applies a piecewise linear periodic function to the intermediate value from the linear operator of the processing node to compute an output value of the processing node,
wherein a particular piecewise linear periodic function for a particular processing node comprises at least three linear segments, wherein a first one of the linear segments has a negative slope and a second one of the linear segments has a positive slope.
2. The method of claim 1 , wherein the electronic device that executes the MT network is a mobile device.
3. The method of claim 1 , wherein the electronic device that executes the MT network is an Internet of Things (TOT) device.
4. The method of claim 1 , wherein the piecewise linear periodic function for each processing node of the plurality of processing nodes is a continuous function.
5. The method of claim 1 , wherein the piecewise linear periodic function of each processing node is defined by a set of parameters that are configured through training.
6. The method of claim 5 , wherein the set of weight values of each processing node are also configured through training.
7. The method of claim 5 , wherein:
each of the segments of the particular piecewise linear periodic function comprises a function of the form y=mx+b for X1≤x≤X2; and
the training comprises adjusting the slope m, the value b, and the length of each function defined by X1 and X2.
8. The method of claim 5 , wherein the training is performed by a computing device different than the electronic device that performs the method, wherein the training comprises:
propagating a set of input values through the MT network to generate a set of output values;
determining an error of the generated set of output values compared to a set of predetermined output values for the set of input values; and
back-propagating the error through the MT network to adjust the sets of parameters of the MT network, thereby adjusting the piecewise linear functions.
9. The method of claim 5 , wherein the piecewise linear periodic functions of a set of the processing nodes are different functions as a result of the training.
10. The method of claim 1 , wherein each of the piecewise linear periodic functions of the plurality of processing nodes has a same number of linear segments.
11. The method of claim 1 , wherein a first piecewise linear periodic function of a first processing node has a first number of linear segments and a second piecewise linear periodic function of a second processing node has a second, different number of linear segments.
12. The method of claim 1 , wherein at least a set of the piecewise linear periodic functions emulate sinusoidal functions.
13. The method of claim 12 , wherein the emulation of sinusoidal functions enables the MT network to implement Fourier transforms.
14. The method of claim 1 further comprising presenting an output based on the generated set of output data.
15. The method of claim 1 , wherein the captured set of input data comprises an image and the MT network identifies objects in the image.
16. The method of claim 1 , wherein the captured set of input data comprises audio and the MT network performs voice recognition for the audio.
17. The method of claim 1 , wherein the MT network is a neural network.
18. The method of claim 1 , wherein at least a subset of the piecewise linear periodic functions are cup functions.
19. The method of claim 1 , wherein at least a subset of the piecewise linear periodic functions emulate a Boolean operator.
20. The method of claim 1 , wherein for at least a subset of the plurality of layers, the sets of input values of the linear operators of the processing nodes of the layers comprise sets of output values of processing nodes of prior layers.