Neural network output layer for machine learning
Techniques for a neural network output layer for machine learning are disclosed. A plurality of processing elements within a reconfigurable fabric is configured to implement a data flow graph, where the data flow graph implements a neural network. The data flow graph can include machine learning or deep learning. A layer is implemented, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one, where the second vector sums to a value of one using fixed-point calculations. The layer can include a final layer within the neural network. The layer that maps the first vector includes a Softmax function. Results of the neural network are classified based on a value of the second vector. The classifying can include part of a machine learning or a deep learning process.
1. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, based on a parabolic estimator function,
within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector.
2. The method of claim 1 wherein the layer is a final layer within the neural network.
3. The method of claim 1 wherein the layer that maps the first vector comprises a Softmax function.
4. The method of claim 3 wherein the layer that maps the first vector is an output layer for the neural network.
5. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network; and
the layer is based on a parabolic estimator function.
6. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network,
the layer is based on a parabolic estimator function; and
the parabolic estimator function is implemented using lookup tables.
7. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network,
the layer is based on a parabolic estimator function;
the parabolic estimator function is implemented using lookup tables; and
the lookup tables comprise three lookup tables.
8. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network,
the layer is based on a parabolic estimator function;
the parabolic estimator function is implemented using lookup tables;
the lookup tables comprise three lookup tables; and
further comprising two multiplex per element of an output vector.
9. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network,
the layer is based on a parabolic estimator function;
the parabolic estimator function is implemented using lookup tables;
the lookup tables comprise three lookup tables;
further comprising two multiplex per element of an output vector; and
no more than three multipliers per element of an output vector are used.
10. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network,
the layer is based on a parabolic estimator function;
the parabolic estimator function is implemented using lookup tables;
the lookup tables comprise three lookup tables; and
the three lookup tables are each 256 elements or fewer in length.
11. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network,
the layer is based on a parabolic estimator function;
the parabolic estimator function is implemented using lookup tables; and
the lookup tables comprise equidistant points.
12. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network,
the layer is based on a parabolic estimator function;
the parabolic estimator function is implemented using lookup tables;
the lookup tables comprise equidistant points; and
values between two equidistant points with a lookup table correspond to an integer number of intervals between the two points.
13. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network,
the layer is based on a parabolic estimator function;
the parabolic estimator function is implemented using lookup tables;
the lookup tables comprise equidistant points;
values between two equidistant points with a lookup table correspond to an integer number of intervals between the two points; and
the integer number of intervals is based on a power of two.
14. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network;
the layer is based on a parabolic estimator function; and
the parabolic estimator function is implemented using one lookup table.
15. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network;
the layer is based on a parabolic estimator function; and
the parabolic estimator function is based on a moving set of three points.
16. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network;
the layer is based on a parabolic estimator function; and
the parabolic estimator function implements e Xi or k Xi for the layer, where Xi are arguments of the Softmax function, k is a real number greater than 1, and e is the natural logarithm base.
17. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network;
the layer is based on a parabolic estimator function;
the parabolic estimator function implements e Xi or k Xi for the layer, where Xi are arguments of the Softmax function, k is a real number greater than 1, and e is the natural logarithm base; and
k Xi is calculated using 16-bit fixed-point arithmetic.
18. A processor-implemented method for data manipulation comprising:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector;
wherein the layer that maps the first vector comprises a Softmax function,
the layer that maps the first vector is an output layer for the neural network;
the layer is based on a parabolic estimator function;
the parabolic estimator function implements e Xi or k Xi for the layer, where Xi are arguments of the Softmax function, k is a real number greater than 1, and e is the natural logarithm base; and
k is equal to 2.
19. The method of claim 1 further comprising using 16-bit fixed-point arithmetic for training passes.
20. The method of claim 6 wherein lookup table entries are calculated offline.
21. A computer program product embodied in a non-transitory computer readable medium for data manipulation, the computer program product comprising code which causes one or more processors to perform operations of:
configuring a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implementing a layer, based on a parabolic estimator function,
within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums of one using fixed point calculations; and
classifying results of the neural network based on a value of the second vector.
22. A computer system for data manipulation comprising:
a memory which stores instructions;
one or more processors coupled to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to
configure a plurality of processing elements within a reconfigurable fabric to implement a data flow graph, wherein the data flow graph implements a neural network;
implement a layer, based on a parabolic estimator function,
within the neural network, that maps a first vector of real values to a second vector of real values bounded by zero and one wherein the second vector sums to a value of one using fixed point calculations; and
classify results of the neural network based on a value of the second vector.