Deep neural network
A hardware neural network system includes an input buffer for input neurons (Nbin), an output buffer for output neurons (Nbout), and a third buffer for synaptic weights (SB) connected to a Neural Functional Unit (NFU) and a control logic (CP) for performing synapses and neurons computations. The NFU pipelines a computation into stages, the stages including weight blocks (WB), an adder tree, and a non-linearity function.
1. A method comprising:
providing a hardware-software codesign to transform existing floating-point networks to 8-bit dynamic fixed-point networks with 4-bit integer power-of-two weights, consisting of 1 sign bit and 3-bit exponent, without changing a network topology, the power-of-two weights enabling a multiplier-free hardware accelerator to perform computation on dynamic fixed-point precision,
wherein the hardware-software codesign utilizes a two-phase training method comprising an 8-bit dynamic fixed-point with power-of-two weight training using backpropagation with gradients computed from true labels phase followed by an 8-bit dynamic fixed-point with power-of-two weight training using backpropagation with gradients computed from true labels and gradients computed from floating-point teacher network's logits phase to improve the accuracy of the dynamic fixed-point network with power-of-two weights.
2. The method of claim 1 wherein the teacher is a floating point network and the student network is a dynamic fixed-point network with power-of-two weights.