CONSTRAINED CLUSTERING ALGORITHM FOR EFFICIENT HARDWARE IMPLEMENTATION OF A DEEP NEURAL NETWORK ENGINE
A method and a system for operating a deep neural network. In the method and system, a subset of floating-point values are used to represent weights in the DNN; the floating-point values are quantized onto a flexible-power-of-two (FPoT) alphabet; values in the FPoT alphabet are listed in a plurality of regions; and an empty region among the plurality of regions is merged to neighbour regions to output dusters of the weights in merged regions, the merged regions having respective centroids and boundary lines in between.
1 . A method for operating a deep neural network (DNN), comprising:
using a subset of floating-point values to represent weights in the DNN;
quantizing the floating-point values onto a flexible-power-of-two (FPoT) alphabet;
listing values in the FPoT alphabet in a plurality of regions; and
merging an empty region among the plurality of regions to neighbor regions to output clusters of the weights in merged regions, the merged regions having respective centroids and boundary lines in between.
2 . The method of claim 1 , wherein the respective centroids and boundary lines of the merged regions are formed by a constrained clustering algorithm consolidating the clusters of the weights in the DNN into a reduced number of clusters.
3 . The method of claim 2 , further comprising providing the centroids and boundary lines for the reduced number of clusters to the DNN.
4 . The method of claim 1 , wherein the listing values in the FPoT alphabet comprises listing all values in the FPoT alphabet in the plurality of regions.
5 . The method of claim 1 , wherein the merging comprises merging each empty region among the plurality of regions with a neighbor region.
6 . The method of claim 1 , wherein the merging comprises:
identifying a best region, out of two adjacent neighbor regions, to merge with the empty region;
identifying a best direction to merge the best region; and
merging the best region in the best direction.
7 . The method of claim 1 , wherein the identifying a best region comprises re-adjusting quantized values of the floating-point values produced in the quantizing to reduce a mean square error in the merged regions.
8 . The method of claim 1 , wherein the using a subset of floating-point values comprises:
scaling and shifting the floating-point values of a general weight distribution of the weights in the DNN to provide a symmetric weight distribution.
9 . The method of claim 1 , further comprising merging non-empty regions among the plurality of regions with respective neighbor regions.
10 . The method of claim 9 , wherein the merging non-empty regions comprises:
identifying a best region, out of two adjacent neighbor regions, to merge with the non-empty region;
identifying a best direction to merge the best region; and
merging the best region in the best direction.
11 . A memory system for operating a deep neural network (DNN), comprising:
a data source; and
a controller configured to calculate parameters corresponding to the DNN, wherein the controller is programmed to:
use a subset of floating-point values to represent weights in the DNN;
quantize the floating-point values onto a flexible-power-of-two (FPoT) alphabet;
list values in the FPoT alphabet in a plurality of regions; and
merge an empty region among the plurality of regions to neighbor regions to output clusters of the weights in merged regions, the merged regions having respective centroids and boundary lines in between.
12 . The system of claim 11 , wherein the respective centroids and boundary lines of the merged regions are formed by a constrained clustering algorithm consolidating the dusters of the weights in the DNN into a reduced number of dusters.
13 . The system of claim 12 , wherein the controller is further programmed to provide the centroids and boundary lines for the reduced number of clusters to the DNN.
14 . The system of claim 11 , wherein the controller is further programmed to list all values in the FPoT alphabet in the plurality of regions.
15 . The system of claim 11 , wherein the controller is further programmed to merge each empty region among the plurality of regions with a neighbor region.
16 . The system of claim 11 , wherein the controller is further programmed to:
identify a best region, out of two adjacent neighbor regions, to merge with the empty region;
identify a best direction to merge the best region; and
merge the best region in the best direction.
17 . The system of claim 11 , wherein the controller further programmed to re-adjust quantized values of the floating-point values produced in the quantizing to reduce a mean square error in the merged regions.
18 . The system of claim 11 , wherein the controller is further programmed to scale and shift the floating-point values of a general weight distribution of the weights in the DNN to provide a symmetric weight distribution.
19 . The system of claim 11 , wherein the controller is further programmed to merge non-empty regions among the plurality of regions with respective neighbor regions.
20 . The system of claim 19 , wherein the controller is further programmed to:
identify a best region, out of two adjacent neighbor regions, to merge with the non-empty region;
identify a best direction to merge the best region; and
merge the best region in the best direction.