Composite binary decomposition network
Embodiments are directed to a composite binary decomposition network. An embodiment of a computer-readable storage medium includes executable computer program instructions for transforming a pre-trained first neural network into a binary neural network by processing layers of the first neural network in a composite binary decomposition process, where the first neural network having floating point values representing weights of various layers of the first neural network. The composite binary decomposition process includes a composite operation to expand real matrices or tensors into a plurality of binary matrices or tensors, and a decompose operation to decompose one or more binary matrices or tensors of the plurality of binary matrices or tensors into multiple lower rank binary matrices or tensors.
1. A non-transitory computer-readable storage medium having stored thereon executable computer program instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
transforming a first neural network into a binary neural network by processing layers of the first neural network in a composite binary decomposition process, the first neural network having floating-point values, the composite binary decomposition process including:
a composite operation to expand real matrices or tensors of the first neural network into a first group of a plurality of binary matrices or tensors of the binary neural network; and
a decompose operation to decompose one or more binary matrices or tensors of the first group into a second group of a plurality of low rank binary matrices or tensors, the binary matrices or tensors of the second group having lower rank than the binary matrices or tensors of the first group.
2. The medium of claim 1 , wherein the first neural network is a pre-trained neural network.
3. The medium of claim 1 , wherein the composite binary decomposition process is performed without use of training data.
4. The medium of claim 1 , further comprising executable computer program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
normalizing a weight tensor for each layer of the first neural network.
5. The medium of claim 4 , further comprising executable computer program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
flattening of the normalized weight tensor to generate a matrix of real values for the composite operation.
6. The medium of claim 1 , wherein the decompose operation includes decomposing a matrix of binary values into a first binary matrix and a second binary matrix, wherein the first binary matrix is a vertically dominant matrix and the second binary matrix is a horizontally dominant matrix.
7. The medium of claim 1 , further comprising executable computer program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
identifying the one or more binary matrices or tensors of the first group as sparse binary matrices or tensors to be decomposed into multiple lower rank binary matrices or tensors of the second group.
8. The medium of claim 1 , wherein the composite binary decomposition process includes processing of every layer of the first neural network.
9. The medium of claim 1 , wherein the first neural network is a convolutional neural network (CNN).
10. A method comprising:
obtaining a first neural network, the first neural network having floating point values; and
transforming the first neural network into a binary neural network by processing layers of the first neural network in a composite binary decomposition process, the composite binary decomposition process including:
a composite operation to expand real matrices or tensors of the first neural network into a first group of a plurality of binary matrices or tensors of the binary neural network; and
a decompose operation to decompose one or more binary matrices or tensors of the first group into a second group of a plurality of low rank binary matrices or tensors, the binary matrices or tensors of the second group having lower rank than the binary matrices or tensors of the first group.
11. The method of claim 10 , wherein the first neural network is a pre-trained neural network.
12. The method of claim 10 , wherein the composite binary decomposition process is performed without use of training data.
13. The method of claim 10 , further comprising:
normalizing a weight tensor for each layer of the first neural network.
14. The method of claim 13 , further comprising:
flattening of the normalized weight tensor to generate a matrix of real values for the composite operation.
15. The method of claim 10 , wherein the decompose operation includes decomposing a matrix of binary values into a first binary matrix and a second binary matrix, wherein the first binary matrix is a vertically dominant matrix and the second binary matrix is a horizontally dominant matrix.
16. The method of claim 10 , wherein the one or more binary matrices or tensors of the first group are sparse binary matrices or tensors.
17. The method of claim 10 , wherein the composite binary decomposition process includes processing of every layer of the first neural network.
18. The method of claim 10 , wherein the first neural network is a convolutional neural network (CNN).
19. An apparatus comprising:
one or more processors to process data; and
a memory to store data, including data for deep learning processing;
wherein the apparatus is to transform a first neural network into a binary neural network in a composite binary decomposition process, the first neural network having floating point values, the composite binary decomposition process including:
a composite operation to expand real matrices or tensors of the first neural network into a first group of a plurality of binary matrices or tensors of the binary neural network; and
a decompose operation including the apparatus to decompose one or more binary matrices or tensors of the first group into a second group of a plurality of low rank binary matrices or tensors, the binary matrices or tensors of the second group having lower rank than the binary matrices or tensors of the first group.
20. The apparatus of claim 19 , further comprising one or more graphics processing units (GPUs), wherein the one or more GPUs perform the composite binary decomposition process at least in part.
21. The apparatus of claim 19 , further comprising a composite binary decomposition mechanism to support generation of the binary neural network, to support operation of the binary neural network, or both.
22. The apparatus of claim 21 , wherein the composite binary decomposition mechanism includes an accelerator for generation of the binary neural network.
23. The apparatus of claim 19 , wherein the first neural network is a pre-trained neural network and wherein the apparatus is to perform the composite binary decomposition process without use of training data.
24. The apparatus of claim 19 , wherein the decompose operation includes the apparatus to decompose a matrix of binary values into a first matrix and a second matrix, the first matrix being a vertical matrix and the second matrix being a horizontal matrix.
25. The apparatus of claim 19 , wherein the one or more binary matrices or tensors of the first group are sparse binary matrices or tensors.