MAPPING NEURAL NETWORKS TO HARDWARE
A neural network is mapped to hardware by defining a plurality of layer groups, each layer group comprising one or more layers of the neural network that are processed in a single pass through the hardware. The layer groups are grouped into tile groups, each tile group comprising a set of layers groups that are evaluated when executing the neural network. Grouping the layer groups into a tile group comprises selecting a layer group that precedes a first layer group in the tile group and determining a number of times that input data to the layer group is read from memory. In response to this number exceeding a threshold, it is determined whether to merger the layer group into the tile group by determining an amount of space in on-chip memory required for storing pre-fetched input data and assessing one or more criteria relating to output data of the layer group.
1 . A method of mapping a neural network to hardware, the method comprising:
defining a plurality of layer groups, each layer group comprising one or more layers of the neural network that are processed in a single pass through the hardware; and
grouping the layer groups into one or more tile groups, each tile group comprising a set of layers groups that are evaluated when executing the neural network, wherein grouping the layer groups into a tile group comprises:
selecting a layer group that precedes, in execution order, a first layer group in the tile group and determining an input pre-fetch ratio for the layer group, the input pre-fetch ratio corresponding to a number of times that input data to the layer group is read from memory;
comparing the input pre-fetch ratio to an input pre-fetch force factor, wherein the input pre-fetch force factor defines a threshold for pre-fetching input data; and
in response to the input pre-fetch ratio exceeding the pre-fetch force factor, assessing one or more criteria relating to output data of the layer group, the criteria including a size of a pre-fetch buffer configured to store the input data and in response to determining that the one or more criteria relating to output data of the layer group are satisfied, merging the layer group into the tile group.
2 . The method according to claim 1 , wherein in response to determining that the one or more criteria relating to output data of the layer group are not satisfied, the layer group is not merged into the tile group.
3 . The method according to claim 1 , the method further comprising:
in response to the input pre-fetch ratio exceeding the pre-fetch force factor, pre-allocating space in on-chip memory for the input data.
4 . The method according to claim 3 , wherein in response to the input pre-fetch ratio exceeding the pre-fetch force factor and further in response to determining that the one or more criteria relating to output data of the layer group are not satisfied, releasing the pre-allocation.
5 . The method according to claim 1 , wherein grouping the layer groups into a tile group further comprises:
in response to the input pre-fetch ratio equalling the pre-fetch force factor, assessing one or more criteria relating to output data of the layer group, the criteria including a size of a pre-fetch buffer configured to store the input data and in response to determining that the one or more criteria relating to output data of the layer group are satisfied, merging the layer group into the tile group.
6 . The method according to claim 5 , the method further comprising:
in response to the input pre-fetch ratio equalling the pre-fetch force factor, pre-allocating space in on-chip memory for the input data.
7 . The method according to claim 6 , wherein in response to the input pre-fetch ratio equalling the pre-fetch force factor and further in response to determining that the one or more criteria relating to output data of the layer group are not satisfied, releasing the pre-allocation.
8 . The method according to claim 1 , wherein grouping the layer groups into a tile group further comprises, when assessing one or more criteria relating to output data of the layer group, disregarding any pre-allocation of space in on-chip memory for the input data of the first layer group in the tile group.
9 . The method according to claim 1 , wherein assessing one or more criteria relating to output data of the layer group, the criteria including a size of a pre-fetch buffer configured to store the input data, comprises:
splitting the input data into a plurality of passes; and
assessing one or more criteria relating to output data of the layer group, the criteria including a size of a pre-fetch buffer configured to store a pass of the input data.
10 . The method according to claim 1 , further comprising:
storing the mapped neural network in a memory for execution on the hardware.
11 . The method according to claim 1 , further comprising:
executing the neural network on the hardware.
12 . A computing device comprising:
a processor; and
a memory arranged to store computer executable instructions that, when executed by the processor, cause the processor to:
define a plurality of layer groups, each layer group comprising one or more layers of the neural network that are processed in a single pass through the hardware; and
group the layer groups into one or more tile groups, each tile group comprising a set of layers groups that are evaluated when executing the neural network, wherein grouping the layer groups into a tile group comprises:
selecting a layer group that precedes, in execution order, a first layer group in the tile group and determining an input pre-fetch ratio for the layer group, the input pre-fetch ratio corresponding to a number of times that input data to the layer group is read from memory;
comparing the input pre-fetch ratio to an input pre-fetch force factor, wherein the input pre-fetch force factor defines a threshold for pre-fetching input data; and
in response to the input pre-fetch ratio exceeding the pre-fetch force factor, assessing one or more criteria relating to output data of the layer group, the criteria including a size of a pre-fetch buffer configured to store the input data and in response to determining that the one or more criteria relating to output data of the layer group are satisfied, merging the layer group into the tile group.
13 . The computing device according to claim 12 , wherein the computer executable instructions, when executed by the processor, further cause the processor, in response to determining that the one or more criteria relating to output data of the layer group are not satisfied, to not merge the layer group into the tile group.
14 . The computing device according to claim 12 , wherein the computer executable instructions, when executed by the processor, further cause the processor to:
in response to the input pre-fetch ratio exceeding the pre-fetch force factor, pre-allocate space in on-chip memory for the input data.
15 . The computing device according to claim 14 , wherein the computer executable instructions, when executed by the processor, further cause the processor, in response to the input pre-fetch ratio exceeding the pre-fetch force factor and further in response to determining that the one or more criteria relating to output data of the layer group are not satisfied, to release the pre-allocation.
16 . The computing device according to claim 12 , wherein grouping the layer groups into a tile group further comprises:
in response to the input pre-fetch ratio equalling the pre-fetch force factor, assessing one or more criteria relating to output data of the layer group, the criteria including a size of a pre-fetch buffer configured to store the input data and in response to determining that the one or more criteria relating to output data of the layer group are satisfied, merging the layer group into the tile group.
17 . The computing device according to claim 16 , wherein the computer executable instructions, when executed by the processor, further cause the processor, in response to the input pre-fetch ratio equalling the pre-fetch force factor, to pre-allocate space in on-chip memory for the input data.
18 . The computing device according to claim 17 , wherein the computer executable instructions, when executed by the processor, further cause the processor, in response to the input pre-fetch ratio equalling the pre-fetch force factor and further in response to determining that the one or more criteria relating to output data of the layer group are not satisfied, to release the pre-allocation.
19 . The computing device according to claim 12 , wherein grouping the layer groups into a tile group further comprises, when assessing one or more criteria relating to output data of the layer group, disregarding any pre-allocation of space in on-chip memory for the input data of the first layer group in the tile group.
20 . The computing device according to claim 12 , wherein assessing one or more criteria relating to output data of the layer group, the criteria including a size of a pre-fetch buffer configured to store the input data, comprises:
splitting the input data into a plurality of passes; and
assessing one or more criteria relating to output data of the layer group, the criteria including a size of a pre-fetch buffer configured to store a pass of the input data.
21 . The computing device according to claim 12 , wherein the computer executable instructions, when executed by the processor, further cause the processor, to store the mapped neural network in memory for execution on the hardware.