Apparatus for processing a neural network
View Patent ↗An apparatus for processing a neural network comprises an image memory into which an input image is written tile-by-tile, each tile overlapping a previous tile to a limited extent; a weights memory for storing weight information for a plurality of convolutional layers of a neural network, including at least two pooling layers; and a layer processing engine configured to combine information from the image and weights memories to generate an output map and to write the output map to image memory. The apparatus is configured to store a limited number of values from adjacent a boundary of an output map for a given layer. The layer processing engine is configured to combine the output map values from a previously processed image tile with the information from the image memory and the weights when generating an output map for a layer of the neural network following the given layer.
1. An apparatus comprising:
an image memory configured to receive an input image tile-by-tile, a first tile at least partially overlapping a second tile;
a weights memory configured to store weight information associated with a plurality of convolutional layers of a neural network, said neural network including a first pooling layer, a second pooling layer, and a third pooling layer; and
a layer processing engine configured to:
combine, as first combined information, information associated with said image memory and said weight information associated with said weights memory;
generate a first output map for said first tile based at least in part on the first combined information;
write said first output map to said image memory;
generate map values adjacent a boundary of said first output map, said map values generated after said first pooling layer and before said third pooling layer of said neural network;
store, as stored map values in said image memory, said map values adjacent said boundary of said first output map;
combine, as second combined information, said stored map values adjacent the boundary of said first output map with input map information from said image memory and said weight information from said weights memory; and
generate a second output map associated with said second tile based at least in part on said second combined information.
2. An apparatus according to claim 1 wherein:
said neural network comprises a first convolutional layer followed by the first pooling layer, a second convolutional layer followed by the second pooling layer, a third convolutional layer followed by the third pooling layer, and a fourth convolutional layer followed by a fourth pooling layer, and
said map values adjacent the boundary of said first output map are generated after the second pooling layer and before the fourth pooling layer.
3. An apparatus according to claim 1 wherein said map values comprise one or more of: two columns of output values; and two rows of output values from adjacent said boundary of said first output map.
4. An apparatus according to claim 1 wherein:
said input image is read from a top corner, left-to-right, row-by-row, and
said map values adjacent said boundary comprises one or more of: a right boundary, a left boundary, a top boundary, and a bottom boundary of said first output map, and said layer processing engine is further configured to:
access said stored map values associated with said first tile from said image memory to generate said second output map.
5. An apparatus according to claim 1 wherein image said second tile overlaps said first tile by less than 25%.
6. An apparatus according to claim 1 wherein said input image comprises 256×256 pixels and wherein said input image is written to said image memory in 25 64×64 pixel tiles.
7. A system comprising:
one or more processors; and
non-transitory computer-readable storage media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving, from a first memory, an input image tile-by-tile, a first tile at least partially overlapping a second tile;
receiving, from a second memory different from the first memory, weight information associated with a plurality of convolutional layers of a neural network, said neural network including a first pooling layer, a second pooling layer, and a third pooling layer;
combining, as first combined information, information associated with said first memory and said weight information associated with said second memory;
generating a first output map for said first tile based at least in part on the first combined information;
writing said first output map to said first memory;
generating map values adjacent a boundary of said first output map, said map values generated after said first pooling layer and before said third pooling layer of said neural network;
storing, as stored map values, said map values adjacent said boundary of said first output map;
combining, as second combined information, said stored map values adjacent the boundary of said first output map with input map information from said first memory and said weight information from said second memory; and
generating a second output map associated with said second tile based at least in part on said second combined information.
8. The system of claim 7 , wherein:
said neural network comprises a first convolutional layer followed by the first pooling layer, a second convolutional layer followed by the second pooling layer, a third convolutional layer followed by the third pooling layer, and a fourth convolutional layer followed by a fourth pooling layer, and
said map values adjacent the boundary of said first output map are generated after the second pooling layer and before the fourth pooling layer.
9. The system of claim 7 , wherein said map values comprise one or more of: two columns of output values and two rows of output values from adjacent said boundary of said first output map.
10. The system of claim 7 , wherein:
said input image is read from a top corner, left-to-right, row-by-row, and
said stored map values comprise one or more of: a right boundary, a bottom boundary a left boundary, and a top boundary.
11. The system of claim 7 , wherein image said second tile overlaps said first tile by less than 25%.
12. The system of claim 7 , wherein said input image comprises 256×256 pixels and wherein said input image is written to an image memory in 25 64×64 pixel tiles.
13. The system of claim 7 , wherein the first tile at least partially overlapping the second tile comprises a horizontal overlap or a vertical overlap.
14. The system of claim 7 , wherein a first number of pixels associated with the first pooling layer is greater than a second number of pixels associated with the second pooling layer.
15. A method comprising:
receiving, from a first memory, an input image tile-by-tile, a first tile at least partially overlapping a second tile;
receiving, from a second memory different from the first memory, weight information associated with a plurality of convolutional layers of a neural network, said neural network including a first pooling layer, a second pooling layer, and a third pooling layer;
combining, as first combined information, information associated with said first memory and said weight information associated with said second memory;
generating a first output map for said first tile based at least in part on the first combined information;
writing said first output map to said first memory;
generating map values adjacent a boundary of said first output map, said map values generated after said first pooling layer and before said third pooling layer of said neural network;
storing, as stored map values, said map values adjacent said boundary of said first output map;
combining, as second combined information, said stored map values adjacent the boundary of said first output map with input map information from said first memory and said weight information from said second memory; and
generating a second output map associated with said second tile based at least in part on said second combined information.
16. The method of claim 15 , wherein:
said neural network comprises a first convolutional layer followed by the first pooling layer, a second convolutional layer followed by the second pooling layer, a third convolutional layer followed by the third pooling layer, and a fourth convolutional layer followed by a fourth pooling layer, and
said map values adjacent the boundary of said first output map are generated after the second pooling layer and before the fourth pooling layer.
17. The method of claim 15 , wherein said map values comprise one or more of: two columns of output values; and two rows of output values from adjacent said boundary of said first output map.
18. The method of claim 15 , wherein said stored map values comprise one or more of: a right boundary, a bottom boundary, a left boundary, and a top boundary.
19. The method of claim 15 , wherein image said second tile overlaps said first tile by less than 25%.
20. The method of claim 15 , wherein said input image comprises 256×256 pixels and wherein said input image is written to an image memory in 25 64×64 pixel tiles.