Methods of batch-based DNN processing for efficient analytics
Technology is disclosed herein to execute an inference model by a processor which includes a reshape layer. In an implementation, the reshape layer of the inference model receives an output produced by a previous layer of the inference model and inserts padding into the output, then supplies the padded output as an input to a next layer of the inference model. In an implementation, the inference model includes a stitching layer at the beginning of the inference model and an un-stitch layer at the end of the model. The stitching layer of the inference model stitches together multiple input images into an image batch and supplies the image batch as an input to a subsequent layer. The un-stitch layer receives output from a penultimate layer of the inference model and unstitches the output to produce multiple output images corresponding to the multiple input images.
1 . A method of executing an inference model by a processor, the method comprising:
by a stitching layer of the inference model:
receiving multiple images;
stitching together the multiple images to form a batch input; and
supplying the batch input to a subsequent layer of the inference model; and
by a reshape layer of the inference model:
receiving an output produced by a previous layer of the inference model, wherein the output comprises multiple feature maps corresponding to the multiple images;
inserting padding into the output, resulting in a padded output, including by inserting side padding between the multiple feature maps; and
supplying the padded output as an input to a next layer of the inference model.
2 . The method of claim 1 wherein the stitching layer comprises a layer at a beginning of the inference model, and wherein the inference model further includes an un-stitch layer that comprises a layer at an end of the inference model, and other layers positioned between the stitching layer and the un-stitch layer, and wherein the other layers include the subsequent layer, the previous layer, the reshape layer, and the next layer.
3 . The method of claim 2 wherein inserting the padding into the output by the processor further comprises, at the reshape layer:
inserting top padding above each of the multiple feature maps;
inserting bottom padding below each of the multiple feature maps, wherein the bottom padding extends past a right side of a rightmost feature map; and
inserting additional side padding only to a left of a leftmost feature map of the multiple feature maps.
4 . The method of claim 3 wherein executing the inference model by the processor further comprises, at the un-stitch layer:
receiving a second output from a penultimate layer of the inference model, wherein the second output comprises second feature maps corresponding to the multiple images; and
unstitching the second output to separate the second feature maps.
5 . The method of claim 4 wherein the previous layer comprises a convolutional layer, and wherein the next layer comprises another convolutional layer.
6 . The method of claim 5 wherein each image, of the multiple images, comprises one of: an entirely different image relative to each other of the multiple images, or a different portion of a single image.
7 . The method of claim 2 wherein the inference model further includes a second reshape layer positioned between two layers of the inference model other than the previous layer and the next layer.
8 . The method of claim 7 wherein executing the inference model by the processor further comprises, at the second reshape layer of the inference model:
receiving a second output produced by a previous one of the two layers, wherein the second output comprises second feature maps corresponding to the multiple images;
inserting second padding into the second output, resulting in a second padded output, including by inserting second side padding between the second feature maps; and
supplying the second padded output as an input to a next one of the two layers.
9 . A computing apparatus comprising:
one or more computer-readable storage media;
one or more processors operatively coupled with the one or more computer-readable storage media; and
program instructions stored on the one or more computer-readable storage media that, when executed by the one or more processors, direct the computing apparatus to at least:
execute an inference model, wherein the inference model includes a stitching layer and a reshape layer, and wherein to execute the inference model, the program instructions direct the computing apparatus to:
at the stitching layer of the inference model:
receive multiple images;
stitch together the multiple images to form a batch input; and
supply the batch input to a subsequent layer of the inference model; and
at the reshape layer of the inference model:
receive an output produced by a previous layer of the inference model, wherein the output comprises multiple feature maps corresponding to the multiple images;
insert padding into the output, resulting in a padded output, wherein to insert the padding into the output, the program instructions direct the computing apparatus to insert side padding between the multiple feature maps; and
supply the padded output as an input to a next layer of the inference model.
10 . The computing apparatus of claim 9 wherein the stitching layer comprises a layer at a beginning of the inference model, wherein the inference model further includes an un-stitch layer that comprises a layer at an end of the inference model, and other layers positioned between the stitching layer and the un-stitch layer, and wherein the other layers include the subsequent layer, the previous layer, the reshape layer, and the next layer.
11 . The computing apparatus of claim 10 wherein the program instructions further direct the computing apparatus to, at the reshape layer of the inference model:
insert top padding above each of the multiple feature maps;
insert bottom padding below each of the multiple feature maps, wherein the bottom padding extends past a right side of a rightmost feature map; and
insert additional side padding only to a left of a leftmost feature map of the multiple feature maps.
12 . The computing apparatus of claim 11 wherein the program instructions further direct the computing apparatus to, at the un-stitch layer of the inference model:
receive a second output from a penultimate layer of the inference model, wherein the second output comprises second feature maps corresponding to the multiple images; and
unstitch the second output to separate the second feature maps.
13 . The computing apparatus of claim 12 wherein the previous layer comprises a convolutional layer, and wherein the next layer comprises another convolutional layer.
14 . The computing apparatus of claim 13 wherein each image, of the multiple images, comprises one of: an entirely different image relative to each other of the multiple images, or a different portion of a single image.
15 . The computing apparatus of claim 10 wherein the inference model further includes a second reshape layer positioned between two layers of the inference model other than the previous layer and the next layer.
16 . The computing apparatus of claim 15 wherein the program instructions further direct the computing apparatus to, at the second reshape layer of the inference model:
receive a second output produced by a previous one of the two layers, wherein the second output comprises second feature maps corresponding to the multiple images;
insert second padding into the second output, resulting in a second padded output, wherein to insert the second padding into the second output, the program instructions further direct the computing apparatus to insert second side padding between the second feature maps; and
supply the second padded output as an input to a next one of the two layers.
17 . A system comprising:
memory circuitry configured to store an inference model that includes a stitching layer and a reshape layer; and
processing circuitry coupled with the memory circuitry and configured to execute the inference model;
wherein the stitching layer of the inference model, when executed by the processing circuitry, causes the processing circuitry to:
receive multiple images;
stitch together the multiple images to form a batch input; and
supply the batch input to a subsequent layer of the inference model; and
wherein the reshape layer of the inference model, when executed by the processing circuitry, causes the processing circuitry to:
receive an output produced by a previous layer of the inference model, wherein the output comprises multiple feature maps corresponding to the multiple images;
insert padding into the output, resulting in a padded output, wherein to insert the padding into the output, the reshape layer causes the processing circuitry to, when executed, insert side padding between the multiple feature maps; and
supply the padded output as an input to a next layer of the inference model.
18 . The system of claim 17 wherein the stitching layer comprises a layer at a beginning of the inference model, wherein the inference model further includes an un-stitch layer that comprises a layer at an end of the inference model, and other layers positioned between the stitching layer and the un-stitch layer, and wherein the other layers include the subsequent layer, the previous layer, the reshape layer, and the next layer.
19 . The system of claim 18 wherein the reshape layer of the inference model, when executed by the processing circuitry, further causes the processing circuitry to:
insert top padding above each of the multiple feature maps;
insert bottom padding below each of the multiple feature maps, wherein the bottom padding extends past a right side of a rightmost feature map; and
insert additional side padding only to a left of a leftmost feature map of the multiple feature maps.
20 . The system of claim 19 wherein the un-stitch layer of the inference model, when executed by the processing circuitry, causes the processing circuitry to:
receive a second output from a penultimate layer of the inference model, wherein the second output comprises second feature maps corresponding the multiple images; and
unstitch the second output to separate the second feature maps.