Accelerator circuitry for acceleration of non-maximum suppression for object detection
This document describes an accelerator circuitry for facilitating acceleration of non-maximum suppression (NMS) for detection of objections within an image.
1 . An accelerator circuitry for facilitating acceleration of object detection operations of an image, the circuitry comprising:
a first set of processing elements performing a plurality of first computations in parallel, each first computation comprising:
receiving a unique bounding box associated with detected features within the image, and projecting the received bounding box to a score map cell in a first-stage three-dimensional confidence score map based on dimensions, a confidence score and a spatial location of the bounding box, and a down sampling ratio of the first-stage three-dimensional confidence score map;
a second set of processing elements communicatively coupled to the first set of processing elements, and to first and second data buffers, the second set of processing elements performing a plurality of second computations comprising:
partitioning each channel of the first-stage three-dimensional confidence score map into a plurality of regions, whereby dimensions of regions in each channel are different from dimensions of regions in other channels of the first-stage three-dimensional confidence score map,
mapping each region in each of the channels of the first-stage three-dimensional confidence score map to a corresponding kernel index in the first data buffer, and for each region, storing, at the corresponding kernel index mapped to the region, a score map cell that has a highest score in the region,
a third set of processing elements communicatively coupled to the first and second data buffers, the third set of processing elements performing a plurality of third computations comprising:
retrieving the score map cells stored in the first data buffer,
forming a padded post first-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each channel of the padded post first-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels of the padded post first-stage three-dimensional confidence score map to a corresponding kernel index in the second data buffer, and for each region, storing, at the corresponding kernel index mapped to the region, a score map cell that has a highest score in the region,
wherein the accelerator circuitry accelerates detection of objects in the image based at least in part on the score map cells stored in the second data buffer.
2 . The accelerator circuitry according to claim 1 , wherein before the accelerator circuitry accelerates detection of objects in the image based at least in part on the score map cells stored in the second data buffer, the plurality of second computations performed by the second set of processing elements further comprises:
retrieving the score map cells stored in the second data buffer,
generating a second-stage three-dimensional confidence score map based on the retrieved score map cells,
concatenating channels of the second-stage three-dimensional confidence score map that each have a similar scale to form a plurality of scale-concatenated channels, wherein each scale-concatenated channel is associated with a scale of a channel of the second-stage three-dimensional confidence score map;
partitioning each of the plurality of scale-concatenated channels into regions,
mapping each region in each of the plurality of scale-concatenated channels to its corresponding kernel index in the first data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region, and
wherein the plurality of third computations performed by the third set of processing elements further comprises:
retrieving the score map cells stored in the first data buffer,
forming a padded second-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each channel of the padded post second-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels of the padded post second-stage three-dimensional confidence score map to its corresponding kernel index in the second data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region.
3 . The accelerator circuitry according to claim 2 , whereby the plurality of second computations performed by the second set of processing elements further comprises:
retrieving the score map cells stored in the second data buffer,
generating a third-stage three-dimensional confidence score map based on the retrieved score map cells,
concatenating channels of the third-stage three-dimensional confidence score map that each have a similar ratio to form a plurality of ratio-concatenated channels, wherein each ratio-concatenated channel is associated with a ratio of a channel of the third-stage three-dimensional confidence score map;
partitioning each of the plurality of ratio-concatenated channels into regions,
mapping each region in each of the plurality of ratio-concatenated channels to its corresponding kernel index in the first data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region, and
wherein the plurality of third computations performed by the third set of processing elements further comprises:
retrieving the score map cells stored in the first data buffer,
forming a padded post third-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each channel of the padded post third-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels of the padded post third-stage three-dimensional confidence score map to its corresponding kernel index in the second data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region.
4 . The accelerator circuitry according to claim 3 , whereby the plurality of second computations performed by the second set of processing elements further comprises:
retrieving the score map cells stored in the second data buffer,
generating a fourth-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each of the channels in the fourth-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels to its corresponding kernel index in the first data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region, and
wherein the plurality of third computations performed by the third set of processing elements further comprises:
retrieving the score map cells stored in the first data buffer,
forming a padded post fourth-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each channel of the padded post fourth-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels of the post fourth-stage three-dimensional confidence score map to its corresponding kernel index in the second data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region.
5 . The accelerator circuitry according to claim 4 whereby each of the channels of the padded post first-stage three-dimensional confidence score map, the padded post second-stage three-dimensional confidence score map, the padded post third-stage three-dimensional confidence score map and the padded post fourth-stage three-dimensional confidence score map comprise border score map cells padded with zeroes.
6 . The accelerator circuitry according to claim 1 , whereby each first computation performed by the first set of processing elements to project the received bounding box to the score map cell in the first-stage three-dimensional confidence score map comprises each first computation:
performing channel recovery on the received bounding box based on the dimensions of the bounding box to determine a channel of the first-stage three-dimensional confidence score map for the received bounding box; and
performing spatial recovery on the received bounding box based on the spatial location of the receiving bounding box, and the down sampling ratio of the first-stage three-dimensional confidence score map to determine a spatial location of a score map cell on the channel of the first-stage three-dimensional confidence score map associated with the received bounding box.
7 . The accelerator circuitry according to claim 6 , whereby each first computation performed by the first set of processing elements to perform channel recovery on the received bounding box based on the dimensions of the bounding box comprises each first computation:
identifying a channel of the first-stage three-dimensional confidence score map to be used as the channel for the received bounding box based on Euclidean-distances of the channels of the first-stage three-dimensional confidence score map to the dimensions of the bounding box.
8 . The accelerator circuitry according to claim 1 , whereby kernel indices in the first and second data buffers are grouped into read groups in each data buffer, whereby kernel indices in each read group are read sequentially when it is determined that the read group contains a valid score map cell.
9 . The accelerator circuitry according to claim 1 , whereby kernel indices in the first and second data buffers are grouped into read groups in each data buffer, whereby kernel indices in each read group are skipped when it is determined that the read group does not contain a valid score map cell.
10 . The accelerator circuitry according to claim 1 further comprising a fourth set of processing elements communicatively provided between the second and third sets of processing elements, and the first and second data buffers, the fourth set of processing elements performing a plurality of fourth computations comprising:
arbitrating kernel index conflicts at the first and second data buffers.
11 . A method to facilitate acceleration of object detection operations of an image, the method comprising:
performing a plurality of first computations in parallel using a first set of processing elements, each first computation comprising the steps of:
receiving a unique bounding box associated with detected features within the image, and projecting the received bounding box to a score map cell in a first-stage three-dimensional confidence score map based on dimensions, a confidence score and a spatial location of the bounding box, and a down sampling ratio of the first-stage three-dimensional confidence score map;
performing a plurality of second computations using a second set of processing elements communicatively coupled to the first set of processing elements, and to first and second data buffers, the second set of second computations comprising the steps of:
partitioning each channel of the first-stage three-dimensional confidence score map into a plurality of regions, whereby dimensions of regions in each channel are different from dimensions of regions in other channels of the first-stage three-dimensional confidence score map,
mapping each region in each of the channels of the first-stage three-dimensional confidence score map to a corresponding kernel index in the first data buffer, and for each region, storing, at the corresponding kernel index mapped to the region, a score map cell that has a highest score in the region,
performing a plurality of third computations using a third set of processing elements communicatively coupled to the first and second data buffers, the third set of computations comprising the steps of:
retrieving the score map cells stored in the first data buffer,
forming a padded post first-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each channel of the padded post first-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels of the padded post first-stage three-dimensional confidence score map to a corresponding kernel index in the second data buffer, and for each region, storing, at the corresponding kernel index mapped to the region, a score map cell that has a highest score in the region,
wherein detection of objects in the image are accelerated based at least in part on the score map cells stored in the second data buffer.
12 . The method according to claim 11 , wherein before the method of detecting objects in the image are accelerated based at least in part on the score map cells stored in the second data buffer, the plurality of second computations further comprises the steps of:
retrieving the score map cells stored in the second data buffer,
generating a second-stage three-dimensional confidence score map based on the retrieved score map cells,
concatenating channels of the second-stage three-dimensional confidence score map that each have a similar scale to form a plurality of scale-concatenated channels, wherein each scale-concatenated channel is associated with a scale of a channel of the second-stage three-dimensional confidence score map;
partitioning each of the plurality of scale-concatenated channels into regions,
mapping each region in each of the plurality of scale-concatenated channels to its corresponding kernel index in the first data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region, and
wherein the plurality of third computations further comprises the steps of:
retrieving the score map cells stored in the first data buffer,
forming a padded second-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each channel of the padded post second-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels of the padded post second-stage three-dimensional confidence score map to its corresponding kernel index in the second data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region.
13 . The method according to claim 12 , whereby the plurality of second computations further comprises the steps of:
retrieving the score map cells stored in the second data buffer,
generating a third-stage three-dimensional confidence score map based on the retrieved score map cells,
concatenating channels of the third-stage three-dimensional confidence score map that each have a similar ratio to form a plurality of ratio-concatenated channels, wherein each ratio-concatenated channel is associated with a ratio of a channel of the third-stage three-dimensional confidence score map;
partitioning each of the plurality of ratio-concatenated channels into regions, mapping each region in each of the plurality of ratio-concatenated channels to its corresponding kernel index in the first data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region, and
wherein the plurality of third computations further comprises the steps of:
retrieving the score map cells stored in the first data buffer,
forming a padded post third-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each channel of the padded post third-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels of the padded post third-stage three-dimensional confidence score map to its corresponding kernel index in the second data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region.
14 . The method according to claim 13 , the plurality of second computations further comprises the steps of:
retrieving the score map cells stored in the second data buffer,
generating a fourth-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each of the channels in the fourth-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels to its corresponding kernel index in the first data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region, and
wherein the plurality of third computations further comprises the steps of:
retrieving the score map cells stored in the first data buffer,
forming a padded post fourth-stage three-dimensional confidence score map based on the retrieved score map cells,
partitioning each channel of the padded post fourth-stage three-dimensional confidence score map into regions,
mapping each region in each of the channels of the post fourth-stage three-dimensional confidence score map to its corresponding kernel index in the second data buffer, and for each region, storing, at its corresponding kernel index mapped to the region, a score map cell that has a highest score in the region.
15 . The method according to claim 14 whereby each of the channels of the padded post first-stage three-dimensional confidence score map, the padded post second-stage three-dimensional confidence score map, the padded post third-stage three-dimensional confidence score map and the padded post fourth-stage three-dimensional confidence score map comprise border score map cells padded with zeroes.
16 . The method according to claim 11 , whereby the step of projecting the received bounding box to the score map cell in the first-stage three-dimensional confidence score map by each of the first computations further comprises each first computation:
performing channel recovery on the received bounding box based on the dimensions of the bounding box to determine a channel of the first-stage three-dimensional confidence score map for the received bounding box; and
performing spatial recovery on the received bounding box based on the spatial location of the receiving bounding box, and the down sampling ratio of the first-stage three-dimensional confidence score map to determine a spatial location of a score map cell on the channel of the first-stage three-dimensional confidence score map associated with the received bounding box.
17 . The method according to claim 16 , whereby the step of performing channel recovery on the received bounding box based on the scale and ratio of the bounding box by each of the first computation further comprises each first computation:
identifying a channel of the first-stage three-dimensional confidence score map to be used as the channel for the received bounding box based on Euclidean-distances of the channels of the first-stage three-dimensional confidence score map to the dimensions of the bounding box.
18 . The method according to claim 11 , whereby kernel indices in the first and second data buffers are grouped into read groups in each data buffer, whereby kernel indices in each read group are read sequentially when it is determined that the read group contains a valid score map cell.
19 . The method according to claim 11 , whereby kernel indices in the first and second data buffers are grouped into read groups in each data buffer, whereby kernel indices in each read group are skipped when it is determined that the read group does not contain a valid score map cell.
20 . The method according to claim 11 further comprising the steps of:
performing a plurality of fourth computations using a fourth set of processing elements communicatively provided between the second and third sets of processing elements, and the first and second data buffers, the plurality of fourth computations comprising the steps of:
arbitrating kernel index conflicts at the first and second data buffers.