IP Library Granted Patent US 12,175,739
Granted Patent B2
US 12,175,739 · App. 17/149,437 · Granted Dec 24, 2024

Performing non-maximum suppression in parallel

Inventors: Yichun Shen (Shanghai, CN); Wanli Jiang (Shanghai, CN); Junghyun Kwon (Santa Clara, CA); Siyi Li (Shanghai, CN); Minwoo Park (Saratoga, CA); Sangmin Oh (San Jose, CA)
Assignee: NVIDIA Corporation
G06V10/95G06F18/2193G06V10/22G06V10/40G06V10/467
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,175,739
App. No.
17/149,437
Granted
Dec 24, 2024
Kind
B2
Abstract

Apparatuses, systems, and techniques to perform non-maximum suppression (NMS) in parallel to remove redundant bounding boxes. In at least one embodiment, two or more parallel circuits to perform two or more portions of a NMS algorithm in parallel to remove one or more redundant bounding boxes corresponding to one or more objects within one or more digital images.

Claims (61)

1. A processor comprising: one or more circuits to perform one or more parallel comparisons to select one of a plurality of bounding boxes based, at least in part, on an intersection over union (IoU) of only a subset of the plurality of bounding boxes having centers within a maximum radius of pixels from a bounding box within the subset.

2. The processor of claim 1 , wherein the one or more circuits are to perform two or more portions of a non-maximum suppression (NMS) algorithm in parallel and:

initiate a plurality of suppression processes to remove one or more redundant bounding boxes corresponding to one or more objects; and

define an area that covers a subset of a plurality of bounding boxes for each suppression process.

3. The processor of claim 1 , wherein the one or more circuits are to perform two or more portions of a non-maximum suppression (NMS) algorithm in parallel and:

identify a set of candidate points from an output feature map comprising a plurality of points, each point in the output feature map corresponding to one of the plurality of bounding boxes and comprising a confidence score, wherein each point of the set of candidate points comprises a confidence score that satisfies a confidence threshold; and

cause each of a plurality of parallel suppression processes to be performed in connection with a respective candidate point, wherein each of the plurality of parallel suppression processes is to identify a set of neighboring points that are within an area associated with the respective candidate point.

4. The processor of claim 3 , wherein each of the plurality of parallel suppression processes is to:

calculate an IoU value of the respective candidate point and a neighboring point in the identified set;

determine whether the IoU value satisfies an IoU threshold and a confidence score of the candidate point satisfies a criterion pertaining to a confidence score of the neighboring point; and

identify the candidate point as a redundant bounding box to be removed responsive to the IoU value satisfying the IoU threshold and the confidence score satisfying the criterion pertaining to the confidence score of the neighboring point.

5. The processor of claim 3 , wherein, to identify the set of neighboring points, the parallel suppression process is to:

calculate a distance between the respective candidate point and a second point in the output feature map; and

identify the second point as a neighboring point responsive to the distance satisfying a distance threshold, wherein the distance threshold corresponds to the area associated with the respective candidate point.

6. The processor of claim 5 , wherein, to calculate the distance, the parallel suppression process is to calculate a cosine distance between the respective candidate point and the second point.

7. The processor of claim 1 , wherein the one or more circuits comprise two or more parallel circuits to use a neural network to detect one or more objects within one or more images, and wherein a non-maximum suppression (NMS) algorithm is performed as a layer of the neural network.

8. The processor of claim 1 , wherein the one or more circuits are to perform two or more portions of a non-maximum suppression (NMS) algorithm in parallel and:

identify a set of candidate boxes comprising a set of candidate points from an output feature map comprising a plurality of points, each point in the output feature map corresponding to a first anchor box with a first confidence score and a second anchor box with a second confidence score, wherein at least one of the first confidence score satisfies a first confidence threshold or the second confidence score satisfies a second confidence threshold; and

initiate each of a plurality of parallel suppression processes in connection with a respective candidate point, wherein each of the plurality of parallel suppression processes is to identify at least one of a first set of neighboring points that are within a first area associated with the respective candidate point or a second set of neighboring points that are within a second area associated with the respective candidate point.

9. A system comprising:

one or more processors to perform one or more parallel comparisons to select one of a plurality of bounding boxes based, at least in part, on an intersection over union (IoU) of only a subset of the plurality of bounding boxes having centers within a maximum radius of pixels from a bounding box within the subset.

10. The system of claim 9 , wherein the one or more processors:

initiate a plurality of parallel suppression processes to remove one or more redundant bounding boxes corresponding to one or more objects; and

define an area that covers a subset of a plurality of bounding boxes for each suppression process.

11. The system of claim 9 , wherein the one or more processors comprise two or more circuits to use one or more neural networks to detect one or more objects within one or more images, wherein the one or more neural networks comprise a layer to perform a non-maximum suppression (NMS) algorithm in parallel.

12. The system of claim 9 , wherein the one or more processors use one or more neural networks comprising:

multiple layers to output an output feature map comprising a plurality of points, each point in the output feature map corresponding to a bounding box and a comprising a confidence score; and

a layer to identify a set of candidate points from the output feature map, wherein each point of the set of candidate points comprises a confidence score that satisfies a confidence threshold, wherein the layer is to cause each of a plurality of parallel suppression processes to be performed in connection with a respective candidate point, and wherein each of the plurality of parallel suppression processes is to identify a set of neighboring points that are within an area associated with the respective candidate point.

13. The system of claim 12 , wherein each of the plurality of parallel suppression processes is to:

calculate an intersection over union (IoU) value of the respective candidate point and a neighboring point in the identified set;

determine whether the IoU value satisfies an IoU threshold and a confidence score of the candidate point satisfies a criterion pertaining to a confidence score of the neighboring point; and

identify the candidate point as a redundant bounding box to be removed responsive to the IoU value satisfying the IoU threshold and the confidence score satisfying the criterion pertaining to the confidence score of the neighboring point.

14. The system of claim 12 , wherein, to identify the set of neighboring points, the parallel suppression process is to:

calculate a distance between the respective candidate point and a second point in the output feature map; and

identify the second point as a neighboring point responsive to the distance satisfying a distance threshold, wherein the distance threshold corresponds to the area associated with the respective candidate point.

15. The system of claim 14 , wherein, to calculate the distance, the parallel suppression process is to calculate a cosine distance between the respective candidate point and the second point.

16. The system of claim 9 , wherein the one or more processors are to perform two or more portions of a non-maximum suppression (NMS) algorithm in parallel and:

identify a set of candidate boxes comprising a set of candidate points from an output feature map comprising a plurality of points, each point in the output feature map corresponding to a first anchor box with a first confidence score and a second anchor box with a second confidence score, wherein at least the first confidence score satisfies a first confidence threshold or the second confidence score satisfies a second confidence threshold; and

initiate each of a plurality of parallel suppression processes in connection with a respective candidate point, and wherein each of the plurality of parallel suppression processes is to identify at least one of a first set of neighboring points that are within a first area associated the respective candidate point or a second set of neighboring points that are within a second area associated the respective candidate point.

17. A method comprising:

identifying a plurality of bounding boxes; and

performing one or more parallel comparisons to one of the plurality of bounding boxes based, at least in part,

on an intersection over union (IoU) of only a subset of the plurality of bounding boxes having centers within a maximum radius of pixels from a bounding box within the subset.

18. The method of claim 17 , further comprising performing two or more portions of a non-maximum suppression (NMS) algorithm in parallel by at least:

initiating a plurality of parallel suppression processes to remove one or more redundant bounding boxes corresponding to one or more objects; and

defining an area that covers a subset of a plurality of bounding boxes for each suppression process.

19. The method of claim 17 , further comprising performing two or more portions of a non-maximum suppression (NMS) algorithm in parallel by at least:

identifying a set of candidate points from an output feature map comprising a plurality of points, each point in the output feature map corresponding to a bounding box and comprising a confidence score, wherein each point of the set of candidate points comprises a confidence score that satisfies a confidence threshold; and

causing each of a plurality of parallel suppression processes to be performed in connection with a respective candidate point, wherein each of the plurality of parallel suppression processes is to identify a set of neighboring points that are within an area associated the respective candidate point.

20. The method of claim 19 , wherein performing the two or more portions of the NMS algorithm in parallel, for each of the plurality of parallel suppression processes, comprises:

calculating an intersection over union (IoU) value of the respective candidate point and a neighboring point in the identified set;

determining whether the IoU value is satisfies an IoU threshold and a confidence score of the candidate point satisfies a criterion pertaining to a confidence score of the neighboring point; and

identifying the candidate point as a redundant bounding box to be removed responsive to the IoU value satisfying than the IoU threshold and the confidence score satisfying the criterion pertaining to the confidence score of the neighboring point.

21. The method of claim 19 , wherein identifying the set of candidate points comprises:

calculating a distance between the respective candidate point and a second point in the output feature map; and

identifying the second point as a neighboring point responsive to the distance satisfying a distance threshold, wherein the distance threshold corresponds to the area associated with the respective candidate point.

22. The method of claim 21 , wherein calculating the distance comprises calculating a cosine distance between the respective candidate point and the second point.

23. The method of claim 17 , further comprising detecting, using a neural network, one or more objects within one or more images, and performing two or more portions of a non-maximum suppression (NMS) algorithm in parallel in a layer of the neural network.

24. The method of claim 17 , further comprising performing two or more portions of a non-maximum suppression (NMS) algorithm in parallel by at least:

identifying a set of candidate boxes comprising a set of candidate points from an output feature map comprising a plurality of points, each point in the output feature map corresponding to a first anchor box with a first confidence score and a second anchor box with a second confidence score, wherein at least one of the first confidence score satisfies a first confidence threshold or the second confidence score satisfies a second confidence threshold; and

initiating each of a plurality of parallel suppression processes in connection with a respective candidate point, wherein each of the plurality of parallel suppression processes is to identify at least one of a first set of neighboring points that are within a first area associated with the respective candidate point or a second set of neighboring points that are within a second area associated with the respective candidate point.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 4, 2021
From: SHEN, YICHUN; JIANG, WANLI; KWON, JUNGHYUN; LI, SIYI; PARK, MINWOO; OH, SANGMIN
To: NVIDIA CORPORATION
Reel/Frame 056446/0888 →
Continuity (1)
Related Publication 20220222477A1 · Jul 14, 2022