Instance segmentation inferred from machine learning model output
Techniques for using instance segmentation with machine learning (ML) models are discussed herein. An image can be provided as input to a ML model, which can generate, as an output from the ML model, a feature map comprising a plurality of features. Each feature of the plurality of features can comprise a confidence score, classification information, and a region of interest (ROI) determined in accordance with a non-maximal suppression (NMS) technique. Individual ROIs that are similar can be associated together for segmentation purposes. That is, instead of requiring a second ML model and/or a second operation to segment the image (e.g., identify which pixels correspond with the detected object, for example, by outputting a mask or set of lines and/or curves), the techniques discussed herein substantially simultaneously detect an object (e.g., determine an ROI) and segment the image.
1. A method comprising:
providing, as input to a machine-learning (ML) model, an image;
receiving, as output from the ML model, a feature map comprising a plurality of features, the plurality of features comprising a plurality of regions of interest (ROIs) and an individual feature of the plurality of features associated with a portion of the image and comprising a confidence score and a first region of interest (ROI);
determining, based at least in part on a non-maximal suppression (NMS) technique, an output ROI, the NMS technique comprising determining a subset of the plurality of ROIs; and
determining an instance segmentation based at least in part on determining portions of the image associated with the subset, the instance segmentation identifying a portion of the image as being associated with an object.
2. The computer-implemented method as claim 1 recites, wherein determining the instance segmentation is based at least in part on determining that the subset of ROIs are indicated as being suppressed via the NMS technique.
3. The computer-implemented method as claim 1 recites, wherein determining to output the output ROI comprises:
determining that the output ROI is associated with a maximum confidence score from among the confidence scores associated with the subset of the plurality of ROIs; and
outputting the output ROI in association with a region of the image as representing the object.
4. The computer-implemented method as claim 1 recites, further comprising outputting a mask corresponding to the instance segmentation to identify pixels of the image as representing the object.
5. The computer-implemented method as claim 4 recites, further comprising determining, based at least in part on at least one of the output ROI or the instance segmentation, a trajectory for controlling motion of an autonomous vehicle.
6. The computer-implemented method as claim 1 recites, further comprising training the ML model based at least in part on a test image and a ground truth associated therewith, wherein the ground truth identifies a mask.
7. The computer-implemented method as claim 1 recites, further comprising training the ML model to substantially simultaneously output multiple ROIs for each feature corresponding to multiple classifications, the multiple classifications comprising one or more of a vehicle, a pedestrian, or a bicycle.
8. The computer-implemented method as claim 1 recites, wherein the ROI is a bounding box.
9. A system comprising:
one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
providing, as input to a machine-learning (ML) model, an image;
receiving, from the ML model, a feature map comprising a plurality of features, a feature of the plurality of features comprising a first region of interest (ROI) and a first confidence score;
determining to output a second ROI and to suppress the first ROI based at least in part on a non-maximal suppression (NMS) technique; and
determining an instance segmentation based at least in part on one or more ROIs suppressed by the NMS technique, the instance segmentation comprising portions of the image associated with the one or more ROIs suppressed by the NMS technique and a portion of the image associated with the second ROI.
10. The system as claim 9 recites, wherein the NMS technique comprises:
determining that the first confidence score associated with the first ROI meets or exceeds a threshold confidence score;
determining a degree of alignment between the first ROI and the second ROI;
including the first ROI and the second ROI in a subset of ROIs based at least in part on determining that the degree of alignment meets or exceeds a threshold degree of alignment; and
determining to output the second ROI based at least in part on determining that a second confidence score associated with the second ROI meets or exceeds the first confidence score.
11. The system as claim 9 recites, wherein the operations further comprise outputting the instance segmentation as a mask that identifies pixels of the image as representing an object associated with an object classification.
12. The system as claim 9 recites, the operations further comprising:
determining, based at least in part on the instance segmentation, a trajectory configured to cause an autonomous vehicle to traverse a portion of an environment; and
providing the trajectory to the autonomous vehicle.
13. The system as claim 9 recites, the operations further comprising:
training the ML model based at least in part on providing, as input to the ML model and substantially simultaneously, a batch of images that includes a first predefined number of images that are associated with a first object classification and a second predefined number of images that are associated with a second object classification,
wherein the first predefined number and the second predefined number are determined based at least in part on one or more confidence scores associated with the plurality of features.
14. The system as claim 9 recites, the operations further comprising:
training the ML model based at least in part on a test image and a ground truth associated therewith, wherein the ground truth identifies a reference region and a mask.
15. A non-transitory computer-readable medium having a set of instructions that, when executed, cause one or more processors to perform operations comprising:
receiving, at a machine-learning (ML) model, an image;
determining, by the ML model, a feature map comprising a plurality of features, a feature of the plurality of features comprising a region of interest (ROI) and an identification that the ROI was suppressed according to a non-maximal suppression (NMS) technique; and
determining, based at least in part on the identification, an instance segmentation indicating one or more discrete portions of the image as being associated with an object, the instance segmentation comprising a portion of the image associated with the ROI.
16. The non-transitory computer-readable medium as claim 15 recites, wherein the feature map has dimensions that are a fraction of dimensions of the image.
17. The non-transitory computer-readable medium as claim 15 recites, the operations further comprising:
training the ML model based at least in part on a test image and a ground truth associated therewith, wherein the ground truth identifies a reference region and a mask.
18. The non-transitory computer-readable medium as claim 17 recites, wherein the training the ML model comprises providing, as input to the ML model, a batch of images that comprises a first predefined number of images that are associated with a first object classification and a second predefined number of images that are associated with a second object classification,
wherein the first predefined number and the second predefined number are based at least in part on one more confidence levels of the plurality of features.
19. The non-transitory computer-readable medium as claim 15 recites, wherein the NMS technique comprises:
determining that the first confidence score associated with the first ROI meets or exceeds a threshold confidence score;
determining a degree of alignment between the first ROI and the second ROI;
including the first ROI and the second ROI in a subset of ROIs based at least in part on determining that the degree of alignment meets or exceeds a threshold degree of alignment; and
determining to output the second ROI based at least in part on determining that a second confidence score associated with the second ROI meets or exceeds the first confidence score.
20. The non-transitory computer-readable medium as claim 19 recites, wherein the one or more discrete portions of the image are associated with the subset of ROIs.