System and methods for multiple instance segmentation and tracking
This disclosure provides for methods and a system for multiple instance segmentation and tracking. According to an aspect a method is provided. The method includes sending an image to a backbone network and generating image feature outputs. The method further includes sending the image feature outputs to a spatial attention module for generating a feature map associated with objects in the image. The method further includes sending the feature map to a category feature module for generating an instance category output indicating the objects. The method further includes sending the image feature outputs to a mask generating module for generating masks. The method further includes generating: the instance category output via the category feature module, and the masks via the mask generating module. In some embodiments, the method further includes generating re-identification embedding information associated with the objects based on image feature outputs.
1. A method comprising:
sending an input image to a backbone network;
generating, via the backbone network, one or more image feature outputs;
sending the one or more image feature outputs to a spatial attention module for generating a feature map associated with one or more objects in the input image;
sending the feature map to a category feature module for generating an instance category output indicating the one or more objects;
sending the one or more image feature outputs directly from the backbone network to a mask generating module for generating one or more masks, each associated with an object in the input image; and
generating:
the instance category output via the category feature module; and
the one or more masks via the mask generating module.
2. The method of claim 1 , wherein the generating the one or more masks via the mask generating module comprises:
sending the one or more image feature outputs to an instance mask feature module for generating a second set of one or more masks, each associated with an object in the input image;
sending the second set of one or more masks to a mask refinement module for generating the one or more masks; and
generating the one or more masks via the mask refinement module.
3. The method of claim 2 , wherein:
the one or more masks is generated based on one or more low-level image features included in the one or more image feature outputs; and
the generating the one or more masks via the mask refinement module comprises:
sending the one or more low-level features to the mask refinement module; and
concatenating the second set of one or more masks with the one or more low-level features.
4. The method of claim 3 , wherein the second set of one or more masks are trained based on Tversky loss to generate dilated masks.
5. The method of claim 4 , wherein the one or more masks are trained based on one or more of edge loss and dice loss.
6. The method of claim 3 , wherein each of the one or more mask is an instance mask and:
corresponds to an object in the input image; and
comprises a plurality of pixels, each indicating a probability of belonging to the object in the input image.
7. The method of claim 1 , wherein the instance category output indicates the one or more object via assigning a higher score to portions of the input image comprising the one or more object than non-object portions.
8. The method of claim 1 further comprising:
sending the one or more image feature outputs to a re-identification (Re-ID) feature module for generating Re-ID embedding information associated with the one or more objects in the input image; and
generating the Re-ID embedding information via the Re-ID feature module.
9. The method of claim 8 further comprising:
sending, to a tracking and matching module, tracking information comprising one or more of: the Re-ID embedding information, the instance category output, and the one or more masks; and
tracking, by the tracking and matching module, the one or more objects based on the tracking information.
10. The method of claim 1 , wherein the backbone network is a deep layer aggregation network.
11. A method comprising:
sending an input image to a backbone network;
generating, via the backbone network, one or more image feature outputs;
sending the one or more image feature outputs to a category indicating module for generating an instance category output indicating one or more objects in the input image;
sending the one or more image feature outputs directly from the backbone network to an instance mask feature module for generating a first set of one or more masks, each associated with an object in the input image;
sending the first set of one or more masks to a mask refinement module for generating a second set of one or more masks; and
generating:
the instance category output via the category indicating module; and
the second set of one or more masks via the mask refinement module.
12. The method of claim 11 , wherein the generating the instance category output via the category indicating module comprises:
sending the one or more image feature outputs to a spatial attention module for generating a feature map indicating the one or more objects in the input image; and
sending the feature map to a category feature module for generating the instance category output;
generating the instance category output via the category feature module.
13. The method of claim 11 , wherein:
the second set of one or more masks is generated based on one or more low-level image features included in the one or more image feature outputs; and
the generating the second set of one or more masks via the mask refinement module comprises:
sending the one or more low-level features to the mask refinement module; and
concatenating the first set of one or more masks with the one or more low-level features.
14. Method of claim 11 further comprising:
sending the one or more image feature outputs to a re-identification (Re-ID) feature module for generating Re-ID embedding information associated with the one or more objects in the input image; and
generating the Re-ID embedding information via the Re-ID feature module.
15. The method of claim 14 further comprising:
sending, to a tracking and matching module, tracking information comprising one or more of: the Re-ID embedding information, the instance category output, and the second set of one or more masks; and
tracking, by the tracking and matching module, the one or more objects based the tracking information.
16. A method comprising:
sending an input image to a backbone network;
generating, via the backbone network, one or more image feature outputs;
sending the one or more image feature outputs to a category indicating module for generating an instance category output indicating one or more objects in the input image;
sending the one or more image feature outputs directly from the backbone network to a mask generating module for generating one or more masks, each associated with an object in the input image;
sending the one or more image feature outputs to a re-identification (Re-ID) feature module for generating Re-ID embedding information associated with the one or more objects in the input image; and
generating:
the instance category output via the category indicating module;
the one or more masks via the mask generating module; and
the Re-ID embedding information via the Re-ID feature module.
17. The method of claim 16 , wherein the generating the instance category output via the category indicating module comprises:
sending the one or more image feature outputs to a spatial attention module for generating a feature map indicating the one or more objects in the input image;
sending the feature map to a category feature module for generating the instance category output; and
generating the instance category output via the category feature module.
18. The method of claim 16 , wherein the generating the one or more masks via the mask generating module comprises:
sending the one or more image feature outputs to an instance mask feature module for generating a second set of one or more masks, each associated with an object in the input image;
sending the second set of one or more masks to a mask refinement module for generating the one or more masks; and
generating the one or more masks via the mask refinement module.
19. The method of claim 18 , wherein:
the one or more masks is generated based on one or more low-level image features included in the one or more image feature outputs; and
the generating the one or more masks via the mask refinement module comprises:
sending the one or more low-level image features to the mask refinement module;
concatenating the second set of one or more masks with the one or more low-level features.
20. The method of claim 16 further comprising:
sending, to a tracking and matching module, tracking information comprising one or more of: the Re-ID embedding information, the instance category output, and the one or more masks; and
tracking, by the tracking and matching module, the one or more objects based on the tracking information.