Granted Patent B2

US 12,205,338 · App. 17/518,026 · Granted Jan 21, 2025

Inventors: Jihye Kim (Anyang-si, KR); Raymond Alexander Yeh (Urbana, IL); Alexander Gerhard Schwing (Urbana, IL); Yuan-Ting Hu (Urbana, IL)

Assignees: THE BOARD OF TRUSTEES OF THE UNIVERSITY OF ILLINOIS; Samsung Electronics Co., Ltd.

G06V10/25G06T7/11G06V10/24

Abstract

Claims (62)

1. A segmentation method comprising:

receiving image frames comprising a current frame and an adjacent frame to the current frame;

determining a feature map to aggregate the image frames based on temporal information between the current frame and the adjacent frame;

extracting a feature of a region of interest (ROI) corresponding to instances included in the current frame from the feature map;

predicting a class of an object corresponding to the ROI based on the feature of the ROI; and

segmenting the instances by correcting an amodal mask predicted corresponding to the class of the object based on the feature of the ROI, including weighting the amodal mask by a confidence map and segmenting the instances based on the weighted amodal mask.

2. The segmentation method of claim 1 , wherein the segmenting of the instances comprises:

predicting an amodal mask corresponding to the class of the object based on the feature of the ROI; and

performing the segmentation of the instances by repeatedly applying the predicted amodal mask to the feature of the ROI.

3. The segmentation method of claim 2 , wherein the predicting of the amodal mask comprises propagating the feature of the ROI from a visible area of a target instance corresponding to the class of the object to an occluded area of the target instance and predicting the amodal mask corresponding to the target instance.

4. The segmentation method of claim 3 , wherein the predicting of the amodal mask comprises:

spatially propagating the feature of the ROI by transferring a feature corresponding to the visible area through convolution layers and expanding a receptive field to the occluded area;

expanding a spatial dimension of the feature of the ROI using deconvolution layers; and

predicting the amodal mask corresponding to the target instance in the expanded spatial dimension.

5. The segmentation method of claim 1 , wherein the segmenting of the instances comprises:

repeatedly predicting the amodal mask by spatially propagating the feature of the ROI from a visible area of a target instance corresponding to the class of object class to an occluded area of the target instance;

predicting a modal mask corresponding to the visible area based on the feature of the ROI;

predicting an occluded mask corresponding to the occluded area based on the feature of the ROI; and

performing the segmentation of the instances based on a combination of the amodal mask, the modal mask, and the occluded mask.

6. The segmentation method of claim 5 , wherein the segmenting of the instances based on the combination of the amodal mask, the modal mask, and the occluded mask comprises:

calculating a first confidence corresponding to a pixel-wise probability of the modal mask;

calculating a second confidence corresponding to a pixel-wise probability of the occluded mask;

weighting the amodal mask by a confidence map based on at least one of the first confidence or the second confidence; and

performing the segmentation of the instances using the weighted amodal mask.

7. The segmentation method of claim 5 , wherein the spatially propagating of the feature of the ROI comprises transferring a feature corresponding to the visible area through convolution layers to the occluded area.

8. The segmentation method of claim 1 , wherein the segmenting of the instances comprises:

predicting an initial attention mask corresponding to the class of the object based on the feature of the ROI;

extracting an initial mask corresponding to the class of the object from the initial attention mask;

generating the amodal mask by repeatedly applying the feature of the ROI to the initial mask; and

segmenting the instances using the amodal mask.

9. The segmentation method of claim 8 , wherein the generating of the amodal mask comprises:

performing first masking by applying the initial mask to the feature of the ROI;

predicting an attention mask corresponding to the object class based on a first masked feature generated through the first masking;

performing second masking by applying the attention mask to the feature of the ROI; and

generating the amodal mask based on a second masked feature generated through the second masking.

10. The segmentation method of claim 1 , wherein the extracting of the feature of the ROI comprises extracting the feature of the ROI from the feature map using a region proposal network (RPN).

11. The segmentation method of claim 1 , wherein the extracting of the feature of the ROI comprises:

selecting an instance comprising an occluded area in the current frame from among the instances; and

extracting a feature of the ROI corresponding to the selected instance from the feature map using a region proposal network (RPN).

12. The segmentation method of claim 11 , wherein the extracting of the feature of the ROI corresponding to the selected instance comprises:

filtering bounding boxes respectively corresponding to the instances using a cascade structure based on a non-maximum suppression (NMS) scheme, the bounding boxes comprising locations of the instances and objectness scores corresponding to the instances; and

extracting the feature of the ROI corresponding to the selected instance from the feature map based on the filtered bounding boxes.

13. The segmentation method of claim 1 , wherein the predicting of the class of the object comprises:

calculating scores of all object classes included in the ROI by bounding boxes respectively corresponding to the instances based on the feature of the ROI; and

predicting the class of the object based on the scores of all the object classes.

14. The segmentation method of claim 1 , wherein the determining of the feature map comprises:

extracting features corresponding to the image frames;

spatially aligning the features with the current frame via warping between the current frame and the adjacent frame; and

calculating the feature map by aggregating the aligned features.

15. The segmentation method of claim 14 , wherein the spatially aligning of the features comprises:

estimating an optical flow that represents a pixel-level motion between the current frame and the adjacent frame; and

spatially aligning the features with the current frame via the warping, based on the optical flow.

16. The segmentation method of claim 1 , wherein the segmenting of the instances comprise repetitively segmenting the instances by correcting the amodal mask predicted corresponding to the class of the object based on the feature of the ROI.

17. The segmentation method of claim 1 , wherein the adjacent frame comprises a frame temporally previous to the current frame or a frame temporally following the current frame.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the segmentation method of claim 1 .

19. A segmentation apparatus comprising:

a communication interface configured to receive image frames comprising a current frame and at least one adjacent frame adjacent to the current frame; and

a processor configured to determine a feature map to aggregate the image frames based on temporal information between the current frame and the adjacent frame, to extract a feature of a region of interest (ROI) corresponding to instances included in the current frame from the feature map, to predict a class of an object corresponding to the ROI based on the feature of the ROI, and to segment the instances by correcting an amodal mask predicted corresponding to the class of the object based on the feature of the ROI, including weighting the amodal mask by a confidence map and segmenting the instances based on the weighted amodal mask.

20. The segmentation apparatus of claim 19 , wherein the processor is further configured to predict an amodal mask corresponding to the class of the object based on the feature of the ROI, and to perform the segmentation of the instances by repeatedly applying the predicted amodal mask to the feature of the ROI.

21. The segmentation apparatus of claim 20 , wherein the processor is further configured to propagate the feature of the ROI from a visible area of a target instance corresponding to the class of the object to an occluded area of the target instance, and to predict the amodal mask corresponding to the target instance.

22. The segmentation apparatus of claim 19 , wherein the processor is further configured to repeatedly predict the amodal mask by spatially propagating the feature of the ROI from a visible area of a target instance corresponding to the class of the object to an occluded area of the target instance, to predict a modal mask corresponding to the visible area based on the feature of the ROI, to predict an occluded mask corresponding to the occluded area based on the feature of the ROI, and to perform the segmentation of the instances based on a combination of the amodal mask, the modal mask, and the occluded mask.

23. The segmentation apparatus of claim 19 , wherein the processor is further configured to predict an initial attention mask corresponding to the class of the object based on the feature of the ROI, to extract an initial mask corresponding to the class of the object from the initial attention mask, to generate the amodal mask by repeatedly applying the feature of the ROI to the initial mask, and to segment the instances using the amodal mask.

Assignments (2)

Priority Claims (1)

Continuity (2)

References Cited (12)

Cited By (1)

US 12,462,390