Hierarchical occlusion module and unseen object amodal instance segmentation system and method using the same
The hierarchical occlusion inference method according to the exemplary embodiment of the present disclosure includes: deriving a bounding box feature of the object instance by receiving a region of interest color-depth FPN feature and the object region of interest feature derived from a cluttered scene image including at least one object instance, deriving a visible feature of the object instance by fusing the object region of interest feature and the bounding box feature, deriving an amodal feature of the object instance by fusing the object region of interest feature, the bounding box feature, and the visible feature, deriving an occlusion feature of the object instance by fusing the object region of interest feature, the bounding box feature, the visible feature, and the amodal feature, and inferring occlusion of an object instance by de-convoluting the occlusion feature of the object instance.
1 . A hierarchical occlusion inference method performed by at least one processor of a computer system, comprising:
acquiring an image feature and an object region of interest feature derived from a cluttered scene image including at least one object instance, by the computer system;
deriving a bounding box feature of the object instance from the image feature and the object region of interest feature, by the computer system;
deriving a visible feature of the object instance by fusing the object region of interest feature and the bounding box feature, by the computer system;
deriving an occlusion including region feature of the object instance by fusing the object region of interest feature and the visible feature, by the computer system; and
deriving an occlusion including region mask of the object instance from the occlusion including region feature, by the computer system,
wherein the deriving of a bounding box feature of the object instance from the image feature and the object region of interest feature, by the computer system, includes:
deriving the bounding box feature of the object instance based on a region of interest color-depth feature pyramid network (FPN) feature and the object region of interest feature derived from the cluttered scene image including at least one object instance, by the computer system.
2 . The hierarchical occlusion inference method according to claim 1 , wherein the deriving of an occlusion including region feature of the object instance by fusing the object region of interest feature and the visible feature, by the computer system, includes:
fusing the object region of interest feature and the visible feature in a hierarchical fusion module or a fully connected layer including a convolution layer, by the computer system.
3 . The hierarchical occlusion inference method according to claim 1 , wherein the deriving of an occlusion including region mask of the object instance from the occlusion including region feature, by the computer system, includes:
extracting the occlusion including region mask of the object instance by deconvoluting or computing the occlusion including region feature of the object instance in a fully connected layer, by the computer system.
4 . The hierarchical occlusion inference method according to claim 1 , wherein the deriving of a visible feature of the object instance by fusing the object region of interest feature and the bounding box feature, by the computer system, includes:
fusing the object region of interest feature and the bounding box feature in a hierarchical fusion module or a fully connected layer including a convolution layer, by the computer system.
5 . A hierarchical occlusion inference method performed by at least one processor of a computer system, comprising:
acquiring an image feature and an object region of interest feature derived from a cluttered scene image including at least one object instance, by the computer system;
deriving a bounding box feature of the object instance from the image feature and the object region of interest feature, by the computer system;
deriving a visible feature and an occlusion including region feature of the object instance by fusing the image feature and the bounding box feature, by the computer system;
deriving an occlusion feature of the object instance by fusing the object region of interest feature, the visible feature, and the occlusion including region feature, by the computer system; and
inferring occlusion of the object instance from the occlusion feature of the object instance, by the computer system,
wherein the deriving of a bounding box feature of the object instance from the image feature and the object region of interest feature, by the computer system, includes:
deriving the bounding box feature of the object instance based on a region of interest color-depth feature pyramid network (FPN) feature and the object region of interest feature derived from the cluttered scene image including at least one object instance, by the computer system.
6 . The hierarchical occlusion inference method according to claim 5 , wherein the deriving of an occlusion feature of the object instance by fusing the object region of interest feature, the visible feature, and the occlusion including region feature, by the computer system, includes:
fusing the object region of interest feature, the visible feature, and the occlusion including region feature in a hierarchical fusion module or a fully connected layer including a convolution layer, by the computer system.
7 . The hierarchical occlusion inference method according to claim 5 , wherein the inferring of occlusion of the object instance from the occlusion feature of the object instance, by the computer system, includes:
inferring the occlusion of the object instance by deconvoluting or computing the occlusion feature of the object instance in a fully connected layer, by the computer system.
8 . The hierarchical occlusion inference method according to claim 5 , wherein the image feature includes at least one of a color feature, a depth feature, and a color-depth fusion feature.
9 . The hierarchical occlusion inference method according to claim 5 , wherein the occlusion including region feature includes at least one of an amodal feature and an invisible feature.
10 . The hierarchical occlusion inference method according to claim 5 , further comprising:
extracting a class and a bounding box of the object instance based on the bounding box feature of the object instance, by the computer system.
11 . The hierarchical occlusion inference method according to claim 10 , wherein the extracting of a class and a bounding box of the object instance based on the bounding box feature of the object instance, by the computer system, includes:
extracting the class and the bounding box by supplying the bounding box feature to a fully connected layer, by the computer system.
12 . The hierarchical occlusion inference method according to claim 5 , further comprising:
extracting a visible mask of the object instance from the visible feature of the object instance, by the computer system.
13 . The hierarchical occlusion inference method according to claim 12 , wherein the extracting of a visible mask of the object instance from the visible feature of the object instance, by the computer system, includes:
extracting the visible mask of the object instance by deconvoluting or computing the visible feature of the object instance in a fully connected layer, by the computer system.
14 . An unseen object amodal instance segmentation system, comprising:
a processor; and
a computer readable medium including a command which allows the processor to execute an unseen object amodal instance segmentation network,
wherein the unseen object amodal instance segmentation network derives and fuses a color feature and a depth feature by receiving a cluttered scene image including at least one invisible object instance to generate a color-depth fusion feature, extracts a color-depth FPN feature by applying the color-depth fusion feature to a feature pyramid network and extracting an object region of interest by applying the color-depth fusion feature to a region proposal network, extracts a region of interest color-depth FPN feature and an object region of interest feature from the color-depth FPN feature and the object region of interest feature, derives a bounding box feature and a bounding box of the invisible object instance from the region of interest color-depth FPN feature and an object region of interest feature, derives a visible feature and a visible mask of the invisible object instance by fusing the object region of interest feature and the bounding box feature, derives an amodal feature and an amodal mask of the invisible object instance by fusing the object region of interest feature, the bounding box feature, and the visible feature, derives an occlusion feature and occlusion of the invisible object instance by fusing the object region of interest feature, the bounding box feature, the visible feature, and the amodal feature, and segments the invisible object instance based on the bounding box, the visible mask, the amodal mask, and occlusion of the invisible object instance.
15 . The unseen object amodal instance segmentation system according to claim 14 , wherein the unseen object amodal instance segmentation network extracts the bounding box of the invisible object instance by supplying the bounding box feature to a fully connected layer.
16 . The unseen object amodal instance segmentation system according to claim 14 , wherein the unseen object amodal instance segmentation network derives the visible feature of the invisible object instance by fusing the object region of interest feature and the bounding box feature in a hierarchical fusion module including a convolution layer, and extracts the visible mask of the invisible object instance by deconvoluting a visible feature of the invisible object instance.
17 . The unseen object amodal instance segmentation system according to claim 14 , wherein the unseen object amodal instance segmentation network derives the amodal feature of the invisible object instance by fusing the object region of interest feature, the bounding box feature, and the visible feature in a hierarchical fusion module including a convolution layer, and extracts the amodal mask of the invisible object instance by deconvoluting an amodal feature of the invisible object instance.
18 . The unseen object amodal instance segmentation system according to claim 14 , wherein the unseen object amodal instance segmentation network derives the occlusion feature of the invisible object instance by fusing the object region of interest feature, the bounding box feature, the visible feature, and the amodal feature in the hierarchical fusion module including the convolution layer and infers the occlusion of the invisible object instance by deconvoluting an occlusion feature of the invisible object instance.
19 . The unseen object amodal instance segmentation system according to claim 14 , further comprising:
a machine learning module which trains at least one model parameter of the unseen object amodal instance segmentation network to segment the invisible object instance from at least one cluttered scene image.