Counterfactual context-aware texture learning for camouflaged object detection
A counterfactual context-aware texture learning network system, including: a camera configured to capture an input image; a processor configured to perform camouflaged object detection on the input image; and a memory configured to store a texture-aware refinement module (TRM), a context-aware fused module (CFM), and a counterfactual intervention module (CIM); wherein the processor is configured to execute program instructions of the TRM, the CFM, and the CIM; the TRM is configured to extract dimension features from the input image; the CFM is configured to infuse multi-scale contextual features; the CIM is configured to identify a camouflaged object with counterfactual intervention via the processor; the TRM includes: a receptive field block (RFB) configured to expand a receptive field and extract texture features; and a position attention module (PAM) and a channel attention module (CAM) configured to further refine texture-aware features and obtain discriminant feature representation.
1. A counterfactual context-aware texture learning network system, comprising:
a camera configured to capture an input image;
a processor configured to perform camouflaged object detection on the input image; and
a memory configured to store a texture-aware refinement module (TRM), a context-aware fused module (CFM), and a counterfactual intervention module (CIM);
wherein the processor is configured to execute program instructions of the TRM, the CFM, and the CIM;
the TRM is configured to extract dimension features from the input image;
the CFM is configured to infuse multi-scale contextual features;
the CIM is configured to identify a camouflaged object with counterfactual intervention via the processor;
the TRM comprises:
a receptive field block (RFB) configured to expand a receptive field and extract texture features; and
a position attention module (PAM) and a channel attention module (CAM) configured to further refine texture-aware features and obtain discriminant feature representation;
the RFB comprises five branches b k , (k=1,2,3,4,5), each branch of the five branches comprising a 1×1 convolution operation to reduce a channel size to 64;
each branch where k>2 further comprises a 1×(2i−1) convolutional layer, a (2i−1)×1 convolutional layer, and a (2i−1)×(2i−1) convolutional layer, with a dilation rate of (2i−1), where i=k−1;
each branch where k>1 is concatenated, input into a second 1×1 convolution operation, and added with a branch of the five branches where k=1;
a result of the RFB is input into a Rectified Linear Unit (ReLU) activation function to obtain an output feature f i ′∈ C×H×W , where C, H and W represent a channel number, a channel height, and a channel width, respectively;
the output feature f′ is input into the PAM and the CAM,
the PAM is configured to:
obtain three feature maps B, C, and D through three convolution layers, where {B, C, and D}∈ C×H×W , and the three feature maps are reshaped to C×N ; and
multiply the transpose of B by C, and perform a softmax layer to calculate the spatial attention map sa∈ N×N :
s
a
i
j
=
exp
(
B
i
·
C
j
)
∑
i
=
1
N
exp
(
B
i
·
C
j
)
(
1
)
where sa ij denotes the j th position's impact on the i th position;
a loss function L=L BCE W +L IoU W is used to train the counterfactual context-aware texture learning network system to learn effective textures, where L BCE W is the weighted binary cross entropy (BCE) loss which restricts each pixel, and Lou is a weighted intersection-over-union (IoU) loss that focuses on a global structure; and
a total loss is formulated as:
L total =L ( Y,y )+λ L ( Y effect ,y ) (2)
where y is a ground truth, λ=0:1, L(Y, y) are main clues which learn general texture features, Y is a prediction of the main clues, and λL(Y effect , y) is a counterfactual term that penalizes a wrong prediction affected by contextual biases;
thereby performing the camouflaged object detection in the input image with enhanced accuracy.
2. The counterfactual context-aware texture learning network system of claim 1 , wherein the PAM is configured to:
multiply the transpose of sa by a matrix of the D feature map and reshape an aggregated attentive features result to C×H×W ;
multiply the aggregated attentive features result by a scale parameter η and apply an element-wise sum operation with the output feature f′ to obtain spatial feature maps f p ∈ C×H×W :
f
i
p
=
η
∑
j
=
1
N
(
s
a
i
j
D
j
)
+
f
′
(
3
)
where η is initialized as 0 and gradually learns more weight, and f p is a weighted sum at each position which enhances a semantic representation of the feature.
3. The counterfactual context-aware texture learning network system of claim 2 , wherein the CAM is configured to reshape the f i ′ to C×N , multiply a transpose of f′ by the f′ matrix, and apply a softmax layer to obtain channel attention maps ca∈ C×C :
c
a
i
j
=
exp
(
f
i
′
·
f
i
′
)
∑
i
=
1
N
exp
(
f
i
′
·
f
i
′
)
(
4
)
where ca ij denotes a j th channel's impact on an i th channel.
4. The counterfactual context-aware texture learning network system of claim 3 , wherein the CAM is configured to multiply a transpose of ca by the f′ matrix, reshape to C×H×W multiply by a scale parameter β, and apply an element-wise sum operation with the original input feature f′ to obtain channel feature maps f c ∈ C×H×W :
f
i
c
=
β
∑
j
=
1
C
(
c
a
i
j
f
j
′
)
+
f
′
(
5
)
where β gradually learns a weight from an initial value of 0, and CF is a weighted sum at all channels and original features, which models long-range semantic dependencies.
5. The counterfactual context-aware texture learning network system of claim 4 , wherein an element-wise sum operation is performed between f′, f s and f c to obtain final texture-aware refinement feature maps F r :
F r =f′+f s +f c (6).
6. The counterfactual context-aware texture learning network system of claim 5 , wherein the CFM comprises:
a coordinate attention module configured to model long-range dependencies; and
two multiscale channel attention (MSCA) modules configured to aggregate local and global contexts in two different scale branches.
7. The counterfactual context-aware texture learning network system of claim 6 , wherein the coordinate attention module is configured to:
up-sample a high-level feature F hig and perform an element-wise sum operation with a low-level feature F low ;
use two spatial extents of pooling kernels, (H, 1) and (1, W), to encode results along a horizontal coordinate and a vertical coordinate, respectively, calculate an output of a c-th channel at height h as:
z
c
h
(
h
)
=
1
W
∑
0
≤
i
<
W
x
c
(
h
,
i
)
;
(
7
)
and
calculate an output of the c-th channel at width w as:
z
c
w
(
w
)
=
1
H
∑
0
≤
j
<
H
x
c
(
j
,
w
)
(
8
)
where x c are features of the c-th channel.
8. The counterfactual context-aware texture learning network system of claim 7 , wherein the coordinate attention module is configured to:
obtain a pair of direction-aware feature maps by (6) and (7);
obtain intermediate feature maps
f
i
∈
ℝ
C
r
×
H
,
where r is a reduction ratio for controlling a block size, which are configured to extract spatial information in both a horizontal direction and a vertical direction as follows:
f i =δ(Conv 1×1 ( C ( z h ,z w ))) (9)
where C is a concatenation operation, Conv 1×1 is a shared 1×1 convolutional layer, and δ is a non-linear activation function.
9. The counterfactual context-aware texture learning network system of claim 8 , wherein the coordinate attention module is configured to:
split f i along a spatial dimension into two separate tensors
f
h
∈
ℝ
C
r
×
H
and
f
w
∈
ℝ
C
r
×
W
,
and utilize a second Conv 1×1 and sigmoid function to obtain attention weights g h and g w ; and
obtain direction-aware and position-sensitive features F dp as follows:
F c dp ( i,j )= x c ×g c h ( i )× g c w ( j ) (10).
10. The counterfactual context-aware texture learning network system of claim 9 , wherein the high-level feature F hig and the low-level feature F low are input into the two MSCA, and two-level fusion semantics are obtained.
11. The counterfactual context-aware texture learning network system of claim 10 , wherein each of the two MSCA are configured to:
obtain local contexts L(x)∈ C×H×W and global contexts G(x)∈ C×H×W as follows:
L ( F )= B ( PWC 2 (δ( B ( PWC 1 ( F ))) (11)
G ( F )= B ( PWC 2 (δ( B ( PWC 1 ( GAP ( F )))) (12)
where F are input features, PWC 1 and PWC 2 are point-wise convolutions (PWC), which exploit point-wise channel interactions for each spatial position where a kernel size of PWC 1 and PWC 2 are
C
r
×
C
×
1
×
1
and
C
r
×
C
×
1
×
1
respectively, B is a batch normalization layer, δ is an activation function of the ReLU, and GAP is a global average pooling;
obtain refined multi-scale features F′ of MSCA as follows:
F′=F⊗M ( F )= F ⊗σ( L ( F )⊕ G ( F )) (13)
where M(F)∈ C×H×W are generated attentional weights, ⊕ is an element-wise sum operation, σ is a sigmoid function, and ⊗ is an element-wise multiplication operation; and
obtain a fusion of contextual features F fusion as follows:
F fusion =(Conv 3×3 ( C ( F low ′,F dp ,F hig ′)) (14)
where F low ′ and F hig ′ are refined multi-scale features from the low-level feature and the high-level feature, respectively, and F dp are the aggregated features of the direction-aware and feature and position-sensitive feature.
12. The counterfactual context-aware texture learning network system of claim 11 , wherein the CIM is configured to calculate an intervention output prediction Y according to:
Y
T
¯
(
X
)
=
P
(
Y
❘
do
(
T
)
)
=
∑
c
P
(
Y
❘
do
(
T
¯
,
c
)
P
(
c
❘
X
)
(
15
)
where X is the input image, T are hypothetical textures, P is a pixel-image level classifier, and c is the contextual biases to be mitigated from X; and calculate a total effect of learned textures according to:
Y effect =Y T ( X )− Y T ( X ) (16).