Time-series image description method for dam defects based on local self-attention
A time-series image description method for dam defects based on local self-attention mechanism is provided, including: performing frame sampling on an input time-series image of dam defect, extracting a feature sequence using a convolutional neural network and using the sequence as an input to a self-attention encoder, where the encoder includes a Transformer network based on a variable self-attention mechanism that dynamically establishes contextual feature relations for each frame; generating description text using a long short term memory (LSTM) network based on a local attention mechanism to enable each word predicted to be feature related to an image frame, improving text generation accuracy by establishing a contextual dependency between image and text. A dynamic mechanism is added to the present application for calculating the global self-attention of image frames, and LSTM networks with added local attention directly establish the correspondence between image and text modal data.
1 . A time-series image description method for dam defects based on local self-attention mechanism, comprising following steps:
(1) performing frame sampling on an input time-series image and extracting a feature sequence using a convolutional neural network;
(2) encoding the feature sequence of the time-series image using a Transformer network based on a variable self-attention mechanism to dynamically establish a contextual relation for each frame; and
(3) generating description text using a long short term memory (LSTM) network based on a local attention mechanism to enable each word predicted to focus on a corresponding image frame;
wherein in step (2), specific steps of encoding the feature sequence of the time-series image using a Transformer network based on a variable self-attention mechanism comprise:
(2.1) obtaining a query vector q, a keyword vector k and a value vector v corresponding to each sampled frame using a linear fully-connected layer:
q =Linear( X )= W Q X
k =Linear( X )= W K X
v =Linear( X )= W V X,
where vector q directs a current feature image to selectively focus on contextual features in a time-based dimension; vector k is used to calculate attention weights of a current feature map and other feature maps; and vector v is used to add information from the current feature map to a self-attention weight;
(2.2) adding a dot product of vector q and vector k to a current image block to obtain the attention weight as follows:
Attention
(
q
,
k
,
v
)
=
softmax
(
q
k
T
d
k
)
v
,
where d k is input vector dimension, obtained by dividing an input sequence dimension by a number of self-attention heads; vector q and vector k are dot-produced to obtain similarity scores of respective sequence elements, divided by √{square root over (d k )} for normalisation to ensure a stability of gradient propagation in the convolutional neural network;
(2.3) introducing a multi-headed deformable coding structure into the Transformer network, enabling a model to sample and calculate attention weight for only a set of key frames around a current frame, namely assigning a certain number of keyword vectors k to the query vector q for each element in the sequence:
Atten
(
z
q
,
p
q
,
X
t
)
=
∑
m
=
1
K
W
m
[
∑
k
=
1
K
A
m
q
k
·
W
m
′
x
v
(
p
q
+
Δ
p
m
q
k
)
]
,
where P q is a position reference point of the current frame, W m and W m ′ are weighted learnable feature matrices, Δp mqk and A mpk represent a sampling offset and a self-attention weight of a k th sampling point in a m th self-attention head, respectively, and are capable of being normalized as Σ k∈Ω A mpk =1, and are obtained by training through a fully-connected network and finally linearly projected into the query vector to obtain a sampled frame feature map {circumflex over (x)} t containing contextual information.
2 . The time-series image description method for dam defects based on local self-attention mechanism according to claim 1 , wherein in step (1), specific steps of performing frame sampling on an input time-series image and extracting a feature sequence using a convolutional neural network comprise:
(1.1) dividing the input time-series image into T′ segments of equal length without overlap, randomly selecting a frame X, from each segment to form a set of [x 1 , x 2 , . . . , x T ]; and
(1.2) using the convolutional neural network to process each sampled image frame, extracting a feature map as input to the self-attention encoder and recording as F t =[X 1 , X 2 , . . . , X T ], where X, is a feature representation of each sampled image frame.
3 . The time-series image description method for dam defects based on local self-attention mechanism according to claim 1 , wherein in step (3), specific steps of generating description text using the LSTM network based on the local attention mechanism comprise:
(3.1) {circumflex over (x)} t is a feature representation sequence for each sampled frame in the time-series image, calculating a conditional probability of generating each word by decoding the feature representation sequence to obtain a description text {S n } of a corresponding event:
log
p
(
S
n
❘
"\[LeftBracketingBar]"
x
n
)
=
∑
j
=
1
t
log
p
(
S
n
❘
"\[LeftBracketingBar]"
S
n
<
j
,
s
)
,
where S t represents a t th word of the text, and probability of each word is capable of being expressed as:
p ( S n,i |S n <j,s )=softmax( g ( h j )
h j =f ( h j-1 ,s ),
where h j is a hidden layer of a recurrent neural network, f is a function calculating a hidden state at a current position based on an output of the hidden layer at a previous position and a current vector, with an output converted into a vector of a same dimension as a vocabulary through a function g;
(3.2) introducing a context relation vector c t in a process of calculating the text, and splicing c t with a hidden layer state h t of the sequence then multiplying by a parameter matrix w c having learnable weights, and obtaining a hidden layer state with attention mechanism:
{tilde over (h)} =tan h ( W c [c t :h t ]),
finally, outputting a corresponding word sequence by a Softmax function and fully-connected neural network:
p ( S n,t |S n <j,x n )=softmax( W s ·{tilde over (h)} t )
(3.3) calculating a center position P t of current attention, namely the local attention mechanism, when generating each target word:
p t =S ·sigmoid( v p T tan h ( W p ·h t )),
enabling output word to focus on a position of an input sequence related, where a position matrix W p and a penalty term v p are both characteristic parameters with learnable weights, S is a length of the input sequence, then an attention window corresponding to the position is [p t −D, p t +D], and calculating hidden layer vectors of input and output sequences by an align function and constraining the hidden layer vectors by a Gaussian distribution to obtain the attention weight:
a
t
(
s
)
=
align
(
h
t
,
h
s
)
exp
(
-
(
s
-
p
t
)
2
2
σ
2
)
;
finally, concatenating the contextual features, the attention weight and words generated above as the input through the LSTM network, and obtaining output words at the current position by calculating through the fully-connected network and the Softmax activation function.
4 . A computer device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the computer program to implement the time-series image description method for dam defects based on local self-attention mechanism according to claim 1 .
5 . A computer device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the computer program to implement the time-series image description method for dam defects based on local self-attention mechanism according to claim 2 .
6 . A computer device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the computer program to implement the time-series image description method for dam defects based on local self-attention mechanism according to claim 3 .
7 . A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the time-series image description method for dam defects based on local self-attention mechanism according to claim 1 .
8 . A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the time-series image description method for dam defects based on local self-attention mechanism according to claim 2 .
9 . A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the time-series image description method for dam defects based on local self-attention mechanism according to claim 3 .