Method and system for identifying human interactive behavior based on multi-view comparison

1 . A method for identifying human interactive behavior based on multi-view comparison, comprising:

acquiring position information of human joints in each frame of video data;

constructing a skeleton spatio-temporal graph based on the position information of human joints in each frame;

based on the skeleton spatio-temporal graph, adaptively deleting an edge and a node of the skeleton spatio-temporal graph through a graph convolution neural network, and constructing an enhanced node-dropping view and an enhanced edge-dropping view;

adopting an information bottleneck principle, increasing a difference between an enhanced view and an original skeleton spatio-temporal graph, simultaneously maximizing information related to a behavior recognition task, and reserving minimum enough information for the behavior recognition task in each view to obtain a multi-view representation; wherein the enhanced view is selected from the group consisting of: (i) the enhanced node-dropping view and (ii) the enhanced edge-dropping view; and

performing classification based on the multi-view representation to obtain a human interactive behavior recognition result;

wherein the adopting an information bottleneck principle, increasing a difference between one of the enhanced node-dropping view and the enhanced edge-dropping view and an original skeleton spatio-temporal graph, simultaneously maximizing information related to a behavior recognition task specifically comprises:

minimizing mutual information between one of the enhanced node-dropping view and the enhanced edge-dropping view and the original skeleton spatio-temporal graph by using negative contrastive learning loss to remove redundant information in each view and reserve remaining information;

wherein expressions for the minimizing mutual information between the enhanced view and the original skeleton spatio-temporal graph by using negative contrastive learning loss to remove redundant information in each view and reserve remaining information are:

min

(

E

,

E

~

)

⁢

L

~

rec

+

I

⁡

(

E

n

;

E

~

n

)

+

I

⁡

(

E

e

;

E

~

e

)

,

I

⁡

(

E

n

;

E

~

n

)

=

∑

v

i

∈

V

log

⁢

exp

⁡

(

s

⁡

(

v

i

,

v

~

i

)

)

∑

v

j

∈

v

exp

⁡

(

s

⁡

(

v

i

,

v

~

j

)

)

,

I

⁡

(

E

e

;

E

~

e

)

=

e

i

∈

u

⁢

log

⁢

exp

⁡

(

s

⁡

(

e

i

,

e

~

i

)

)

∑

e

j

∈

u

exp

⁡

(

s

⁡

(

e

i

,

e

~

j

)

)

,

where E represents the original skeleton spatio-temporal graph, {tilde over (E)} represents one of the enhanced node-dropping view and the enhanced edge-dropping view, and {tilde over (L)} rec represents a Bayesian personalized ranking (BPR) loss, such that the difference between the enhanced view and the original skeleton spatio-temporal graph is maximized, and I(E n ; {tilde over (E)} n ) represents mutual information between two corresponding node views, and I(E e ; {tilde over (E)} e ) represents mutual information between two corresponding edge views, s(⋅) represents similarity between two vectors, E n and E e are original views without node and edge deleted, {tilde over (E)} n and {tilde over (E)} e are the enhanced node-dropping and edge-dropping views, v i and {tilde over (v)} i are vector representations of corresponding nodes in the corresponding node views, e i and {tilde over (e)} j are vector representations corresponding edges in the corresponding edge views.

2 . The method according to claim 1 , wherein the constructing a skeleton spatio-temporal graph based on the position information of human joints in each frame specifically comprises:

in a spatial dimension, determining spatial positions of the human joints according to coordinate information of the human joints in each frame, and then drawing corresponding edges according to a natural structure of a human body to obtain a spatial topological graph of a skeleton sequence; and

after constructing the spatial topological graph of the skeleton sequence, connecting nodes representing a same joint in adjacent frames to form a skeleton spatio-temporal sequence graph as the skeleton spatio-temporal graph.

3 . The method according to claim 1 , wherein the based on the skeleton spatio-temporal graph, adaptively deleting an edge and a node of the skeleton spatio-temporal graph through a graph convolution neural network, and constructing an enhanced node-dropping view and an enhanced edge-dropping view specifically comprises:

learning a deletable node of each layer of the graph convolution neural network, and creating a node-dropping view as the enhanced node-dropping view after shielding an influential node; and

learning a deletable edge of each layer of the graph convolution neural network, and creating an edge-dropping view as the enhanced edge-dropping view after filtering out a noise edge.

4 . The method according to claim 3 , wherein an expression for the learning a deletable node at each layer of the graph convolution neural network, and creating a node-dropping view as the enhanced edge-dropping view after shielding an influential node is:

G

ND

(

l

)

=

{

{

v

i

⊙

ρ

i

(

l

)

❘

v

i

∈

V

}

,

ε

}

,

where v i represents an i-th node in an I-th network layer, and

ρ

i

(

l

)

∈

(

0

,

1

)

 comes from a Bernoulli distribution parameterized as

ω

i

(

l

)

,

 indicating whether the node v i is reserved or not, and ε represents a set of edges.

5 . The method according to claim 1 , wherein the skeleton spatio-temporal graph input the graph convolution neural network is in a form of an adjacency matrix with a size of n*n, where n represents a number of nodes in the skeleton spatio-temporal graph; when there is a connection between two nodes, an element value of a corresponding position in the adjacency matrix is 1, and vice versa is 0.

6 . A non-transitory computer-readable storage medium, stored with a computer program, wherein the computer program is configured to, when being executed by a processor, implement steps in the method for identifying human interactive behavior based on multi-view comparison according to claim 1 .

7 . A computer device, comprising a memory, a processor and a computer program stored in the memory and capable of being run on the processor, wherein the processor is configured to, when executing the computer program, implement the steps in the method for identifying human interactive behavior based on multi-view comparison according to claim 1 .