Granted Patent B2

US 12,412,426 · App. 17/999,284 · Granted Sep 9, 2025

Inventor: Binquan Li (Guangzhou, CN)

Assignee: BIGO TECHNOLOGY PTE. LTD.

G06V40/20G06N3/045G06N3/08G06V10/764G06V10/7715G06V10/806G06V10/82G06V20/40G06V20/41G06V20/44G06V20/46G06V20/49H04N19/172H04N19/513H04N19/52

Abstract

Claims (85)

1. An action recognition method, comprising:

acquiring grouped video data by grouping original compressed video data;

inputting the grouped video data into a first predetermined model and determining, based on an output result of the first predetermined model, target grouped video data containing an action;

acquiring to-be-recognized grouped video data by decoding the target grouped video data; and

inputting the to-be-recognized grouped video data into a second predetermined model and determining, based on an output result of the second predetermined model, a type of an action contained in the to-be-recognized grouped video data;

wherein acquiring the to-be-recognized grouped video data by decoding the target grouped video data comprises:

acquiring a to-be-recognized segmented video image by decoding the target grouped video data;

acquiring frequency domain information in the to-be-recognized segmented video image and generating a corresponding frequency domain graph based on the frequency domain information; and

determining the to-be-recognized segmented video image and the corresponding frequency domain graph as the to-be-recognized grouped video data.

2. The method according to claim 1 , wherein acquiring the grouped video data by grouping the original compressed video data comprises:

acquiring interval compressed videos by performing interval division on the original compressed video data based on a predetermined grouping rule; and

acquiring the grouped video data by extracting I frame data and P frame data in each of the interval compressed videos based on a predetermined extraction policy, wherein the P frame data comprises at least one of the following information corresponding to a P frame: motion vector information and pigment change residual information.

3. The method according to claim 2 , wherein the first predetermined model comprises a first 2D residual network, a first concatenation layer, and a first fully connected layer; wherein

feature maps with a same dimension are acquired by inputting the grouped video data into the first predetermined model and processing the grouped video data by the first 2D residual network;

a concatenated feature map is acquired by concatenating the feature maps by the first concatenation layer based on a frame sequence; and

a classification result indicating whether an action is contained is acquired by processing the concatenated feature map by the first fully connected layer.

4. The method according to claim 3 , wherein

acquiring the grouped video data by extracting the I frame data and the P frame data in each of the interval compressed videos based on the predetermined extraction policy comprises:

extracting the I frame data and the P frame data in each of the interval compressed videos based on the predetermined extraction policy;

cumulatively transforming the P frame data such that transformed P frame data depends on forward adjacent I frame data; and

determining the grouped video data based on the I frame data and the transformed P frame data; and

the first predetermined model further comprises an addition layer located before the concatenation layer, a feature map corresponding to the P frame data in the feature maps is denoted as a P frame feature map, and a feature map corresponding to the I frame data in the feature maps is denoted as an I frame feature map; wherein

a P frame feature map to which the I frame feature map is added is acquired by processing the P frame feature map and the I frame feature map by the addition layer; and

the concatenated feature map is acquired by concatenating the I frame feature map and the P frame feature map acquired upon the addition by the first concatenation layer based on the frame sequence.

5. The method according to claim 3 , wherein

a feature shift layer is comprised in the first predetermined model, wherein the feature shift layer is before a residual structure of the first 2D residual network; and

at least one of up-shifted grouped video data and down-shifted grouped video data is acquired by processing the grouped video data by the feature shift layer prior to being input into the residual structure.

6. The method according to claim 2 , wherein extracting the P frame data in each of the interval compressed video based on the predetermined extraction policy comprises:

extracting a predetermined number of P frame data in each of the interval compressed videos in an equal-interval manner;

wherein a predetermined number of P frame data in each of the interval compressed videos is extracted in a random-interval manner during training of the first predetermined model.

7. The method according to claim 1 , wherein the second predetermined model comprises a model based on an efficient convolutional network for online video understanding (ECO) architecture.

8. The method according to claim 7 , wherein the second predetermined model comprises a second concatenation layer, a second 2D residual network, a 3D residual network, a third concatenation layer, and a second fully connected layer; wherein

concatenated image data is acquired by inputting the to-be-recognized grouped video data into the second predetermined model and concatenating the to-be-recognized segmented video image and the corresponding frequency domain graph by the second concatenation layer;

a 2D feature map is acquired by processing the concatenated image data by the second 2D residual network;

a 3D feature map is acquired by inputting an intermediate layer output result of the second 2D residual network into the 3D residual network and processing the intermediate layer output result by the 3D residual network;

a to-be-recognized feature map is acquired by concatenating the 2D feature map and the 3D feature map by the third concatenation layer; and

a corresponding action type label is acquired by processing the to-be-recognized feature map by the second fully connected layer.

9. The method according to claim 8 , wherein the second predetermined model further comprises a first pooling layer and a second pooling layer; wherein

a one-dimensional 2D eigenvector containing a first number of elements is acquired by processing the 2D feature by the first pooling layer prior to being input into the third concatenation layer;

a one-dimensional 3D eigenvector containing a second number of elements is acquired by processing the 3D feature map by the second pooling layer prior to being input into the third concatenation layer; and

acquiring the to-be-recognized concatenated feature map by concatenating the 2D feature map and the 3D feature map by the third concatenation layer comprises:

acquiring a to-be-recognized concatenated vector by concatenating the one-dimensional 2D eigenvector and the one-dimensional 3D eigenvector by the third concatenation layer.

10. The method according to claim 9 , wherein the first pooling layer comprises a multi-receptive-field pooling layer.

11. The method according to claim 10 , wherein the first pooling layer comprises a first-level local pooling layer, a second-level global pooling layer, and a vector merging layer, the first-level local pooling layer containing at least two pooling kernels with different sizes; wherein

at least two groups of 2D pooled feature maps with different scales are acquired by processing the 2D feature map by the first-level local pooling layer;

at least two groups of eigenvectors are acquired by processing the at least two groups of 2D pooled feature maps with different scales by the second-level global pooling layer; and

the one-dimensional 2D eigenvector containing the first number of elements is acquired by processing the at least two groups of eigenvectors by the vector merging layer.

12. An action recognition computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when loading and executing the computer program, is caused to perform:

acquiring grouped video data by grouping original compressed video data;

inputting the grouped video data into a first predetermined model and determining, based on an output result of the first predetermined model, target grouped video data containing an action;

acquiring to-be-recognized grouped video data by decoding the target grouped video data; and

inputting the to-be-recognized grouped video data into a second predetermined model and determining, based on an output result of the second predetermined model, a type of an action contained in the to-be-recognized grouped video data;

wherein acquiring the to-be-recognized grouped video data by decoding the target grouped video data comprises:

acquiring a to-be-recognized segmented video image by decoding the target grouped video data;

acquiring frequency domain information in the to-be-recognized segmented video image and generating a corresponding frequency domain graph based on the frequency domain information; and

determining the to-be-recognized segmented video image and the corresponding frequency domain graph as the to-be-recognized grouped video data.

13. A non-transitory computer-readable storage medium storing a computer program therein, wherein the computer program, when loaded and executed by a processor, causes the processor to perform:

acquiring grouped video data by grouping original compressed video data;

inputting the grouped video data into a first predetermined model and determining, based on an output result of the first predetermined model, target grouped video data containing an action;

acquiring to-be-recognized grouped video data by decoding the target grouped video data; and

inputting the to-be-recognized grouped video data into a second predetermined model and determining, based on an output result of the second predetermined model, a type of an action contained in the to-be-recognized grouped video data;

wherein acquiring the to-be-recognized grouped video data by decoding the target grouped video data comprises:

acquiring a to-be-recognized segmented video image by decoding the target grouped video data;

acquiring frequency domain information in the to-be-recognized segmented video image and generating a corresponding frequency domain graph based on the frequency domain information; and

determining the to-be-recognized segmented video image and the corresponding frequency domain graph as the to-be-recognized grouped video data.

14. The action recognition computer device according to claim 12 , wherein the processor, when loading and executing the computer program, is caused to perform:

acquiring interval compressed videos by performing interval division on the original compressed video data based on a predetermined grouping rule; and

acquiring the grouped video data by extracting I frame data and P frame data in each of the interval compressed videos based on a predetermined extraction policy, wherein the P frame data comprises at least one of the following information corresponding to a P frame: motion vector information and pigment change residual information.

15. The action recognition computer device according to claim 14 , wherein the first predetermined model comprises a first 2D residual network, a first concatenation layer, and a first fully connected layer; wherein

feature maps with a same dimension are acquired by inputting the grouped video data into the first predetermined model and processing the grouped video data by the first 2D residual network;

a concatenated feature map is acquired by concatenating the feature maps by the first concatenation layer based on a frame sequence; and

a classification result indicating whether an action is contained is acquired by processing the concatenated feature map by the first fully connected layer.

16. The action recognition computer device according to claim 15 , wherein the processor, when loading and executing the computer program, is caused to perform:

extracting the I frame data and the P frame data in each of the interval compressed videos based on the predetermined extraction policy;

cumulatively transforming the P frame data such that transformed P frame data depends on forward adjacent I frame data; and

determining the grouped video data based on the I frame data and the transformed P frame data; and

the first predetermined model further comprises an addition layer located before the concatenation layer, a feature map corresponding to the P frame data in the feature maps is denoted as a P frame feature map, and a feature map corresponding to the I frame data in the feature maps is denoted as an I frame feature map; wherein

a P frame feature map to which the I frame feature map is added is acquired by processing the P frame feature map and the I frame feature map by the addition layer; and

the concatenated feature map is acquired by concatenating the I frame feature map and the P frame feature map acquired upon the addition by the first concatenation layer based on the frame sequence.

17. The action recognition computer device according to claim 15 , wherein

a feature shift layer is comprised in the first predetermined model, wherein the feature shift layer is before a residual structure of the first 2D residual network; and

at least one of up-shifted grouped video data and down-shifted grouped video data is acquired by processing the grouped video data by the feature shift layer prior to being input into the residual structure.

18. The action recognition computer device according to claim 14 , wherein the processor, when loading and executing the computer program, is caused to perform:

extracting a predetermined number of P frame data in each of the interval compressed videos in an equal-interval manner;

wherein a predetermined number of P frame data in each of the interval compressed videos is extracted in a random-interval manner during training of the first predetermined model.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (48)