Method and apparatus for detecting key point of image, computer device and storage medium

1 . A method for detecting a key point of an image, the method comprising:

acquiring a to-be-detected image;

performing, based on a type of the to-be-detected image, preprocessing on the to-be-detected image to generate a multi-channel image with the type of the to-be-detected image, wherein the type of the to-be-detected image comprises a discrete frame type or a continuous frame type;

inputting the multi-channel image into a pre-trained key point detection model, wherein the pre-trained key point detection model comprises a regression branch and an inter-frame residual branch which are parallel;

in a case that the type of the to-be-detected image is the discrete frame type, acquiring a regression prediction value outputted by the regression branch and determining coordinates of a plurality of key points in the to-be-detected image based on the regression prediction value;

in a case that the type of the to-be-detected image is the continuous frame type, acquiring a regression prediction value outputted by the regression branch and a residual prediction value outputted by the inter-frame residual branch, and determining coordinates of a plurality of key points in the to-be-detected image based on the regression prediction value and the residual prediction value; and

wherein, in a training process of the key point detection model, the key point detection model comprises the regression branch, the inter-frame residual branch and a type classification branch which are parallel.

2 . The method according to claim 1 , wherein the to-be-detected image is a multi-color channel image, and the performing, based on a type of the to-be-detected image, preprocessing on the to-be-detected image to generate a multi-channel image with the type comprises:

in a case that the type of the to-be-detected image is the discrete frame type, generating a first grayscale image, with a same resolution as the to-be-detected image and with a value of each pixel value being zero, as a grayscale residual image, combining the multi-color channel image with the grayscale residual image to generate a multi-channel image with the discrete frame type; and

in a case that the type of the to-be-detected image is the continuous frame type, determining a second grayscale image of the to-be-detected image, acquiring a previous frame image adjacent to the to-be-detected image, determining a third grayscale image of the previous frame image, determining a difference between the second grayscale image and the third grayscale image as a grayscale residual image, and combining the multi-color channel image with the grayscale residual image to generate a multi-channel image with the continuous frame type.

3 . The method according to claim 2 , wherein the multi-color channel image is a three-channel image with a red channel, a green channel and a blue channel, and the generated multi-channel image is a four-channel image with a red channel, a green channel, a blue channel, and a grayscale residual channel.

4 . The method according to claim 1 , wherein in a case that the type of the to-be-detected image is the continuous frame type, the determining coordinates of a plurality of key points in the to-be-detected image based on the regression prediction value and the residual prediction value comprises:

taking the regression prediction value of the to-be-detected image as a direct regression value of the to-be-detected image;

acquiring a regression prediction value outputted by the regression branch after a previous frame image adjacent to the to-be-detected image is inputted to the key point detection model, taking a sum of the regression prediction value of the previous frame image and the residual prediction value of the to-be-detected image as an estimated regression value of the to-be-detected image;

calculating a weighted sum of the direct regression value and the estimated regression value as an estimated key point value of the to-be-detected image; and

determining the coordinates of the plurality of key points in the to-be-detected image based on the estimated key point value.

5 . The method according to claim 1 , wherein before performing, based on the type of the to-be-detected image, preprocessing on the to-be-detected image to generate the multi-channel image with the type, the method further comprises:

performing contour enhancement processing on the to-be-detected image to generate a to-be-detected image with an enhanced contour.

6 . The method according to claim 1 , wherein, in the training process, the key point detection model further comprises a backbone network configured to receive the multi-channel image, output an intermediate feature map, and feed the intermediate feature map into the regression branch, the inter-frame residual branch, and the type classification branch.

7 . The method according to claim 1 , wherein

the regression branch comprises a plurality of first convolutional neural network layers, a flattening layer and a first fully connected layer in series;

the inter-frame residual branch comprises a plurality of second convolutional neural network layers, a flattening layer and a second fully connected layer in series; and

the type classification branch comprises a plurality of third convolutional neural network layers, a flattening layer and a third fully connected layer in series.

8 . The method according to claim 7 , wherein each of the first fully connected layer and the second fully connected layer comprises a first number of output neurons, and the first number depends on a total number of the plurality of key points; and/or the third fully connected layer comprises a second number of output neurons, and the second number depends on a type of a loss function in the type classification branch.

9 . The method according to claim 6 , wherein before acquiring the to-be-detected image, the training process of the key point detection model comprises:

acquiring a training set, wherein the training set comprises a plurality of multi-channel images with the discrete frame type, a true value of each multi-channel image with the discrete frame type, a plurality of multi-channel images with the continuous frame type and a true value of each multi-channel image with the continuous frame type;

initializing layer weights in the key point detection model;

iteratively performing, by using the training set, forward propagation for calculating a total loss function and backward propagation for updating the layer weights on the key point detection model until a value of the total loss function converges, to determine trained layer weights, wherein the multi-channel images with the discrete frame type are used for training the layer weights in the regression branch and the type classification branch, and the multi-channel images with the continuous frame type are used for training the layer weights in the regression branch, the inter-frame residual branch and the type classification branch; and

determining the pre-trained key point detection model based on the trained layer weights.

10 . The method according to claim 9 , wherein acquiring the plurality of multi-channel images with the continuous frame type and the true value of each multi-channel image with the continuous frame type comprises:

acquiring a single-frame image, and determining a true value of a multi-channel image generated based on the single-frame image;

performing continuous illumination processing, color processing and/or deformation processing on the single-frame image, to generate a plurality of continuous frame images in a simulation manner;

performing preprocessing on each of the plurality of continuous frame images to generate the plurality of multi-channel images with the continuous frame type; and

determining the true value of the multi-channel image with the continuous frame type, generated based on each of the plurality of continuous frame images, based on a deformation amount between the plurality of continuous frame images and the single-frame image and the true value of the multi-channel image generated based on the single-frame image.

11 . The method according to claim 9 , wherein

the true value of each multi-channel image with the discrete frame type in the training set comprises a regression true value of the regression branch and a classification true value of the type classification branch; and

the true value of each multi-channel image with the continuous frame type in the training set comprises a regression true value of the regression branch, a residual true value of the inter-frame residual branch and a classification true value of the type classification branch.

12 . The method according to claim 11 , wherein

the regression branch comprises a first loss function for representing a difference between the regression prediction value outputted by the regression branch and the regression true value;

the inter-frame residual branch comprises a second loss function for representing a difference between the residual prediction value outputted by the inter-frame residual branch and the residual true value; and

the type classification branch comprises a third loss function for representing a difference between a classification prediction value outputted by the type classification branch and the classification true value.

13 . The method according to claim 12 , wherein

the total loss function of the key point detection model is determined based on the first loss function and the third loss function, in a case that the key point detection model is trained by using the multi-channel images with the discrete frame type; and

the total loss function of the key point detection model is determined based on the first loss function, the second loss function and the third loss function, in a case that the key point detection model is trained by using the multi-channel images with the continuous frame type.

14 . An apparatus for detecting a key point of an image, the apparatus comprising:

a processor configured to execute key point detecting instructions comprising:

an image acquiring module, configured to acquire a to-be-detected image;

a preprocessing module, configured to perform, based on a type of the to-be-detected image, preprocessing on the to-be-detected image to generate a multi-channel image with the type, wherein the type comprises a discrete frame type or a continuous frame type;

a model predicting module, configured to input the multi-channel image into a pre-trained key point detection model, wherein the pre-trained key point detection model comprises a regression branch and an inter-frame residual branch which are parallel; and

a key point determining module, configured to: in a case that the type of the to-be-detected image is the discrete frame type, acquire a regression prediction value outputted by the regression branch and determine coordinates of a plurality of key points in the to-be-detected image based on the regression prediction value; and in a case that the type of the to-be-detected image is the continuous frame type, acquire a regression prediction value outputted by the regression branch and a residual prediction value outputted by the inter-frame residual branch, and determine coordinates of a plurality of key points in the to-be-detected image based on the regression prediction value and the residual prediction value, wherein, in a training process of the key point detection model, the key point detection model comprises the regression branch, the inter-frame residual branch and a type classification branch which are parallel.

15 . A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor, when executing the computer program, implements the method according to claim 1 .

16 . A non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method according to claim 1 .