Image description generation method, model training method, device and storage medium

1. An image description generation method, applied to a computing device having one or more processors and memory storing programs to be executed by the one or more processors, the method comprising:

obtaining a target image;

generating a first global feature vector and a first label vector set of the target image;

applying the target image to a matching model and generating a first multi-mode feature vector of the target image through the matching model, the matching model being a model obtained through training according to a training image and reference image description information of the training image; and

generating, via n depth networks, n being a positive integer, target image description information of the target image according to the first global feature vector, a first spliced vector obtained by splicing the first multi-mode feature vector and the first label vector set.

2. The method according to claim 1 , wherein the generating, via the n depth networks, n being the positive integer, the target image description information of the target image according to the first global feature vector, a first spliced vector obtained by splicing the first multi-mode feature vector and the first label vector set comprises:

applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information, wherein the computing model is a model obtained through training according to image description information and the reference image description information of the training image.

3. The method according to claim 2 , wherein the computing model comprises the n depth networks, and n is the positive integer;

the applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to the computing model, to obtain the target image description information comprises:

generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks;

wherein an input parameter of at least one depth network in the n depth networks comprises a spliced vector, when an input parameter of the ith depth network comprises the spliced vector, if i is equal to 1, the spliced vector is a vector obtained by splicing the first multi-mode feature vector and the first label vector set, if i is greater than 1, the spliced vector is a vector obtained by splicing an output vector of the (i−1)th depth network and the first multi-mode feature vector, wherein i is greater than or equal to 1, and less than or equal to n.

4. The method according to claim 3 , wherein n is equal to 2;

the generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks comprises:

splicing the first multi-mode feature vector and the first label vector set, to obtain the first spliced vector;

applying the first spliced vector and the first global feature vector to a first depth network, to obtain a first output vector;

splicing the first output vector and the first multi-mode feature vector, to obtain a second spliced vector; and

applying the second spliced vector to a second depth network, to obtain the target image description information.

5. The method according to claim 1 , further comprising:

obtaining a second global feature vector and a second label vector set of the training image, and a text feature vector of the reference image description information of the training image; and

training the matching model according to the second global feature vector and the text feature vector.

6. The method according to claim 5 , further comprising:

generating a second multi-mode feature vector of the training image through the matching model obtained through training;

applying the second multi-mode feature vector, the second global feature vector, and the second label vector set to the computing model, to obtain the image description information of the training image; and

training, if the reference image description information does not match the image description information of the training image, the computing model according to the image description information and the reference image description information of the training image.

7. A computing device, comprising one or more processors, memory coupled to the one or more processors and a plurality of programs stored in the memory that, when executed by the one or more processors, cause the computing device to perform a plurality of operations comprising:

obtaining a target image;

generating a first global feature vector and a first label vector set of the target image;

applying the target image to a matching model and generating a first multi-mode feature vector of the target image through the matching model, the matching model being a model obtained through training according to a training image and reference image description information of the training image; and

generating, via n depth networks, n being a positive integer, target image description information of the target image according to the first global feature vector, a first spliced vector obtained by splicing the first multi-mode feature vector and the first label vector set.

8. The computing device according to claim 7 , wherein the generating, via the n depth networks, n being the positive integer, the target image description information of the target image according to the first global feature vector, a first spliced vector obtained by splicing the first multi-mode feature vector and the first label vector set comprises:

applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information, wherein the computing model is a model obtained through training according to image description information and the reference image description information of the training image.

9. The computing device according to claim 8 , wherein the computing model comprises the n depth networks, and n is the positive integer;

the applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to the computing model, to obtain the target image description information comprises:

generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks;

wherein an input parameter of at least one depth network in the n depth networks comprises a spliced vector, when an input parameter of the ith depth network comprises the spliced vector, if i is equal to 1, the spliced vector is a vector obtained by splicing the first multi-mode feature vector and the first label vector set, if i is greater than 1, the spliced vector is a vector obtained by splicing an output vector of the (i−1)th depth network and the first multi-mode feature vector, wherein i is greater than or equal to 1, and less than or equal to n.

10. The computing device according to claim 9 , wherein n is equal to 2;

the generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks comprises:

splicing the first multi-mode feature vector and the first label vector set, to obtain the first spliced vector;

applying the first spliced vector and the first global feature vector to a first depth network, to obtain a first output vector;

splicing the first output vector and the first multi-mode feature vector, to obtain a second spliced vector; and

applying the second spliced vector to a second depth network, to obtain the target image description information.

11. The computing device according to claim 7 , wherein the plurality of operations further comprise:

obtaining a second global feature vector and a second label vector set of the training image, and a text feature vector of the reference image description information of the training image; and

training the matching model according to the second global feature vector and the text feature vector.

12. The computing device according to claim 11 , wherein the plurality of operations further comprise:

generating a second multi-mode feature vector of the training image through the matching model obtained through training;

applying the second multi-mode feature vector, the second global feature vector, and the second label vector set to the computing model, to obtain the image description information of the training image; and

training, if the reference image description information does not match the image description information of the training image, the computing model according to the image description information and the reference image description information of the training image.

13. A non-transitory computer readable storage medium storing a plurality of machine readable instructions in connection with a computing device having one or more processors, wherein the plurality of machine readable instructions, when executed by the one or more processors, cause the computing device to perform a plurality of operations including:

obtaining a target image;

generating a first global feature vector and a first label vector set of the target image;

applying the target image to a matching model and generating a first multi-mode feature vector of the target image through the matching model, the matching model being a model obtained through training according to a training image and reference image description information of the training image; and

generating, via n depth networks, n being a positive integer, target image description information of the target image according to the first global feature vector, a first spliced vector obtained by splicing the first multi-mode feature vector and the first label vector set.

14. The non-transitory computer readable storage medium according to claim 13 , wherein the generating, via then depth networks, n being the positive integer, the target image description information of the target image according to the first global feature vector, a first spliced vector obtained by splicing the first multi-mode feature vector and the first label vector set comprises:

applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information, wherein the computing model is a model obtained through training according to image description information and the reference image description information of the training image.

15. The non-transitory computer readable storage medium according to claim 14 , wherein the computing model comprises the n depth networks, and n is the positive integer;

the applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to the computing model, to obtain the target image description information comprises:

generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks;

wherein an input parameter of at least one depth network in the n depth networks comprises a spliced vector, when an input parameter of the ith depth network comprises the spliced vector, if i is equal to 1, the spliced vector is a vector obtained by splicing the first multi-mode feature vector and the first label vector set, if i is greater than 1, the spliced vector is a vector obtained by splicing an output vector of the (i−1)th depth network and the first multi-mode feature vector, wherein i is greater than or equal to 1, and less than or equal to n.

16. The non-transitory computer readable storage medium according to claim 15 , wherein n is equal to 2;

the generating the target image description information according to the first multi-mode feature vector, the first global feature vector, the first label vector set, and the n depth networks comprises:

splicing the first multi-mode feature vector and the first label vector set, to obtain the first spliced vector;

applying the first spliced vector and the first global feature vector to a first depth network, to obtain a first output vector;

splicing the first output vector and the first multi-mode feature vector, to obtain a second spliced vector; and

applying the second spliced vector to a second depth network, to obtain the target image description information.

17. The non-transitory computer readable storage medium according to claim 13 , wherein the plurality of operations further comprise:

obtaining a second global feature vector and a second label vector set of the training image, and a text feature vector of the reference image description information of the training image; and

training the matching model according to the second global feature vector and the text feature vector.

18. The non-transitory computer readable storage medium according to claim 17 , wherein the plurality of operations further comprise:

generating a second multi-mode feature vector of the training image through the matching model obtained through training;

applying the second multi-mode feature vector, the second global feature vector, and the second label vector set to the computing model, to obtain the image description information of the training image; and

training, if the reference image description information does not match the image description information of the training image, the computing model according to the image description information and the reference image description information of the training image.