Granted Patent B2

US 12,361,036 · App. 17/989,613 · Granted Jul 15, 2025

Inventors: Feilong Chen (Shenzhen, CN); Fandong Meng (Shenzhen, CN); Peng Li (Shenzhen, CN); Jie Zhou (Shenzhen, CN)

Assignee: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED

G06F16/3329G06F16/3347G06F16/583

Abstract

Claims (98)

1. A method performed by an electronic device acting as a server that is communicatively connected to a terminal, wherein the electronic device includes an input/output (I/O) system, a processor and a memory and a system bus connecting the I/O system, the processor and the memory together, the method comprising:

receiving, via the I/O system, an input image submitted by the terminal;

acquiring, from the memory, an image feature of the input image and state vectors corresponding to first n rounds of historical question answering dialog, n being a positive integer;

acquiring, via the I/O system, a question feature of a current round of questioning related to the input image submitted by the terminal;

performing, via the processor and using a visual dialog model stored in the memory, multimodal encoding on the image feature of the input image, the state vectors corresponding to the first n rounds of historical question answering dialog, and the question feature of the current round of questioning, to obtain a state vector corresponding to the current round of questioning, the performing further including acquiring a character string feature of an outputted character string corresponding to the current round of questioning by invoking a multimodal incremental transformer decoder in the visual dialog model;

performing, via the processor and using the visual dialog model stored in the memory, multimodal decoding on the state vector corresponding to the current round of questioning and the image feature of the input image, and the character string feature by invoking the multimodal incremental transformer decoder in the visual dialog model, to obtain a decoded feature vector;

determining, via the processor, an actual output answer corresponding to the current round of questioning according to the decoded feature vector, the actual output answer comprising the outputted character string; and

returning, via the I/O system, the actual output answer corresponding to the current round of questioning to the terminal.

2. The method according to claim 1 , wherein the performing, via the processor and using the visual dialog model stored in the memory, multimodal encoding on the image feature of the input image, the state vectors corresponding to the first n rounds of historical question answering dialog, and the question feature of the current round of questioning, to obtain a state vector corresponding to the current round of questioning comprises:

acquiring a state vector corresponding to an i th round of historical question answering dialog, i being a positive integer, an initial value of i being 1;

iterating i, and performing multimodal encoding on the image feature, the state vector corresponding to the i th round of historical question answering dialog, and a question answering feature corresponding to an (i+1) th round of historical question answering dialog by invoking an (i+1) th multimodal incremental transformer encoder in the visual dialog model, to obtain a state vector corresponding to the (i+1) th round of historical question answering dialog, different multimodal incremental transformer encoders corresponding one to one to different historical question answering dialog; and

determining a state vector corresponding to an (n+1) th round of historical question answering dialog obtained by iterating i as the state vector corresponding to the current round of questioning.

3. The method according to claim 2 , wherein the (i+1) th multimodal incremental transformer encoder comprises K sub-transformer encoders, K being a positive integer; and

the performing multimodal encoding on the image feature, the state vector corresponding to the i th round of historical question answering dialog, and a question answering feature corresponding to an (i+1) th round of historical question answering dialog by invoking an (i+1) th multimodal incremental transformer encoder in the visual dialog model, to obtain a state vector corresponding to the (i+1) th round of historical question answering dialog comprises:

acquiring a j th intermediate representation vector, the j th intermediate representation vector being obtained by performing multimodal encoding on the image feature, the state vector corresponding to the i th round of historical question answering dialog, and the question answering feature corresponding to the (i+1) th round of historical question answering dialog j times, the j th intermediate representation vector being a vector corresponding to the (i+1) th round of historical question answering dialog, j being a positive integer, an initial value of j being 1;

iterating j, and performing multimodal encoding on the j th intermediate representation vector, the image feature, and the state vector corresponding to the i th round of historical question answering dialog by invoking a (j+1) th sub-transformer encoder in the (i+1) th multimodal incremental transformer encoder, to obtain a (j+1) th intermediate representation vector, the (j+1) th intermediate representation vector being another vector corresponding to the (i+1) th round of historical question answering dialog, j+1≤K; and

determining a K th intermediate representation vector obtained by iterating j as the state vector corresponding to the (i+1) th round of historical question answering dialog.

4. The method according to claim 3 , wherein the performing multimodal encoding on the j th intermediate representation vector, the image feature, and the state vector corresponding to the i th round of historical question answering dialog by invoking a (j+1) th sub-transformer encoder in the (i+1) th multimodal incremental transformer encoder, to obtain a (j+1) th intermediate representation vector comprises:

performing intermediate encoding on the j th intermediate representation vector by invoking the (j+1) th sub-transformer encoder in the (i+1) th multimodal incremental transformer encoder, to obtain a first subvector;

performing intermediate encoding on the first subvector and the image feature, to obtain a second subvector;

performing intermediate encoding on the second subvector and the state vector corresponding to the i th round of historical question answering dialog, to obtain a third subvector; and

performing intermediate encoding on the third subvector, to obtain the (j+1) th intermediate representation vector.

5. The method according to claim 1 , wherein the determining, via the processor, the actual output answer corresponding to the current round of questioning according to the decoded feature vector comprises:

determining a character string probability according to the decoded feature vector; and

determining the character string in the actual output answer according to the character string probability.

6. The method according to claim 1 , wherein the multimodal incremental transformer decoder comprises T sub-transformer decoders, T being a positive integer; and

the performing, via the processor and using the visual dialog model stored in the memory, multimodal decoding on the state vector corresponding to the current round of questioning, the image feature, and the character string feature by invoking the multimodal incremental transformer decoder, to obtain a decoded feature vector comprises:

acquiring an m th intermediate representation vector, the m th intermediate representation vector being obtained by performing multimodal decoding on the state vector corresponding to the current round of questioning, the image feature, and the character string feature m times, m being a positive integer, an initial value of m being 1;

iterating m, and performing multimodal decoding on the m th intermediate representation vector, the image feature, and the state vector corresponding to the current round of questioning by invoking an (m+1) th sub-transformer decoder in the multimodal incremental transformer decoder, to obtain an (m+1) th intermediate representation vector, m+1≤T; and

determining a T th intermediate representation vector obtained by iterating m as the decoded feature vector.

7. The method according to claim 5 , wherein the performing multimodal decoding on the m th intermediate representation vector, the image feature, and the state vector corresponding to the current round of questioning by invoking an (m+1) th sub-transformer decoder in the multimodal incremental transformer decoder, to obtain an (m+1) th intermediate representation vector comprises:

performing intermediate decoding on the m th intermediate representation vector by invoking the (m+1) th sub-transformer decoder in the multimodal incremental transformer decoder, to obtain the third subvector;

performing intermediate decoding on the third subvector, the image feature, and the state vector corresponding to the current round of questioning, to obtain a fourth subvector; and

performing intermediate decoding on the fourth subvector, to obtain the (m+1) th intermediate representation vector.

8. The method according to claim 6 , wherein the visual dialog model is trained by:

acquiring an image feature sample of an input image sample and state vector samples corresponding to first s rounds of historical question answering dialog samples, s being a positive integer;

acquiring a question feature sample of a current round of questioning samples and a first answer feature of an actual answer corresponding to the current round of questioning samples;

performing multimodal encoding on the image feature sample, the state vector samples corresponding to the first s rounds of historical question answering dialog samples, and the question feature sample by invoking the visual dialog model, to obtain a state vector sample corresponding to the current round of questioning samples; and

performing multimodal decoding on the state vector sample corresponding to the current round of questioning samples, the image feature sample, and the first answer feature by invoking the visual dialog model, to obtain a second answer feature of an actual output answer sample corresponding to the current round of questioning samples; and

training the visual dialog model according to the first answer feature and the second answer feature, to obtain the trained visual dialog model.

9. The method according to claim 8 , wherein the performing multimodal decoding on the state vector sample corresponding to the current round of questioning samples, the image feature sample, and the first answer feature, to obtain a second answer feature of an actual output answer sample corresponding to the current round of questioning samples comprises:

acquiring a character string feature label of first q character strings in the actual answer, the first q character strings in the actual answer corresponding one to one to outputted q character strings in the actual output answer sample, q being a positive integer, the first answer feature comprising the character string feature label; and

obtaining the second answer feature corresponding to a (q+1) character string in the actual output answer sample corresponding to the current round of questioning samples according to the state vector sample corresponding to the current round of questioning samples, the image feature sample, and the character string feature label.

10. An electronic device acting as a server that is communicatively connected to a terminal, the electronic device comprising an input/output (I/O) system, a processor and a memory and a system bus connecting the I/O system, the processor and the memory together, the memory storing at least one instruction, the at least one instruction being executed by the processor and causing the electronic device to implement a method including:

receiving, via the I/O system, an input image submitted by the terminal;

acquiring, from the memory, an image feature of the input image and state vectors corresponding to first n rounds of historical question answering dialog, n being a positive integer;

acquiring, via the I/O system, a question feature of a current round of questioning related to the input image submitted by the terminal;

performing, via the processor and using a visual dialog model stored in the memory, multimodal encoding on the image feature of the input image, the state vectors corresponding to the first n rounds of historical question answering dialog, and the question feature of the current round of questioning, to obtain a state vector corresponding to the current round of questioning, the performing further including acquiring a character string feature of an outputted character string corresponding to the current round of questioning by invoking a multimodal incremental transformer decoder in the visual dialog model;

performing, via the processor and using the visual dialog model stored in the memory, multimodal decoding on the state vector corresponding to the current round of questioning and the image feature of the input image, and the character string feature by invoking the multimodal incremental transformer decoder in the visual dialog model, to obtain a decoded feature vector;

determining, via the processor, an actual output answer corresponding to the current round of questioning according to the decoded feature vector, the actual output answer comprising the outputted character string; and

returning, via the I/O system, the actual output answer corresponding to the current round of questioning to the terminal.

11. The electronic device according to claim 10 , wherein the performing, via the processor and using the visual dialog model stored in the memory, multimodal encoding on the image feature of the input image, the state vectors corresponding to the first n rounds of historical question answering dialog, and the question feature of the current round of questioning, to obtain a state vector corresponding to the current round of questioning comprises:

acquiring a state vector corresponding to an i th round of historical question answering dialog, i being a positive integer, an initial value of i being 1;

iterating i, and performing multimodal encoding on the image feature, the state vector corresponding to the i th round of historical question answering dialog, and a question answering feature corresponding to an (i+1) th round of historical question answering dialog by invoking an (i+1) th multimodal incremental transformer encoder in the visual dialog model, to obtain a state vector corresponding to the (i+1) th round of historical question answering dialog, different multimodal incremental transformer encoders corresponding one to one to different historical question answering dialog; and

determining a state vector corresponding to an (n+1) th round of historical question answering dialog obtained by iterating i as the state vector corresponding to the current round of questioning.

12. The electronic device according to claim 10 , wherein the determining, via the processor, the actual output answer corresponding to the current round of questioning according to the decoded feature vector comprises:

determining a character string probability according to the decoded feature vector; and

determining the character string in the actual output answer according to the character string probability.

13. A non-transitory computer-readable storage medium, storing at least one instruction, the at least one instruction being executed by a processor of an electronic device acting as a server that is communicatively connected to a terminal, wherein the electronic device includes an input/output (I/O) system, a processor and a memory and a system bus connecting the I/O system, the processor and the memory together, and the at least one instruction causing the electronic device to perform a method including:

receiving, via the I/O system, an input image submitted by the terminal;

acquiring, from the memory, an image feature of the input image and state vectors corresponding to first n rounds of historical question answering dialog, n being a positive integer;

acquiring, via the I/O system, a question feature of a current round of questioning related to the input image submitted by the terminal;

performing, via the processor and using a visual dialog model stored in the memory, multimodal encoding on the image feature of the input image, the state vectors corresponding to the first n rounds of historical question answering dialog, and the question feature of the current round of questioning, to obtain a state vector corresponding to the current round of questioning, the performing further including acquiring a character string feature of an outputted character string corresponding to the current round of questioning by invoking a multimodal incremental transformer decoder in the visual dialog model;

performing, via the processor and using the visual dialog model stored in the memory, multimodal decoding on the state vector corresponding to the current round of questioning and the image feature of the input image, and the character string feature by invoking the multimodal incremental transformer decoder in the visual dialog model, to obtain a decoded feature vector;

determining, via the processor, an actual output answer corresponding to the current round of questioning according to the decoded feature vector, the actual output answer comprising the outputted character string; and

returning, via the I/O system, the actual output answer corresponding to the current round of questioning to the terminal.

14. The non-transitory computer-readable storage medium according to claim 13 , wherein the performing, via the processor and using the visual dialog model stored in the memory, multimodal encoding on the image feature of the input image, the state vectors corresponding to the first n rounds of historical question answering dialog, and the question feature of the current round of questioning, to obtain a state vector corresponding to the current round of questioning comprises:

acquiring a state vector corresponding to an i th round of historical question answering dialog, i being a positive integer, an initial value of i being 1;

iterating i, and performing multimodal encoding on the image feature, the state vector corresponding to the i th round of historical question answering dialog, and a question answering feature corresponding to an (i+1) th round of historical question answering dialog by invoking an (i+1) th multimodal incremental transformer encoder in the visual dialog model, to obtain a state vector corresponding to the (i+1) th round of historical question answering dialog, different multimodal incremental transformer encoders corresponding one to one to different historical question answering dialog; and

determining a state vector corresponding to an (n+1) th round of historical question answering dialog obtained by iterating i as the state vector corresponding to the current round of questioning.

15. The non-transitory computer-readable storage medium according to claim 13 , wherein the determining, via the processor, the actual output answer corresponding to the current round of questioning according to the decoded feature vector comprises:

determining a character string probability according to the decoded feature vector; and

determining the character string in the actual output answer according to the character string probability.

16. The electronic device according to claim 11 , wherein the (i+1) th multimodal incremental transformer encoder comprises K sub-transformer encoders, K being a positive integer; and

the performing multimodal encoding on the image feature, the state vector corresponding to the i th round of historical question answering dialog, and a question answering feature corresponding to an (i+1) th round of historical question answering dialog by invoking an (i+1) th multimodal incremental transformer encoder in the visual dialog model, to obtain a state vector corresponding to the (i+1) th round of historical question answering dialog comprises:

acquiring a j th intermediate representation vector, the j th intermediate representation vector being obtained by performing multimodal encoding on the image feature, the state vector corresponding to the i th round of historical question answering dialog, and the question answering feature corresponding to the (i+1) th round of historical question answering dialog j times, the j th intermediate representation vector being a vector corresponding to the (i+1) th round of historical question answering dialog, j being a positive integer, an initial value of j being 1;

iterating j, and performing multimodal encoding on the j th intermediate representation vector, the image feature, and the state vector corresponding to the i th round of historical question answering dialog by invoking a (j+1) th sub-transformer encoder in the (i+1) th multimodal incremental transformer encoder, to obtain a (j+1) th intermediate representation vector, the (j+1) th intermediate representation vector being another vector corresponding to the (i+1) th round of historical question answering dialog, j+1≤K; and

determining a K th intermediate representation vector obtained by iterating j as the state vector corresponding to the (i+1) th round of historical question answering dialog.

17. The electronic device according to claim 16 , wherein the performing multimodal encoding on the j th intermediate representation vector, the image feature, and the state vector corresponding to the i th round of historical question answering dialog by invoking a (j+1) th sub-transformer encoder in the (i+1) th multimodal incremental transformer encoder, to obtain a (j+1) th intermediate representation vector comprises:

performing intermediate encoding on the j th intermediate representation vector by invoking the (j+1) th sub-transformer encoder in the (i+1) th multimodal incremental transformer encoder, to obtain a first subvector;

performing intermediate encoding on the first subvector and the image feature, to obtain a second subvector;

performing intermediate encoding on the second subvector and the state vector corresponding to the i th round of historical question answering dialog, to obtain a third subvector; and

performing intermediate encoding on the third subvector, to obtain the (j+1) th intermediate representation vector.

18. The electronic device according to claim 10 , wherein the multimodal incremental transformer decoder comprises T sub-transformer decoders, T being a positive integer; and

the performing multimodal decoding on the state vector corresponding to the current round of questioning, the image feature, and the character string feature by invoking the multimodal incremental transformer decoder, to obtain a decoded feature vector comprises:

acquiring an m th intermediate representation vector, the m th intermediate representation vector being obtained by performing multimodal decoding on the state vector corresponding to the current round of questioning, the image feature, and the character string feature m times, m being a positive integer, an initial value of m being 1;

iterating m, and performing multimodal decoding on the m th intermediate representation vector, the image feature, and the state vector corresponding to the current round of questioning by invoking an (m+1) th sub-transformer decoder in the multimodal incremental transformer decoder, to obtain an (m+1) th intermediate representation vector, m+1≤T; and

determining a T th intermediate representation vector obtained by iterating m as the decoded feature vector.

19. The non-transitory computer-readable storage medium according to claim 14 , wherein the (i+1) th multimodal incremental transformer encoder comprises K sub-transformer encoders, K being a positive integer; and

the performing multimodal encoding on the image feature, the state vector corresponding to the i th round of historical question answering dialog, and a question answering feature corresponding to an (i+1) th round of historical question answering dialog by invoking an (i+1) th multimodal incremental transformer encoder in the visual dialog model, to obtain a state vector corresponding to the (i+1) th round of historical question answering dialog comprises:

acquiring a j th intermediate representation vector, the j th intermediate representation vector being obtained by performing multimodal encoding on the image feature, the state vector corresponding to the i th round of historical question answering dialog, and the question answering feature corresponding to the (i+1) th round of historical question answering dialog j times, the j th intermediate representation vector being a vector corresponding to the (i+1) th round of historical question answering dialog, j being a positive integer, an initial value of j being 1;

iterating j, and performing multimodal encoding on the j th intermediate representation vector, the image feature, and the state vector corresponding to the i th round of historical question answering dialog by invoking a (j+1) th sub-transformer encoder in the (i+1) th multimodal incremental transformer encoder, to obtain a (j+1) th intermediate representation vector, the (j+1) th intermediate representation vector being another vector corresponding to the (i+1) th round of historical question answering dialog, j+1≤K; and

determining a K th intermediate representation vector obtained by iterating j as the state vector corresponding to the (i+1) th round of historical question answering dialog.

20. The non-transitory computer-readable storage medium according to claim 13 , wherein the multimodal incremental transformer decoder comprises T sub-transformer decoders, T being a positive integer; and

the performing multimodal decoding on the state vector corresponding to the current round of questioning, the image feature, and the character string feature by invoking the multimodal incremental transformer decoder, to obtain a decoded feature vector comprises:

acquiring an m th intermediate representation vector, the m th intermediate representation vector being obtained by performing multimodal decoding on the state vector corresponding to the current round of questioning, the image feature, and the character string feature m times, m being a positive integer, an initial value of m being 1;

iterating m, and performing multimodal decoding on the m th intermediate representation vector, the image feature, and the state vector corresponding to the current round of questioning by invoking an (m+1) th sub-transformer decoder in the multimodal incremental transformer decoder, to obtain an (m+1) th intermediate representation vector, m+1≤T; and

determining a T th intermediate representation vector obtained by iterating m as the decoded feature vector.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (18)