Neural network training method and apparatus, computer device, and storage medium
A neural network training method, apparatus, a storage medium, and a computer device are provided. The method includes: obtaining a training sample set, each training sample including a standard label; inputting the each training sample into a neural network model including n attention networks, the n attention networks respectively mapping the each training sample to n subspaces, each of the n subspaces including a query vector sequence, a key vector sequence, and a value vector sequence; calculating a space difference degree between the n subspaces by using the neural network model; calculating an output similarity degree according to an output of the neural network model and the standard label corresponding to the each training sample; and adjusting a model parameter of the neural network model according to the space difference degree and the output similarity degree until a convergence condition is satisfied to obtain a target neural network model.
1 . A neural network training method, performed by a computer device, the method comprising:
obtaining a training sample set, each training sample in the training sample set including a corresponding standard label;
training a neural network module based on inputting the each training sample in the training sample set into the neural network model, the neural network model comprising n attention networks, the n attention networks respectively mapping the each training sample to n different subspaces, each subspace of the n subspaces comprising a corresponding query vector sequence, a corresponding key vector sequence, and a corresponding value vector sequence, and n being an integer greater than 1;
determining a space difference degree between the n subspaces by using the neural network model;
determining an output similarity degree according to an output of the neural network model and the standard label corresponding to the each training sample; and
retraining the neural network model by adjusting a model parameter of the neural network model according to the space difference degree and the output similarity degree until a convergence condition is satisfied, thereby obtaining a target neural network model based on retraining the neural network model by adjusting the model parameter.
2 . The method according to claim 1 , wherein the n attention networks respectively mapping the each training sample to the n different subspaces comprises:
converting the each training sample into a corresponding source vector sequence;
obtaining a first query parameter matrix, a first key parameter matrix, and a first value parameter matrix, and respectively performing linear transformation on the corresponding source vector sequence according to the first query parameter matrix, the first key parameter matrix, and the first value parameter matrix, to obtain a corresponding basic query vector sequence, a corresponding basic key vector sequence, and a corresponding basic value vector sequence; and
obtaining a space parameter matrix corresponding to the each subspace, and respectively performing linear mapping on the basic query vector sequence, the basic key vector sequence, and the basic value vector sequence according to the space parameter matrix, to obtain the query vector sequence, the key vector sequence, and the value vector sequence that correspond to the each subspace.
3 . The method according to claim 2 , further comprising:
splicing output vector sequences corresponding to the subspaces, and performing linear transformation on a spliced output vector sequence to obtain a network representation sequence; and
using the network representation sequence as an updated source vector sequence to obtain a second query parameter matrix, a second key parameter matrix, and a second value parameter matrix and respectively performing linear transformation on the source vector sequence according to the second query parameter matrix, the second key parameter matrix, and the second value parameter matrix, and outputting a target network representation sequence based on determining that a cycle stop condition is satisfied.
4 . The method according to claim 1 , further comprising:
determining a logical similarity degree between a query vector sequence and a key vector sequence in a current subspace among the n subspaces;
obtaining an attention matrix corresponding to the current subspace through determination according to the logical similarity degree; and
obtaining an output vector sequence corresponding to the current subspace through determination according to the attention matrix and a value vector sequence.
5 . The method according to claim 4 , wherein the determining the space difference degree between the n subspaces by using the neural network model comprises:
determining a subspace input difference degree according to value vector sequences corresponding to adjacent subspaces among the n subspaces;
determining an attention matrix difference degree according to attention matrices corresponding to the adjacent subspaces;
determining a subspace output difference degree according to output vector sequences corresponding to the adjacent subspaces; and
determining the space difference degree according to at least one of the subspace input difference degree, the attention matrix difference degree, or the subspace output difference degree.
6 . The method according to claim 5 , wherein the determining subspace input difference degree according to the value vector sequences corresponding to the adjacent subspaces comprises:
determining value vector similarity degrees between the value vector sequences corresponding to the adjacent subspaces; and
collecting statistics on the value vector similarity degrees to obtain the subspace input difference degree.
7 . The method according to claim 5 , wherein the determining the attention matrix difference degree according to the attention matrices corresponding to the adjacent subspaces comprises:
determining attention matrix similarity degrees between the attention matrices corresponding to the adjacent subspaces; and
collecting statistics on the attention matrix similarity degrees to obtain the attention matrix difference degree.
8 . The method according to claim 5 , wherein the determining the subspace output difference degree according to the output vector sequences corresponding to the adjacent subspaces comprises:
determining output vector similarity degrees between the output vector sequences corresponding to the adjacent subspaces; and
collecting statistics on the output vector similarity degrees to obtain the subspace output difference degree.
9 . The method according to claim 1 , wherein the adjusting a model parameter of the neural network model according to the space difference degree and the output similarity degree until the convergence condition is satisfied comprises:
performing linear determination on the space difference degree and the output similarity degree to obtain a model adjustment reference result; and
based on the model adjustment reference result being maximized, determining that the neural network model satisfies the convergence condition.
10 . The method according to claim 1 , wherein the neural network model is a machine translation model, and comprises an encoding attention unit, a codec attention unit, and a decoding attention unit,
wherein each of the encoding attention unit, the codec attention unit, and the decoding attention unit comprises a plurality of attention networks, and
wherein the codec attention unit is separately connected to the encoding attention unit and the decoding attention unit.
11 . The method according to claim 10 , wherein the training sample is at least one of a text, a video, or an audio, and the standard label corresponding to the training sample is a standard translated text.
12 . A neural network training apparatus, comprising:
at least one memory configured to store computer program code; and
at least one processor configured to access the at least one memory and operate as instructed by the computer program code, the computer program code comprising:
training sample set obtaining code configured to cause the at least one processor to obtain a training sample set, each training sample in the training sample set including a corresponding standard label;
training sample set training code configured to cause the at least one processor to train a neural network module based on inputting the each training sample in the training sample set into the neural network model, the neural network model comprising n attention networks, the n attention networks respectively mapping the each training sample to n different subspaces, each subspace of the n subspaces comprising a corresponding query vector sequence, a corresponding key vector sequence, and a corresponding value vector sequence, and n being an integer greater than 1;
space difference degree determination code configured to cause the at least one processor to determine a space difference degree between the n subspaces by using the neural network model;
output similarity degree determination code configured to cause the at least one processor to determine an output similarity degree according to an output of the neural network model and the standard label corresponding to the each training sample; and
target neural network model generation code configured to cause the at least one processor to retrain the neural network model by adjusting a model parameter of the neural network model according to the space difference degree and the output similarity degree until a convergence condition is satisfied, thereby obtaining a target neural network model based on retraining the neural network model by adjusting the model parameter.
13 . The apparatus according to claim 12 , wherein the training sample set training code further comprises:
training sample conversion code configured to cause the at least one processor to convert the each training sample into a corresponding source vector sequence;
basic vector sequence generation code configured to cause the at least one processor to obtain a first query parameter matrix, a first key parameter matrix, and a first value parameter matrix, and respectively perform linear transformation on the corresponding source vector sequence according to the first query parameter matrix, the first key parameter matrix, and the first value parameter matrix, to obtain a corresponding basic query vector sequence, a corresponding basic key vector sequence, and a corresponding basic value vector sequence; and
space parameter matrix obtaining code configured to cause the at least one processor to obtain a space parameter matrix corresponding to the each subspace, and respectively perform linear mapping on the basic query vector sequence, the basic key vector sequence, and the basic value vector sequence according to the space parameter matrix, to obtain the query vector sequence, the key vector sequence, and the value vector sequence that correspond to the each subspace.
14 . The apparatus according to claim 13 , further comprising:
splicing code configured to cause the at least one processor to splice output vector sequences corresponding to the subspaces, and perform linear transformation on a spliced output vector sequence to obtain a network representation sequence; and
iteration code configured to cause the at least one processor to use the network representation sequence as an updated source vector sequence to obtain a second query parameter matrix, a second key parameter matrix, and a second value parameter matrix and respectively perform linear transformation on the source vector sequence according to the second query parameter matrix, the second key parameter matrix, and the second value parameter matrix, and output a target network representation sequence based on determining that a cycle stop condition is satisfied.
15 . The apparatus according to claim 12 , further comprising:
logical similarity degree determination code configured to cause the at least one processor to determine a logical similarity degree between a query vector sequence and a key vector sequence in a current subspace among the n subspaces;
attention matrix determination code configured to cause the at least one processor to obtain an attention matrix corresponding to the current subspace through determination according to the logical similarity degree; and
output vector sequence determination code configured to cause the at least one processor to obtain an output vector sequence corresponding to the current subspace through determination according to the attention matrix and a value vector sequence.
16 . The apparatus according to claim 15 , wherein the space difference degree determination code is further configured to cause the at least one processor to:
determine a subspace input difference degree according to value vector sequences corresponding to adjacent subspaces among the n subspaces;
determine an attention matrix difference degree according to attention matrices corresponding to the adjacent subspaces;
determine a subspace output difference degree according to output vector sequences corresponding to the adjacent subspaces; and
determine the space difference degree according to at least one of the subspace input difference degree, the attention matrix difference degree, or the subspace output difference degree.
17 . The apparatus according to claim 12 , wherein the target neural network model generation code is further configured to cause the at least one processor to:
perform linear determination on the space difference degree and the output similarity degree to obtain a model adjustment reference result; and
based on the model adjustment reference result being maximized, determine that the neural network model satisfies the convergence condition.
18 . The apparatus according to claim 12 , wherein the neural network model is a machine translation model, and comprises an encoding attention unit, a codec attention unit, and a decoding attention unit,
wherein each of the encoding attention unit, the codec attention unit, and the decoding attention unit comprises a plurality of attention networks, and
wherein the codec attention unit is separately connected to the encoding attention unit and the decoding attention unit.
19 . The apparatus according to claim 18 , wherein the training sample is at least one of a text, a video, or an audio, and the standard label corresponding to the training sample is a standard translated text.
20 . A non-transitory computer-readable storage medium storing computer program code to cause at least one processor to:
obtain a training sample set, each training sample in the training sample set including a corresponding standard label;
train a neural network module based on inputting the each training sample in the training sample set into the neural network model, the neural network model comprising n attention networks, the n attention networks respectively mapping the each training sample to n different subspaces, each subspace of the n subspaces comprising a corresponding query vector sequence, a corresponding key vector sequence, and a corresponding value vector sequence, and n being an integer greater than 1;
determine a space difference degree between the n subspaces by using the neural network model;
determine an output similarity degree according to an output of the neural network model and the standard label corresponding to the each training sample; and
retrain the neural network model by adjusting a model parameter of the neural network model according to the space difference degree and the output similarity degree until a convergence condition is satisfied, thereby obtaining a target neural network model based on retraining the neural network model by adjusting the model parameter.