Generating word vectors by recurrent neural networks based on n-ary characters

1. A method for generating word vectors, comprising:

obtaining individual words by segmenting a corpus;

establishing a feature vector for each obtained word based, at least in part, on one or more n-ary characters corresponding to the obtained word, each n-ary character representing n consecutive characters of a word;

training a recurrent neural network based, at least in part, on the feature vectors of the obtained words and feature vectors of context words associated with each obtained word in the corpus by:

traversing the corpus after word segmentation, and performing the following actions in response a current word being accessed during the traversal:

determining one or more context words and negative sample words associated with the current word in the corpus;

inputting a sequence formed by feature vectors of the context words associated with the current word into a sequence representation layer of the recurrent neural network to obtain a first vector;

inputting a feature vector of the current word into a full connection layer of the recurrent neural network to obtain a second vector; and

inputting a feature vector of a negative sample word associated with the current word into the full connection layer of the recurrent neural network to obtain a third vector; and

updating parameters of the recurrent neural network based, at least in part, on the first vector, the second vector, the third vector, and a loss function, wherein the loss function includes at least a weight parameter of the sequence representation layer, a weight parameter of the full connection layer, an offset parameter of the full connection layer, a hyperparameter, and a similarity calculation function; and

generating a word vector for each obtained word based, at least in part, on the feature vector of the obtained word and the trained recurrent neural network.

2. The method according to claim 1 , wherein characters of an obtained word comprise each character constituting the obtained word and one or more mark characters added to a start position and/or an end position of the obtained word.

3. The method according to claim 1 , wherein establishing the feature vector for each obtained word based, at least in part, on one or more n-ary characters corresponding to the obtained word comprises:

determining a total number of distinct n-ary characters in a collection of respective n-ary characters corresponding to each of the obtained words; and

establishing, for each obtained word, a feature vector whose dimensionality is determined based, at least in part, on the total number.

4. The method according to claim 3 , wherein a number of dimensions of the feature vector is the total number.

5. The method according to claim 3 , wherein values are assigned to individual dimensions of the feature vector to indicate a mapping between n-ary characters and the obtained word.

6. The method according to claim 1 , further comprising performing recurrent calculation using the sequence representation layer of the recurrent neural network based on at least a hidden unit and an activation function.

7. The method according to claim 1 , wherein updating the parameters of the recurrent neural network based on the first vector, the second vector, the third vector, and the loss function comprises:

calculating a first similarity between the second vector and the first vector, and a second similarity between the third vector and the first vector; and

updating parameters of the recurrent neural network based, at least in part, on the first similarity, the second similarity, and the loss function.

8. The method according to claim 1 , wherein generating a word vector for each obtained word based, at least in part, on the feature vector of the obtained word and the trained recurrent neural network specifically comprises:

inputting the feature vector of each obtained word into the full connection layer of the trained recurrent neural network to obtain a vector output.

9. An apparatus for generating word vectors, comprising:

an acquisition module, configured to obtain individual words by segmenting a corpus;

an establishment module, configured to establish a feature vector for each obtained word based, at least in part, on one or more n-ary characters corresponding to the obtained word, each n-ary character representing n consecutive characters of a word;

a training module, configured to train a recurrent neural network based, at least in part, on the feature vectors of the obtained words and feature vectors of context words associated with each obtained word in the corpus by:

traversing the corpus after word segmentation, and performing the following actions in response to a current word being accessed during the traversal:

determining one or more context words and negative sample words associated with the current word in the corpus;

inputting a sequence formed by feature vectors of the context words associated with the current word into a sequence representation layer of the recurrent neural network to obtain a first vector;

inputting a feature vector of the current word into a full connection layer of the recurrent neural network to obtain a second vector; and

inputting a feature vector of a negative sample word associated with the current word into the full connection layer of the recurrent neural network to obtain a third vector; and

updating parameters of the recurrent neural network based, at least in part, on the first vector, the second vector, the third vector, and a loss function, wherein the loss function includes at least a weight parameter of the sequence representation layer, a weight parameter of the full connection layer, an offset parameter of the full connection layer, a hyperparameter, and a similarity calculation function; and

a generation module, configured to generate a word vector for each obtained word based, at least in part, on the feature vector of the obtained word and the trained recurrent neural network.

10. The apparatus according to claim 9 , wherein characters of an obtained word comprise each character constituting the obtained word and one or more mark characters added to a start position and/or an end position of the obtained word.

11. The apparatus according to claim 9 , wherein the establishment module is further configured to establish a feature vector for each obtained word based, at least in part, on n-ary characters corresponding to the obtained word by:

determining, by the establishment module, a total number of distinct n-ary characters in a collection of respective n-ary characters corresponding to each of the obtained words; and

establishing, for each obtained word, a feature vector whose dimensionality is determined based, at least in part, on the total number.

12. The apparatus according to claim 11 , wherein a number of dimensions of the feature vector is the total number.

13. The apparatus according to claim 12 , wherein values are assigned to individual dimensions of the feature vector to indicate a mapping between n-ary characters and the obtained word.

14. The apparatus according to claim 9 , wherein the training module is further configured to perform one or more recurrent calculations according to the following formula:

s t =σ( Ux t +Ws t−1 )

o t =softmax( Vs t )

wherein x t indicates an input unit of the sequence representation layer at the time t, that is, a feature vector of the t+1 th context word associated with the current word; s t indicates a hidden unit of the sequence representation layer at the time t; o t indicates a vector obtained through recurrent calculation of the feature vectors of the first t+1 context words associated with the current word; U, W, and V indicate weight parameters of the sequence representation layer; and σ indicates an activation function.

15. The apparatus according to claim 9 , wherein updating the parameters of the recurrent neural network based, at least in part, on the first vector, the second vector, the third vector, and the loss function comprises:

calculating, by the training unit, a first similarity between the second vector and the first vector, and a second similarity between the third vector and the first vector; and

updating the parameters of the recurrent neural network based, at least in part, on the first similarity, the second similarity, and the loss function.

16. The apparatus according to claim 9 , wherein the loss function comprises:

l

⁡

(

w

,

c

;

U

,

W

,

V

,

Ϛ

,

τ

)

=

log

⁡

(

1

+

∑

m

=

1

λ

⁢

exp

⁢

(

-

γ

·

(

s

⁡

(

w

,

c

)

-

s

⁡

(

w

m

′

,

⁢

c

)

)

)

)

wherein c indicates the first vector; w indicates the second vector; w m ′ indicates the third vector corresponding to the m th negative sample word; U, W, and V indicate weight parameters of the sequence representation layer; ζ indicates a weight parameter of the full connection layer; τ indicates an offset parameter of the full connection layer; γ indicates a hyperparameter; s indicates a similarity calculation function; and λ indicates a number of negative sample words.

17. The apparatus according to claim 9 , wherein the generation module is further configured to generate a word vector for each obtained word based, at least in part, on the feature vector of the obtained word and the trained recurrent neural network by:

inputting, by the generation module, the feature vector of each obtained word into the full connection layer of the trained recurrent neural network to obtain a vector output.

18. An apparatus for generating word vectors, comprising:

at least one processor, and

a memory communicatively connected to the at least one processor, wherein the memory stores instructions that are executed by the at least one processor to cause the at least one processor to implement:

an acquisition module, configured to obtain individual words by segmenting a corpus;

an establishment module, configured to establish a feature vector for each obtained word based, at least in part, on one or more n-ary characters corresponding to the obtained word, each n-ary character representing n consecutive characters of a word;

a training module, configured to train a recurrent neural network based, at least in part, on the feature vectors of the obtained words and feature vectors of context words associated with each obtained word in the corpus by:

traversing the corpus after word segmentation, and performing the following actions in response to a current word being accessed during the traversal:

determining one or more context words and negative sample words associated with the current word in the corpus;

inputting a sequence formed by feature vectors of the context words associated with the current word into a sequence representation layer of the recurrent neural network to obtain a first vector;

inputting a feature vector of the current word into a full connection layer of the recurrent neural network to obtain a second vector; and

inputting a feature vector of a negative sample word associated with the current word into the full connection layer of the recurrent neural network to obtain a third vector; and

updating parameters of the recurrent neural network based, at least in part, on the first vector, the second vector, the third vector, and a loss function, wherein the loss function includes at least a weight parameter of the sequence representation layer, a weight parameter of the full connection layer, an offset parameter of the full connection layer, a hyperparameter, and a similarity calculation function; and

a generation module, configured to generate a word vector for each obtained word based, at least in part, on the feature vector of the obtained word and the trained recurrent neural network.