Extracting entity relationships from digital documents utilizing multi-view neural networks
This disclosure describes methods, non-transitory computer readable storage media, and systems that utilize a plurality of neural networks to determine structural and semantic information via different views of a word sequence and then utilize this information to extract a relationship between word sequence entities. For example, the disclosed systems generate a plurality of sets of encoded word representation vectors utilizing the plurality of neural networks. The disclosed system then extracts the relationship from an overall word representation vector generated based on the sets of encoded word representation vectors. Furthermore, the disclosed system enforces structural and semantic consistency between views via a plurality of constrains involving a control mechanism for the semantic view and a plurality of losses.
1 . A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor, cause a computing device to:
generate, utilizing a long short-term memory neural network, a first set of encoded word representation vectors for a sequence of words comprising at least two entities;
generate, utilizing a ranked neurons long short-term memory neural network, a second set of encoded word representation vectors for the sequence of words;
generate, utilizing a self-attention neural network and from the first set of encoded word representation vectors, a third set of encoded word representation vectors for the sequence of words by:
generating an initial set of encoded word representation vectors utilizing the self-attention neural network; and
generating the third set of encoded word representation vectors by modifying the initial set of encoded word representation vectors according to a control vector selected from the first set of encoded word representation vectors; and
extract, utilizing a classification neural network, a relationship between the at least two entities in the sequence of words by providing the first set of encoded word representation vectors generated by the long short-term memory neural network, the second set of encoded word representation vectors generated by the ranked neurons long short-term memory neural network, and the third set of encoded word representation vectors generated by the self-attention neural network to the classification neural network.
2 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate a sequence of word representation vectors based on word embeddings for the sequence of words and position embeddings indicating positions of the at least two entities within the sequence of words; and
generate the first set of encoded word representation vectors, the second set of encoded word representation vectors, and the third set of encoded word representation vectors from the sequence of word representation vectors.
3 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine an overall word representation vector by combining max-pooling aggregation vectors of the first set of encoded word representation vectors, the second set of encoded word representation vectors, and the third set of encoded word representation vectors with encoded word representation vectors of the at least two entities; and
extract the relationship between the at least two entities from the overall word representation vector utilizing the classification neural network.
4 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate, utilizing the ranked neurons long short-term memory neural network, the second set of encoded word representation vectors by:
generating a set of structural importance scores indicating a proximity of each word in the sequence of words to a root node of a tree structure; and
generating the second set of encoded word representation vectors based on the set of structural importance scores.
5 . The non-transitory computer readable storage medium as recited in claim 4 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate, utilizing the self-attention neural network, key vectors, query vectors, and value vectors for the first set of encoded word representation vectors; and
generate, utilizing the self-attention neural network, the third set of encoded word representation vectors based on the key vectors, the query vectors, and the value vectors.
6 . The non-transitory computer readable storage medium as recited in claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate, utilizing the self-attention neural network, a set of semantic connection scores between each pair of words in the sequence of words indicating a semantic graph structure based on the key vectors and the query vectors;
determine an importance probability distribution from the set of structural importance scores and a semantic probability distribution from the set of semantic connection scores;
determine a structure loss based on a Kullback-Leibler divergence between the importance probability distribution and the semantic probability distribution; and
learn parameters of the long short-term memory neural network, the ranked neurons long short-term memory neural network, or the self-attention neural network according to the structure loss.
7 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine, utilizing a variable discriminator, an estimated mutual information metric between the first set of encoded word representation vectors and the second set of encoded word representation vectors;
determine a mutual information loss based on the estimated mutual information metric; and
learn parameters of the long short-term memory neural network, the ranked neurons long short-term memory neural network, or the self-attention neural network according to the mutual information loss.
8 . The non-transitory computer readable storage medium as recited in claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to extract the relationship between the at least two entities by:
generating, utilizing a classification neural network, an overall word representation vector by combining the first set of encoded word representation vectors, the second set of encoded word representation vectors, and the third set of encoded word representation vectors;
generating, utilizing a classification neural network, a relationship probability distribution over possible relationships for the at least two entities based on the overall word representation vector; and
generating a label indicating the relationship between the at least two entities from the relationship probability distribution.
9 . The non-transitory computer readable storage medium as recited in claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine a classification loss by determining a negative log-likelihood of the relationship probability distribution; and
learn parameters of the long short-term memory neural network, the ranked neurons long short-term memory neural network, or the self-attention neural network according to the classification loss.
10 . A system comprising:
a memory device comprising:
a sequence of words comprising a first entity and a second entity; and
a long short-term memory neural network, a ranked neurons long short-term memory neural network, a self-attention neural network, and a classification neural network;
a computing device configured to cause the system to:
generate word representation vectors for the sequence of words;
generate a first set of encoded word representation vectors, a second set of encoded word representation vectors, and a third set of encoded word representation vectors by processing the word representation vectors utilizing the long short-term memory neural network, the ranked neurons long short-term memory neural network, and the self-attention neural network, the self-attention neural network generating the third set of encoded word representation vectors from the first set of encoded word representation vectors generated by the long short-term memory neural network by:
generating an initial set of encoded word representation vectors utilizing the self-attention neural network; and
generating the third set of encoded word representation vectors by modifying the initial set of encoded word representation vectors according to a control vector selected from the first set of encoded word representation vectors;
generate an overall word representation vector by combining the first set of encoded word representation vectors from the long short-term memory neural network, the second set of encoded word representation vectors from the ranked neurons long short-term memory neural network, and the third set of encoded word representation vectors from the self-attention neural network; and
determine a relationship between the first entity and the second entity in the sequence of words by processing the overall word representation vector utilizing the classification neural network.
11 . The system as recited in claim 10 , wherein the computing device is further configured to cause the system to generate the second set of encoded word representation vectors by:
generating, utilizing a set of forget gates of the ranked neurons long short-term memory neural network, a set of structural importance scores indicating a proximity of each word to a root node of a tree structure; and
generating, based on the set of structural importance scores, the second set of encoded word representation vectors comprising a plurality of hidden vectors from the ranked neurons long short-term memory neural network.
12 . The system as recited in claim 10 , wherein the computing device is further configured to cause the system to generate the third set of encoded word representation vectors by:
generating, for pairs of words in the sequence of words, semantic connection scores indicating semantic connections between the pairs of words by utilizing the self-attention neural network to process corresponding pairs of encoded word representation vectors of the first set of encoded word representation vectors; and
generating the third set of encoded word representation vectors based on the semantic connection scores for the pairs of words in the sequence of words.
13 . The system as recited in claim 10 , wherein the computing device is further configured to cause the system to determine the relationship between the first entity and the second entity by:
generating, from the overall word representation vector, a relationship probability distribution over a plurality of possible relationships between the first entity and the second entity; and
determining the relationship between the first entity and the second entity based on the relationship probability distribution.
14 . The system as recited in claim 10 , wherein the computing device is further configured to cause the system to:
determine a structure loss based on a set of structural importance scores associated with the ranked neurons long short-term memory neural network and a set of semantic importance scores associated with the self-attention neural network; and
learn parameters of the long short-term memory neural network, the ranked neurons long short-term memory neural network, and the self-attention neural network based on the structure loss.
15 . The system as recited in claim 10 , wherein the computing device is further configured to cause the system to:
determine a mutual information loss based on a mutual information metric between the first set of encoded word representation vectors and the second set of encoded word representation vectors; and
learn parameters of the long short-term memory neural network, the ranked neurons long short-term memory neural network, or the self-attention neural network based on the mutual information loss.
16 . The system as recited in claim 10 , wherein the computing device is further configured to cause the system to:
determine a classification loss by comparing a classification prediction generated by the classification neural network from the overall word representation vector to a ground truth classification; and
learn parameters of the long short-term memory neural network, the ranked neurons long short-term memory neural network, or the self-attention neural network based on the classification loss.
17 . A method comprising:
generating word representation vectors for a sequence of words comprising a first entity and a second entity;
generating an overall word representation vector including a plurality of sets of encoded word representation vectors from the word representation vectors by generating a first set of encoded word representation vectors utilizing a long short-term memory neural network, a second set of encoded word representation vectors utilizing a ranked neurons long short-term memory neural network, and a third set of encoded word representation vectors from the first set of encoded word representation vectors utilizing a self-attention neural network, the self-attention neural network generating the third set of encoded word representation vectors by:
generating an initial set of encoded word representation vectors utilizing the self-attention neural network; and
generating the third set of encoded word representation vectors by modifying the initial set of encoded word representation vectors according to a control vector selected from the first set of encoded word representation vectors;
wherein parameters of the long short-term memory neural network, the ranked neurons long short-term memory neural network, and the self-attention neural network are jointly learned based on a classification loss, a structure loss, and a mutual information loss; and
extract a relationship between the first entity and the second entity in the sequence of words by processing the overall word representation vector utilizing a classification neural network.
18 . The method as recited in claim 17 , wherein generating the overall word representation vector comprises:
combining the first set of encoded word representation vectors, the second set of encoded word representation vectors, and the third set of encoded word representation vectors into the overall word representation vector by:
determining max-pooling aggregation vectors of the first set of encoded word representation vectors, the second set of encoded word representation vectors, and the third set of encoded word representation vectors; and
combining the max-pooling aggregation vectors with encoded word representation vectors of the first entity and the second entity.
19 . The method as recited in claim 17 , wherein extracting the relationship between the first entity and the second entity comprises:
generating, utilizing the classification neural network, a plurality of classification probabilities associated with a plurality of possible relationships for the first entity and the second entity; and
selecting, from the plurality of possible relationships, a possible relationship with a highest relationship probability as the relationship between the first entity and the second entity.
20 . The method as recited in claim 17 , further comprising determining a total loss comprising the classification loss, the structure loss, and the mutual information loss by weighting the structure loss with a first weighting parameter and the mutual information loss with a second weighting parameter.