Systems and methods for a transformer network with tree-based attention for natural language processing

1. A method of using a transformer framework with tree-based attention for hierarchical encoding in natural language processing, comprising:

obtaining, at the transformer framework, a pre-parsed constituency tree having a set of terminal nodes and a set of nonterminal nodes corresponding to a natural language sentence;

encoding the natural language sentence at a transformer encoder of the transformer framework by:

determining a set of paths from a root node of the pre-parsed constituency tree to each of the set of terminal nodes based on a structure of the pre-parsed constituency tree;

computing a respective value component of each nonterminal node in the pre-parsed constituency tree by adding hidden states of descendant nodes of the respective nonterminal node along a specific path that includes the respective nonterminal node in a bottom-up manner, and

computing a final representation of the respective nonterminal node in the pre-parsed constituency tree by applying weighted aggregation of respective value components corresponding to the respective nonterminal nodes over all paths that include the respective nonterminal node;

generating an encoded representation of the natural language sentences based on the computed final representations of all nonterminal nodes; and

generating a final representation of the set of nonterminal nodes based on attentions for the set of terminal nodes and the set of nonterminal nodes via serial computations of a transformer network.

2. The method of claim 1 , further comprising:

determining, based on the pre-parsed constituency tree, a first hidden representation vector corresponding to value components of the set of terminal nodes and a second hidden representation vector corresponding to value components of the set of nonterminal nodes;

applying an interpolation function to the first hidden representation vector, the second hidden representation vector and a set of rules indexed by the set of nonterminal nodes; and

obtaining a first tensor from the interpolation function, wherein the tensor has rows and columns arranged according to a structure of the pre-parsed constituency tree.

3. The method of claim 2 , further comprising:

computing a second tensor from the first tensor via an upward cumulative-average operation, wherein each element in the second tensor is computed by dividing a respective nonterminal node representation from the first tensor by a total number of all descendent nodes of the respective nonterminal node in a particular branch, and

wherein each row of the second tensor represents a nonterminal node, and the each element of the second tensor represents a vector representation of the nonterminal node reflecting the particular branch.

4. The method of claim 3 , wherein the computing, by weighted aggregation, the final representation of the set of nonterminal nodes in the pre-parsed constituency tree based on the encoding comprises:

applying a weighting vector to the each element of the second tensor; and

for a particular nonterminal node from the set of nonterminal nodes, combining, into a single accumulation vector, weighed elements from the second tensor corresponding to vector representations of nonterminal nodes in a subtree rooted at the particular nonterminal node.

5. The method of claim 2 , further comprising:

applying hierarchical embedding to the first tensor before elements of the first tensor is accumulated through an upward cumulative-average operation.

6. The method of claim 5 , wherein the hierarchical embedding includes:

constructing a tensor of hierarchical embeddings,

wherein an entry of the tensor of hierarchical embeddings is computed by concatenating a first row vector from a vertical embedding matrix and a second row vector from a horizontal embedding matrix.

7. The method of claim 6 , further comprising:

summing the first tensor with the tensor of hierarchical embeddings; and

applying the upward cumulative-average operation to the summed first tensor with the tensor of hierarchical embeddings.

8. The method of claim 1 , further comprising:

applying a masking function to each node in the pre-parsed constituency tree based on a corresponding affinity value of the respective node,

wherein, when a node query is attending to a particular node in the pre-parsed constituency tree, the masking function prevents the node query from accessing node other than descendants of the particular node.

9. The method of claim 1 , further comprising integrating encoder self-attention into the transformer framework by:

computing, via a tree-based self-attention layer, first output representations for the set of terminal nodes and second output representations for the set of nonterminal nodes;

generating a query-key affinity matrix based on a comparison of the first output representations and the second output representations;

computing first value representations for the set of terminal nodes based on the first output representations;

encoding second value representations for the set of nonterminal nodes using hierarchical accumulation based on the first output representations and the second output representations;

computing final attentions for the set of terminal nodes and the set of nonterminal nodes by taking weighted averages of the encoded second value representations and the first value representations; and

passing the final attentions through serial computations of a transformer network to generate the final representation of the set of nonterminal nodes.

10. The method of claim 9 , further comprising integrating decoder cross-attention into the transformer framework by:

computing affinity score matrices based on a target-side query matrix, the first output representations for the set of terminal nodes and the second output representations for the set of nonterminal nodes;

computing first value representations for the set of terminal nodes based on the first output representations;

computing second value representations for the set of nonterminal nodes using hierarchical accumulation based on the first output representations and the second output representations; and

computing an attention output of decoder cross-attention based on the affinity score matrices, the first value representations and the second value representations.

11. A system for using a transformer framework with tree-based attention for hierarchical encoding in natural language processing, the system comprising:

a memory containing machine readable medium storing machine executable code; and

one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to:

obtain, at the transformer framework, a pre-parsed constituency tree having a set of terminal nodes and a set of nonterminal nodes corresponding to a natural language sentence;

encode the natural language sentence at a transformer encoder of the transformer framework by:

determining a set of paths from a root node of the pre-parsed constituency tree to each of the set of terminal nodes based on a structure of the pre-parsed constituency tree;

computing a respective value component of each nonterminal node in the pre-parsed constituency tree by adding hidden states of descendant nodes of the respective nonterminal node along a specific path that includes the respective nonterminal node in a bottom-up manner, and

computing a final representation of the respective nonterminal node in the pre-parsed constituency tree by applying weighted aggregation of respective value components corresponding to the respective nonterminal nodes over all paths that include the respective nonterminal node;

generate an encoded representation of the natural language sentences base on the computed final representations of all nonterminal nodes; and

generate a final representation of the set of nonterminal nodes based on attentions for the set of terminal nodes and the set of nonterminal nodes via serial computations of a transformer network.

12. The system of claim 11 , further comprising:

determining, based on the pre-parsed constituency tree, a first hidden representation vector corresponding to value components of the set of terminal nodes and a second hidden representation vector corresponding to value components of the set of nonterminal nodes;

applying an interpolation function to the first hidden representation vector, the second hidden representation vector and a set of rules indexed by the set of nonterminal nodes; and

obtaining a first tensor from the interpolation function, wherein the tensor has rows and columns arranged according to a structure of the pre-parsed constituency tree.

13. The system of claim 12 , wherein the machine executable code further causes the one or more processors to:

compute a second tensor from the first tensor via an upward cumulative-average operation, wherein each element in the second tensor is computed by dividing a respective nonterminal node representation from the first tensor by a total number of all descendent nodes of the respective nonterminal node in a particular branch, and

wherein each row of the second tensor represents a nonterminal node, and the each element of the second tensor represents a vector representation of the nonterminal node reflecting the particular branch.

14. The system of claim 13 , wherein the machine executable code further causes the one or more processors to compute, by weighted aggregation, the final representation of the set of nonterminal nodes in the pre-parsed constituency tree based on the encoding by:

applying a weighting vector to the each element of the second tensor; and

for a particular nonterminal node from the set of nonterminal nodes, combining, into a single accumulation vector, weighed elements from the second tensor corresponding to vector representations of nonterminal nodes in a subtree rooted at the particular nonterminal node.

15. The system of claim 12 , wherein the machine executable code further causes the one or more processors to:

apply hierarchical embedding to the first tensor before elements of the first tensor is accumulated through an upward cumulative-average operation.

16. The system of claim 15 , wherein the hierarchical embedding includes:

constructing a tensor of hierarchical embeddings,

wherein an entry of the tensor of hierarchical embeddings is computed by concatenating a first row vector from a vertical embedding matrix and a second row vector from a horizontal embedding matrix.

17. The system of claim 16 , wherein the machine executable code further causes the one or more processors to:

sum the first tensor with the tensor of hierarchical embeddings; and

apply the upward cumulative-average operation to the summed first tensor with the tensor of hierarchical embeddings.

18. The system of claim 11 , wherein the machine executable code further causes the one or more processors to:

apply a masking function to each node in the pre-parsed constituency tree based on a corresponding affinity value of the respective node,

wherein, when a node query is attending to a particular node in the pre-parsed constituency tree, the masking function prevents the node query from accessing node other than descendants of the particular node.

19. The system of claim 11 , wherein the machine executable code further causes the one or more processors to integrate encoder self-attention into the transformer framework by:

computing, via a tree-based self-attention layer, first output representations for the set of terminal nodes and second output representations for the set of nonterminal nodes;

generating a query-key affinity matrix based on a comparison of the first output representations and the second output representations;

computing first value representations for the set of terminal nodes based on the first output representations;

encoding second value representations for the set of nonterminal nodes using hierarchical accumulation based on the first output representations and the second output representations;

computing final attentions for the set of terminal nodes and the set of nonterminal nodes by taking weighted averages of the encoded second value representations and the first value representations; and

passing the final attentions through serial computations of a transformer network to generate the final representation of the set of nonterminal nodes.

20. The system of claim 19 , wherein the machine executable code further causes the one or more processors to integrate decoder cross-attention into the transformer framework by:

computing affinity score matrices based on a target-side query matrix, the first output representations for the set of terminal nodes and the second output representations for the set of nonterminal nodes;

computing first value representations for the set of terminal nodes based on the first output representations;

computing second value representations for the set of nonterminal nodes using hierarchical accumulation based on the first output representations and the second output representations; and

computing an attention output of decoder cross-attention based on the affinity score matrices, the first value representations and the second value representations.