Systems and methods for a supra-graph based attention model for multi-layered embeddings and deep learning applications
Various embodiments of systems and methods for attention models with random features for multi-layered graph embeddings are disclosed.
1. A method, comprising:
providing a multi-layered graph having a plurality of layers, each of the plurality of layers including a plurality of nodes;
obtaining a supra graph by introducing one or more virtual edges between every node in a layer and its counterparts in other layers;
generating a set of features for each of the nodes in the supra graph;
inferring dependencies between nodes in the supra graph by processing each set of features using an attention model, wherein the attention model comprises a plurality of attention layers and wherein each of the plurality of attention layers produces a set of layer-specific node features; and
characterizing each of the plurality of nodes by aggregating layer-specific node features associated with one of the plurality of nodes across each of the plurality of attention layers.
2. The method of claim 1 , wherein the virtual edges are introduced between nodes having the same ID across at least two of the plurality of layers.
3. The method of claim 1 , wherein the virtual edges are introduced between layers based on local neighborhood structure.
4. The method of claim 1 , further comprising determining an edge attention weight for each of the one or more virtual edges, wherein the edge attention weight is determined using a set of attention function parameters and features from neighboring nodes, and wherein the attention function parameters are obtained using back-propagation.
5. The method of claim 1 , wherein the attention model applies a linear transformation to the set of features for each of the nodes in the supra graph using a trainable weight matrix.
6. The method of claim 5 , wherein the attention model produces a set of layer-specific node features using a weighted combination of normalized attention coefficients and linearly transformed features for neighboring nodes.
7. The method of claim 5 , wherein the normalized attention coefficients are obtained by normalizing each of the one or more edge attention weights across neighboring nodes using a softmax function.
8. A method comprising:
providing a multi-layered graph having a plurality of layers, each of the plurality of layers including a plurality of nodes;
obtaining a supra graph by introducing virtual edges between every node in a layer and its counterparts in other layers;
learning a plurality of latent feature vectors for each of the plurality of nodes using an attention model, wherein the attention model comprises a plurality of layers, and wherein each of the plurality of layers comprises a plurality of attention heads and a fusion head;
combining each of the plurality of latent feature vectors for a node determined by each of the plurality of attention heads to produce a layer-specific feature vector for the node using the fusion head, wherein the layer-specific feature vector is associated with one of a plurality of attention layers; and
characterizing each of the plurality of nodes by aggregating latent feature vectors associated with one of the plurality of nodes across each of the plurality of attention layers.
9. The method of claim 8 , wherein each attention head performs the steps of:
transforming a set of node attributes associated with each of the plurality of nodes in the supra graph into a set of weighted node attributes using a linear transformation;
assigning an attention weight to each of the virtual edges in the supra graph with respect to each of a plurality of nodes in connection with a singular node;
normalizing each of the attention weights to produce a set of normalized attention coefficients; and
determining a latent feature vector for each of the plurality of nodes using the set of normalized attention coefficients and the set of weighted node attributes.
10. The method of claim 8 , wherein a latent feature vector of a node comprises a weighted combination of features from neighboring nodes.
11. The method of claim 9 , wherein attention weights for each of the plurality of virtual edges in the supra graph are determined based on features from neighboring nodes.
12. The method of claim 11 , wherein the attention weights for each of the plurality of virtual edges are determined using a scalar dot-product attention function.
13. The method of claim 12 , wherein a self-edge for each node is included in the features from neighboring nodes.
14. The method of claim 11 , wherein an optimal set of attention function parameters are obtained through back-propagation in an end-to-end learning fashion.
15. The method of claim 9 , wherein the attention weights are normalized across neighboring nodes using a softmax function.
16. The method of claim 9 , wherein the linear transformation is achieved using a trainable weight matrix, and wherein a set of optimal values for the trainable weight matrix is obtained through back-propagation in an end-to-end learning fashion.
17. The method of claim 8 , further comprising generating a set of random features for each of the nodes in the supra graph.
18. The method of claim 8 , wherein each fusion head performs a weighted combination of each of the plurality of latent feature vectors such that each of the plurality of attention heads are assigned varying levels of importance.
19. The method of claim 8 , wherein the virtual edges are introduced between nodes having the same ID across at least two of the plurality of layers.
20. The method of claim 8 , further comprising:
processing each of the latent feature vectors obtained from each attention layer and associated with the node using across-layer average pooling; and
processing the output of the across-layer average pooling layer using a feed-forward layer to obtain the node label prediction, wherein the feed-forward layer uses softmax activation.