Systems and methods for a supra-fusion graph attention model for multi-layered embeddings and deep learning applications
Various embodiments of systems and methods for attention models with random features for multi-layered graph embeddings are disclosed.
1. A method, comprising:
providing a multi-layered graph having a plurality of graph layers, each of the plurality of graph layers including a plurality of nodes;
constructing a graph-layer-specific latent feature vector for each of the plurality of nodes using a graph-layer-specific attention model;
inferring dependencies between nodes of the plurality of nodes by processing all sets of graph-layer-specific features using a supra-fusion layer, wherein the supra-fusion layer comprises a plurality of fusion heads;
obtaining a consensus representation from each of the plurality of fusion heads using an overall fusion head, wherein each of the plurality of fusion heads of the supra-fusion layer performs a weighted combination of the graph-layer-specific latent feature vectors for a node from each of a plurality of graph-layer-specific attention heads, and wherein each of the plurality of fusion heads comprises a scaling factor associated with each of the plurality of graph-layer-specific attention heads, wherein each of the scaling factors are shared across each of the plurality of nodes; and
characterizing each node of the plurality of nodes by aggregating layer-specific node features associated with one node of the plurality of nodes across each of a plurality of attention layers.
2. The method of claim 1 , further comprising:
generating a set of random attributes for every node in the graph layer using random initialization.
3. The method of claim 1 , wherein the graph-layer-specific attention model is restricted to a single graph layer.
4. The method of claim 1 , further comprising:
processing each output of the supra-fusion layer using a feed-forward layer.