Method, system, and computer program product for knowledge graph based embedding, explainability, and/or multi-task learning
Methods, systems, and computer program products for knowledge graph based embedding, explainability, and/or multi-task learning may connect task-specific inductive models with knowledge graph completion and enrichment processes.
1 . A computer-implemented method comprising:
reading, with at least one processor, in parallel, with a plurality of threads, graph data associated with a plurality of edges and a plurality of nodes for the plurality of edges of a graph;
for each edge of the plurality of edges, with a thread of the plurality of threads that read the graph data associated with that edge, one of: (i) discarding that edge, (ii) sampling that edge to generate one or more samples and providing the one or more samples to a random queue of a plurality of queues, and (iii) oversampling that edge to generate the one or more samples and providing the one or more samples to the random queue of the plurality of queues, based on frequencies of nodes for that edge, wherein the plurality of queues corresponds to a plurality of groups of threads; and
training, with at least one other processor, in parallel, with the plurality of groups of threads, a plurality of embeddings of a plurality of samples provided to the plurality of queues,
wherein edges for lower frequent nodes having lower degrees in the graph than other nodes having higher degrees in the graph are collected in a shared memory that is accessible by each thread of the plurality of threads, and wherein, for each lower frequent node, each edge for that lower frequent node is represented in the training for a number of times proportional to a degree of that lower frequent node.
2 . The computer-implemented method of claim 1 , wherein at least one central processing unit (CPU) executes the plurality of threads, and wherein a different graphics processing unit (GPU) of a plurality of GPUs executes each group of threads of the plurality of groups of threads.
3 . The computer-implemented method of claim 1 , further comprising:
converting, with the at least one processor, using a hash table, the graph to the graph data associated with the plurality of edges of the graph, wherein the graph data associated with the plurality of edges of the graph includes frequencies of the plurality of edges and frequencies of nodes for the plurality of edges.
4 . The computer-implemented method of claim 1 , wherein the plurality of nodes includes a plurality of different types of nodes, and wherein for each edge of the plurality of edges, the thread of the plurality of threads that read the graph data associated with that edge determines to perform the one of: (i), (ii), and (iii) based on the frequencies of the nodes for that edge only with respect to frequencies of other nodes of a same type of node of the plurality of different types of nodes.
5 . The computer-implemented method of claim 1 , wherein providing the one or more samples to the random queue of the plurality of queues further includes:
for each of the one or more samples, determining, with the at least one processor, a distance between that sample and a negative sample including at least one of a different node than that sample, a same node having a different node type than the same node of that sample, and a same edge having a different edge type than the same edge of that sample; and
in response to determining that the distance satisfies a threshold distance, providing, with the at least one processor, the negative sample to the random queue of the plurality of queues.
6 . The computer-implemented method of claim 1 , wherein training, with the at least one other processor, in parallel, with the plurality of groups of threads, the plurality of embeddings includes:
for each queue of the plurality of queues, generating, with a group of threads of the plurality of groups of threads corresponding to that queue, node embeddings for two nodes and an edge embedding for an edge connecting the two nodes, based on samples provided to that queue, using an objective function that depends on embeddings of two nodes and an embedding of an edge connecting the two nodes; and
storing, in a shared memory, with each group of threads of the plurality of groups of threads, the embeddings generated by that group of threads.
7 . The computer-implemented method of claim 6 , further comprising:
for each queue of the plurality of queues:
determining, with the at least one processor, that (i) the queue is at full capacity and (ii) the group of threads corresponding to that queue is ready for a next batch of training samples; and
in response to determining that (i) the queue is at full capacity and (ii) the group of threads corresponding to that queue is ready for the next batch of training samples, copying, with the at least one processor, the samples provided to that queue from that queue to a memory of the group of threads corresponding to that queue.
8 . The computer-implemented method of claim 6 ,
wherein the objective function is defined according to the following Equation:
L
=
∑
R
r
∈
R
∑
(
e
1
,
r
,
e
2
)
∈
R
r
σ
(
r
(
e
→
1
⊗
e
→
2
)
)
where entity types are represented as E 1 , E 2 . . . E |ε| where ε={E 1 , E 2 . . . E |ε| } is a set of all entity types, R={R 1 , R 2 , . . . R |ε| } denotes a set of relations where each relation R r :E 1 r →E 2 r is a collection of links between two entity types E 1 r , E 2 r ∈ε, each edge is denoted as (e 1 , r, e 2 ) where e 1 ∈E 1 r , e 2 r ∈E 2 r denotes head and tail entities, respectively, and r is a relation type of the head and tail entities, {right arrow over (e)} 1 , {right arrow over (e)} 2 and r respectively denote an embedding representation of the two nodes and the connecting edge of the two nodes, σ(x)=1/(1+e −x ), and ⊗ is an outer product of the embeddings {right arrow over (e)} 1 , {right arrow over (e)} 2 .
9 . A system comprising:
at least one central processing unit (CPU) programmed and/or configured to:
read, in parallel, with a plurality of threads, graph data associated with a plurality of edges and a plurality of nodes for the plurality of edges of a graph; and
for each edge of the plurality of edges, with a thread of the plurality of threads that read the graph data associated with that edge, one of: (i) discard that edge, (ii) sample that edge to generate one or more samples and provide the one or more samples to a random queue of a plurality of queues, and (iii) oversample that edge to generate the one or more samples and provide the one or more samples to the random queue of the plurality of queues, based on frequencies of nodes for that edge, wherein the plurality of queues corresponds to a plurality of groups of threads; and
a plurality of graphics processing units (GPUs), wherein a different GPU of the plurality of GPUs executes each group of threads of the plurality of groups of threads, and wherein the plurality of GPUs are programmed and/or configured to:
train, in parallel, with the plurality of groups of threads, a plurality of embeddings of a plurality of samples provided to the plurality of queues,
wherein edges for lower frequent nodes having lower degrees in the graph than other nodes having higher degrees in the graph are collected in a shared memory that is accessible by each thread of the plurality of threads, and wherein, for each lower frequent node, each edge for that lower frequent node is represented in the training for a number of times proportional to a degree of that lower frequent node.
10 . The system of claim 9 , wherein the at least one CPU is further programmed and/or configured to:
convert, using a hash table, the graph to the graph data associated with the plurality of edges of the graph, wherein the graph data associated with the plurality of edges of the graph includes frequencies of the plurality of edges and frequencies of nodes for the plurality of edges.
11 . The system of claim 9 , wherein the plurality of nodes includes a plurality of different types of nodes, and wherein for each edge of the plurality of edges, the thread of the plurality of threads that read the graph data associated with that edge determines to perform the one of: (i), (ii), and (iii) based on the frequencies of the nodes for that edge only with respect to frequencies of other nodes of a same type of node of the plurality of different types of nodes.
12 . The system of claim 9 , wherein the at least one CPU is further programmed and/or configured to provide the one or more samples to the random queue of the plurality of queues by:
for each of the one or more samples, determine a distance between that sample and a negative sample including at least one of a different node than that sample, a same node having a different node type than the same node of that sample, and a same edge having a different edge type than the same edge of that sample; and
in response to determining that the distance satisfies a threshold distance, provide the negative sample to the random queue of the plurality of queues.
13 . The system of claim 9 , wherein the plurality of GPUs are programmed and/or configured to train, in parallel, with the plurality of groups of threads, the plurality of embeddings by:
for each queue of the plurality of queues, generating, with a group of threads of the plurality of groups of threads corresponding to that queue, node embeddings for two nodes and an edge embedding for an edge connecting the two nodes, based on samples provided to that queue, using an objective function that depends on embeddings of two nodes and an embedding of an edge connecting the two nodes; and
storing, in a shared memory, with each group of threads of the plurality of groups of threads, the embeddings generated by that group of threads.
14 . The system of claim 13 , wherein the at least one CPU is further programmed and/or configured to:
for each queue of the plurality of queues:
determine, that (i) the queue is at full capacity and (ii) the GPU of the group of threads corresponding to that queue is ready for a next batch of training samples; and
in response to determining that (i) the queue is at full capacity and (ii) the group of threads corresponding to that queue is ready for the next batch of training samples, copy the samples provided to that queue from that queue to a memory of the GPU of the group of threads corresponding to that queue.
15 . The system of claim 13 , wherein the objective function is defined according to the following Equation:
L
=
∑
R
r
∈
R
∑
(
e
1
,
r
,
e
2
)
∈
R
r
σ
(
r
(
e
→
1
⊗
e
→
2
)
)
where entity types are represented as E 1 , E 2 . . . E |ε| where ε={E 1 , E 2 . . . E |ε| } is a set of all entity types, R={R 1 , R 2 , . . . R |R| } denotes a set of relations where each relation R r :E 1 r →E 2 r is a collection of links between two entity types E 1 r , E 2 r ∈ε, each edge is denoted as (e 1 , r, e 2 ) where e 1 ∈E 1 r , e 2 ∈E 2 r denotes head and tail entities, respectively, and r is a relation type of the head and tail entities, {right arrow over (e)} 1 , {right arrow over (e)} 2 and r respectively denote an embedding representation of the two nodes and the connecting edge of the two nodes, σ(x)=1/(1+e −x ), and ⊗ is an outer product of the embeddings {right arrow over (e)} 1 , {right arrow over (e)} 2 .
16 . A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:
read, in parallel, with a plurality of threads, graph data associated with a plurality of edges and a plurality of nodes for the plurality of edges of a graph;
for each edge of the plurality of edges, with a thread of the plurality of threads that read the graph data associated with that edge, one of: (i) discard that edge, (ii) sample that edge to generate one or more samples and provide the one or more samples to a random queue of a plurality of queues, and (iii) oversample that edge to generate the one or more samples and provide the one or more samples to the random queue of the plurality of queues, based on frequencies of nodes for that edge, wherein the plurality of queues corresponds to a plurality of groups of threads; and
train, in parallel, with the plurality of groups of threads, a plurality of embeddings of a plurality of samples provided to the plurality of queues,
wherein edges for lower frequent nodes having lower degrees in the graph than other nodes having higher degrees in the graph are collected in a shared memory that is accessible by each thread of the plurality of threads, and wherein, for each lower frequent node, each edge for that lower frequent node is represented in the training for a number of times proportional to a degree of that lower frequent node.
17 . The computer program product of claim 16 , wherein the at least one processor includes at least one central processing unit (CPU) and a plurality of graphics processing units (GPUs), wherein the at least one CPU executes the plurality of threads, and wherein a different GPU of the plurality of GPUs executes each group of threads of the plurality of groups of threads.
18 . The computer program product of claim 17 , wherein the plurality of groups of threads, train, in parallel, the plurality of embeddings by:
for each queue of the plurality of queues, generating, with a group of threads of the plurality of groups of threads corresponding to that queue, node embeddings for two nodes and an edge embedding for an edge connecting the two nodes, based on samples provided to that queue, using an objective function that depends on embeddings of two nodes and an embedding of an edge connecting the two nodes; and
storing, in a shared memory, with each group of threads of the plurality of groups of threads, the embeddings generated by that group of threads.
19 . The computer program product of claim 17 ,
wherein the program instructions, when executed by the at least one CPU, cause the at least one CPU to:
for each queue of the plurality of queues:
determine, that (i) the queue is at full capacity and (ii) the group of threads corresponding to that queue is ready for a next batch of training samples; and
in response to determining that (i) the queue is at full capacity and (ii) the group of threads corresponding to that queue is ready for the next batch of training samples, copy the samples provided to that queue from that queue to a memory of the group of threads corresponding to that queue.
20 . The computer program product of claim 17 ,
wherein an objective function is defined according to the following Equation:
L
=
∑
R
r
∈
R
∑
(
e
1
,
r
,
e
2
)
∈
R
r
σ
(
r
(
e
→
1
⊗
e
→
2
)
)
where entity types are represented as E 1 , E 2 . . . E |ε| where ε={E 1 , E 2 . . . E |ε| } is a set of all entity types, R={R 1 , R 2 , . . . R |R| } denotes a set of relations where each relation R r :E 1 r →E 2 r is a collection of links between two entity types E 1 r , E 2 r ∈ε, each edge is denoted as (e 1 , r, e 2 ) where e 1 ∈E 1 r , e 2 r ∈E 2 r denotes head and tail entities, respectively, and r is a relation type of the head and tail entities, {right arrow over (e)} 1 , {right arrow over (e)} 2 and r respectively denote an embedding representation of the two nodes and the connecting edge of the two nodes, σ(x)=1/(1+e −x ), and ⊗ is an outer product of the embeddings {right arrow over (e)} 1 , {right arrow over (e)} 2 .