High-performance, dynamically specifiable knowledge graph system and methods
A system and method for technology analysis utilizing high-performance, scalable, multitenant, dynamically specifiable, knowledge graph information storage and utilization. The system uses an in-memory associative array for high-performance graph storage and access, with a non-volatile distributed database for scalable backup storage, a scalable, distributed graph service for graph creation, an indexing search engine to increase searching performance, and a graph crawler for graph traversal. One or more of these components may be in the form of a cloud-based service, and in some embodiments the cloud-based services may be containerized to allow for multitenant co-existence with no possibility of data leakage or cross-over. The system uses a cyber-physical graph to represent an enterprise's cyber-physical system and can provide graph analysis, graph security, and graph fusion related tasks to identify potential operational risks.
1. A high-performance, dynamically specifiable knowledge graph system, comprising:
a computing system comprising a processor and a memory;
an in-memory associative array comprising key-value pairs stored in the memory of the computing system and configured to represent a knowledge graph;
an integration subsystem comprising a first plurality of programming instructions that, when operating on the processor, cause the computing system to:
receive one or more data clumps, each data clump comprising entity, link, and attribute data;
determine whether each data clump complies with a schema comprising one or more entity definitions, one or more link definitions, and one or more attribute definitions;
enforce compliance of each data clump by accepting compliant data clumps and rejecting non-compliant data clumps;
send the compliant data clumps to a knowledge graph subsystem for assembly into a new knowledge graph that is fully compliant with the schema; and
the knowledge graph subsystem comprising a second plurality of programming instructions that, when operating on the processor, cause the computing system to:
instantiate the knowledge graph in the in-memory associative array;
receive data clumps from the integration subsystem wherein the data clumps are known to be fully compliant with the schema; and
populate the knowledge graph in the in-memory associative array with the entity, link, and attribute data in each fully compliant data clump.
2. The system of claim 1 , further comprising a graph access control subsystem comprising a third plurality of programming instructions that, when operating on the processor, cause the computing system to assign access rights to a user, the access rights allowing the user to interact with a at least one node, and its associated edges, of the knowledge graph.
3. The system of claim 1 , wherein the integration subsystem is further configured to:
retrieve one or more known schemas from a database;
apply a known schema to the one or more data clumps;
identify any errors in the application of the known schema to the data clumps and compute an error rate based on any identified errors;
wherein if the error rate is below a predetermined threshold value the known schema is added to a list; and
display the list to a user, wherein the user can optionally select a known schema from a plurality of known schemas on the list.
4. The system of claim 1 , further comprising a graph manager subsystem comprising a fourth plurality of programming instructions that, when operating on the processor, cause the computing subsystem to:
receive a network event associated with an observed event;
apply a known schema to create a constrained knowledge graph;
analyze the constrained knowledge graph to identify an operational risk;
generate one or more subgraphs from the constrained knowledge graph based on the identified operational risk, wherein each subgraph maps the identified cybersecurity threat to a threat scenario;
perform graph fusion on the subgraphs to form a fused knowledge graph;
store the fused knowledge graph in a database; and
present the fused knowledge graph to a user for graph or temporal analysis.
5. The system of claim 1 , wherein the knowledge graph is a cyber-physical graph representing an enterprise's cyber-physical system.
6. A high-performance, dynamically specifiable knowledge graph system, comprising:
a computing system comprising a processor and a memory;
a distributed in-memory associative array comprising key-value pairs stored in the memory of a plurality of networked computing devices and configured to represent a knowledge graph;
an integration subsystem comprising a first plurality of programming instructions that, when operating on the processor, cause the computing system to:
retrieve a knowledge graph from a knowledge graph service and implement the knowledge graph in the distributed in-memory associative array;
receive one or more data clumps, each data clump comprising entity, link, and attribute data;
determine whether each data clump complies with a schema comprising one or more entity definitions, one or more link definitions, and one or more attribute definitions;
enforce compliance of each data clump by accepting compliant data clumps and rejecting non-compliant data clumps;
send each compliant data clump to the knowledge graph service for assembly into a new knowledge graph that is fully compliant with the schema, wherein the knowledge graph service:
receives data clumps from the integration subsystem wherein the data clumps are known to be fully compliant with the schema; and
populates the knowledge graph in the distributed in-memory associative array with the entity, link, and attribute data in each fully compliant data clump; and
retrieve the populated knowledge graph from the knowledge graph service and update the distributed in-memory associative array with the populated knowledge graph.
7. The system of claim 6 , wherein the integration subsystem uses a graph access control subsystem to assign access rights to a user, the access rights allowing the user to interact with a at least one node, and its associated edges, of the knowledge graph.
8. The system of claim 6 , wherein the integration subsystem is further configured to:
retrieve one or more known schemas from a database;
apply a known schema to the one or more data clumps;
identify any errors in the application of the known schema to the data clumps and compute an error rate based on any identified errors;
wherein if the error rate is below a predetermined threshold value the known schema is added to a list; and
display the list to a user, wherein the user can optionally select a known schema from a plurality of known schemas on the list.
9. The system of claim 6 , wherein the integration subsystem uses a graph manager subsystem to:
receive a network event associated with an observed event;
apply a known schema to create a constrained knowledge graph;
analyze the constrained knowledge graph to identify an operational risk;
generate one or more subgraphs from the constrained knowledge graph based on the identified cybersecurity threat, wherein each subgraph maps the identified operational risk to a threat scenario;
perform graph fusion on the subgraphs to form a fused knowledge graph;
store the fused knowledge graph in a database; and
present the fused knowledge graph to a user for graph analysis.
10. The system of claim 6 , wherein the knowledge graph is a cyber-physical graph representing an enterprise's cyber-physical system.
11. The system of claim 6 , wherein the integration subsystem, the distributed in-memory associative array, and the knowledge graph service are instantiated as services in a containerized service management application.
12. The system of claim 11 , wherein separate instances of the integration subsystem, the distributed in-memory associative array, and the knowledge graph service are used for each client as services in a containerized service management subsystem.
13. A method for operating a high-performance, dynamically specifiable knowledge graph system, comprising the steps of:
instantiating a distributed in-memory associative array comprising key-value pairs stored in the memory of a plurality of networked computing devices and configured to represent a knowledge graph;
retrieving a knowledge graph from a knowledge graph service and implementing the knowledge graph in the distributed in-memory associative array;
receiving one or more data clumps, each data clump comprising entity, link, and attribute data;
determining whether each data clump complies with a schema comprising one or more entity definitions, one or more link definitions, and one or more attribute definitions;
enforcing compliance of each data clump by accepting compliant data clumps and rejecting non-compliant data clumps;
sending each compliant data clump to the knowledge graph service, wherein the knowledge graph service:
receives compliant data clumps from the integration platform; and
populates the knowledge graph in the distributed in-memory associative array with the entity, link, and attribute data in each data clump; and
retrieving the populated knowledge graph from the knowledge graph service and updating the distributed in-memory associative array with the populated knowledge graph.
14. The method of claim 13 , wherein a graph access control service assigns access rights to a user, the access rights allowing the user to interact with at least one node, and its associated edges, of the knowledge graph.
15. The method of claim 13 , further comprising the steps of:
retrieving one or more known schemas from a database;
applying a known schema to the one or more data clumps;
identifying any errors in the application of the known schema to the data clumps and compute an error rate based on any identified errors;
wherein if the error rate is below a predetermined threshold value the known schema is added to a list; and
displaying the list to a user, wherein the user can optionally select a known schema from a plurality of known schemas on the list.
16. The method of claim 13 , wherein a graph manager service:
receives a network event associated with a cybersecurity event;
applies a known schema to create a constrained knowledge graph;
analyzes the constrained knowledge graph to identify a cybersecurity threat;
generates one or more subgraphs from the constrained knowledge graph based on the identified cybersecurity threat, wherein each subgraph maps the identified cybersecurity threat to a threat scenario;
performs graph fusion on the subgraphs to form a fused knowledge graph;
stores the fused knowledge graph in a database; and
presents the fused knowledge graph to a user for graph or temporal analysis.
17. The method of claim 13 , wherein the knowledge graph is a cyber-physical graph representing an enterprise's cyber-physical system.
18. The method of claim 13 , wherein the distributed in-memory associative array and the knowledge graph service are instantiated as services in a containerized service management application.
19. The method of claim 18 , wherein separate copies of the distributed in-memory associative array, and the knowledge graph service are instantiated for each client as services in a containerized service management application.