Multi-tenant knowledge graph databases with dynamic specification and enforcement of ontological data models
A system and method for a high-performance, scalable, multi-tenant, dynamically specifiable, knowledge graph information storage and utilization. The system uses an in-memory associative array for high-performance graph storage and access, with a non-volatile distributed database for scalable backup storage, a scalable, distributed graph service for graph creation, an indexing search engine to increase searching performance, and a graph crawler for graph traversal. One or more of these components may be in the form of a cloud-based service, and in some embodiments the cloud-based services may be containerized to allow for multi-tenant co-existence with no possibility of data leakage or cross-over.
1. A system for system for high-performance, dynamically specifiable, knowledge graph based information storage and utilization, comprising:
a computing device comprising a processor and a memory;
an in-memory associative array comprising key-value pairs stored in the memory of the computing device and configured to represent a hierarchical knowledge graph;
an integration platform comprising a first plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
receive a schema, the schema comprising an ontological abstraction of data;
compile the schema into data types, each data type comprising an entity definition, one or more link definitions, and one or more attribute definitions;
send the compiled schema to a knowledge graph engine;
receive one or more data clumps, each data clump comprising entity, link, and attribute data;
determine whether each data clump complies with the schema;
enforce compliance of each data clump by atomically accepting compliant data clumps and atomically rejecting non-compliant data clumps;
send the compliant data clumps to the knowledge graph engine for assembly into a new hierarchical knowledge graph that is created to be fully compliant with the schema;
receive a query, the query comprising an operation to be performed on the hierarchical knowledge graph;
perform the operation requested by the query; and
return a result of the query;
the knowledge graph engine comprising a second plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
receive a compiled schema from an the integration platform;
instantiate a the hierarchical knowledge graph in the in-memory associative array;
receive data clumps from the integration platform wherein the data clumps are known to be fully compliant with the schema; and
populate the hierarchical knowledge graph in the in-memory associative array with the entity, link, and attribute data in each fully-compliant data clump.
2. The system of claim 1 , further comprising non-volatile storage which is configured to back up the in-memory associative array.
3. The system of claim 1 , further comprising an indexing search engine comprising a third plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the third plurality of programming instructions, when operating on the processor, cause the computing device to index the contents of the hierarchical knowledge graph to accelerate searching.
4. The system of claim 1 , further comprising a graph crawler comprising a fourth plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the fourth plurality of programming instructions, when operating on the processor, cause the computing device to:
receive the query;
traverse the hierarchical knowledge graph to find the information requested in the query; and
return the information found.
5. The system of claim 1 , further comprising a customer access portal comprising a fifth plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the firth plurality of programming instructions, when operating on the processor, cause the computing device to allow remote access to the system by customers.
6. A system for system for high-performance, scalable, multi-tenant, dynamically specifiable, knowledge graph based information storage and utilization, comprising:
a computing device comprising a processor and a memory;
a distributed in-memory associative array comprising key-value pairs stored in the memory of a plurality of networked computing devices and configured to represent a hierarchical knowledge graph;
an integration platform comprising a first plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
receive a schema, the schema comprising an ontological abstraction of data;
compile the schema into data types, each data type comprising an entity definition, one or more link definitions, and one or more attribute definitions;
send the compiled schema to a cloud-based knowledge graph service, wherein the cloud-based knowledge graph service:
receives the compiled schema from an integration platform;
instantiates a hierarchical knowledge graph;
retrieve the hierarchical knowledge graph from the knowledge graph service and implement the hierarchical knowledge graph in the distributed in-memory associative array;
receive one or more data clumps, each data clump comprising entity, link, and attribute data;
determine whether each data clump complies with the schema;
enforce compliance of each data clump by atomically accepting compliant data clumps and atomically rejecting non-compliant data clumps;
send each compliant data clump to the cloud-based knowledge graph service engine for assembly into a new hierarchical knowledge graph that is created to be fully compliant with the schema, wherein the cloud-based knowledge graph service:
receives data clumps from the integration platform wherein the data clumps are known to be fully compliant with the schema; and
populates the hierarchical knowledge graph in the distributed in-memory associative array with the entity, link, and attribute data in each fully-compliant data clump;
retrieve the populated hierarchical knowledge graph from the knowledge graph service and update the distributed in-memory associative array with the populated knowledge graph;
receive a query, the query comprising an operation to be performed on the hierarchical knowledge graph;
perform the operation requested by the query on the hierarchical knowledge graph in the distributed in-memory associative array; and
return a result of the query.
7. The system of claim 6 , wherein the integration platform uses a cloud-based non-volatile storage service which is configured to back up the in-memory associative array.
8. The system of claim 6 , wherein the integration platform uses a cloud- based indexing search engine service which is configured to index the contents of the hierarchical knowledge graph to accelerate searching.
9. The system of claim 6 , wherein the integration platform uses a cloud-based graph crawler service which is configured to:
receive the query;
traverse the hierarchical knowledge graph to find the information requested in the query; and
return the information found.
10. The system of claim 6 , further comprising a customer access portal comprising a second plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to allow remote access to the system by customers.
11. The system of claim 6 , wherein the integration platform, the distributed in-memory associative array, and the cloud-based knowledge graph service are instantiated as services in a containerized service management application.
12. The system of claim 11 , wherein separate copies of the integration platform, the distributed in-memory associative array, and the cloud-based knowledge graph service are instantiated for each client as services in a containerized service management application.
13. A method for system for high-performance, scalable, multi-tenant, dynamically specifiable, knowledge graph based information storage and utilization, comprising the steps of:
instantiating a distributed in-memory associative array comprising key-value pairs stored in the memory of a plurality of networked computing devices and configured to represent a hierarchical knowledge graph;
receiving a schema, the schema comprising an ontological abstraction of data;
compiling the schema into data types, each data type comprising an entity definition, one or more link definitions, and one or more attribute definitions;
sending the compiled schema to a cloud-based knowledge graph service, wherein the cloud-based knowledge graph service:
receives the compiled schema from an integration platform; and
instantiates a hierarchical knowledge graph;
retrieving the hierarchical knowledge graph from the knowledge graph service and implementing the hierarchical knowledge graph in the distributed in-memory associative array;
receiving one or more data clumps, each data clump comprising entity, link, and attribute data;
determining whether each data clump complies with the schema;
enforcing compliance of each data clump by atomically accepting compliant data clumps and atomically rejecting non-compliant data clumps;
sending each compliant data clump to the cloud-based knowledge graph service, wherein the cloud-based knowledge graph service:
receives compliant data clumps from the integration platform; and
populates the hierarchical knowledge graph in the distributed in-memory associative array with the entity, link, and attribute data in each data clump;
retrieving the populated hierarchical knowledge graph from the knowledge graph service and updating the distributed in-memory associative array with the populated hierarchical knowledge graph;
receiving a query, the query comprising an operation to be performed on the hierarchical knowledge graph;
performing the operation requested by the query on the hierarchical knowledge graph in the distributed in-memory associative array; and
returning a result of the query.
14. The method of claim 13 , wherein the integration platform uses a cloud-based non-volatile storage service which is configured to back up the in-memory associative array.
15. The method of claim 13 , wherein a cloud-based indexing search engine service is used and is configured to index the contents of the hierarchical knowledge graph to accelerate searching.
16. The method of claim 13 , wherein a cloud-based graph crawler service is used and is configured to:
receive the query;
traverse the hierarchical knowledge graph to find the information requested in the query; and
return the information found.
17. The method of claim 13 , further comprising the step of establishing a customer access portal to allow remote access to the system by customers.
18. The method of claim 13 , wherein the distributed in-memory associative array and the cloud-based knowledge graph service are instantiated as services in a containerized service management application.
19. The method of claim 18 , wherein separate copies of the distributed in-memory associative array, and the cloud-based knowledge graph service are instantiated for each client as services in a containerized service management application.