Database architecture for storing multi-structured data
In various embodiments, a system for ingesting, organizing, and storing data is disclosed. The system comprises a processor and a memory unit. A document database is stored by the memory unit. The document database is configured to store data without any foreknowledge of a structure of the data. The data comprises an ontology comprising a plurality of terms describing the data within the document database. A graph database is stored by the memory unit. The graph database comprises a materialized view of the document database. The graph database provides connections between the ontology. The document database and the graph database share common entity profile UUIDs.
1. A data management system for ingesting, organizing, and storing data, the data management system comprising:
a processor;
a memory unit; and
a two-tiered database architecture, comprising:
a document database stored by the memory unit, wherein the document database is configured to store data without any foreknowledge of a structure of the data, wherein the data comprises an ontology comprising a plurality of terms describing the data within the document database; and
a graph database stored by the memory unit, wherein the graph database is configured to index and query underlying data stored in the document database, wherein the graph database comprises a materialized view of the document database to facilitate navigation of the ontology in the document database, wherein the graph database provides connections between the ontology, and wherein the document database and the graph database share common entity profile universally unique identifier (UUID).
2. The data management system of claim 1 , wherein the ontology comprises a provider-defined ontology, wherein the provider-defined ontology comprises a plurality of provider-defined terms describing the data within the document database.
3. The data management system of claim 2 , wherein the ontology comprises a user-defined ontology, wherein the user-defined ontology comprises a plurality of user defined terms describing the data within the document database.
4. The data management system of claim 3 , wherein the ontology comprises a merged ontology, wherein the merged ontology comprises the plurality of provider-defined terms and the plurality of user defined terms, and wherein, when a provider-defined term and a user defined term conflict, the merged ontology comprises the user defined term.
5. The data management system of claim 1 , wherein the data stored in the document database comprises at least one organic attribute, wherein the at least one organic attribute comprises a feature of an entity represented by the data stored in the document database, and wherein the organic attribute is included in the ontology.
6. The data management system of claim 5 , wherein the data stored in the document database comprises at least one derived attribute, wherein the at least one derived attribute comprises a calculated feature of the entity based on at least one organic attribute, and wherein the derived attribute is included in the ontology.
7. The data management system of claim 1 , comprising an application programming interface (API) executed by the processor, wherein the API is configured to:
query the graph database based on at least one user provided term, wherein the user provided term matches at least one term of the ontology, and wherein the graph database returns the node IDs corresponding to at least one node matching the at least one user provided term; and
retrieve at least one document from the document database, wherein the at least one documents comprises the node IDs returned by the graph database.
8. The data management system of claim 1 , wherein the document database and the graph database are stored in a cell-based architecture, wherein the document database and the graph database are isolated from additional data stored by the memory unit.
9. The data management system of claim 1 , wherein the ontology comprises a hierarchical ontology, wherein the plurality of terms comprise at least one parent-child relationship.
10. A computer-implemented method for ingesting, organizing, and accessing data, the method comprising:
storing, by a memory unit, a two-tiered database architecture, comprising: a document database configured to store data without any foreknowledge of a structure of the data, wherein the data comprises an ontology comprising a plurality of terms describing the data;
storing, by a memory unit, a graph database wherein the graph database is configured to index and query underlying data stored in the document database, the graph database comprising a materialized view of the document database to facilitate navigation of the ontology in the document database, wherein the graph database provides connections between the plurality of terms, and wherein the document database and the graph database share common entity profile universally unique identifier (UUID);
accessing, by the processor, data in the document database through the connections of the graph database.
11. The computer-implemented method of claim 10 , wherein the ontology comprises a provider-defined ontology, wherein the provider-defined ontology comprises a plurality of provider-defined terms describing the data within the document database.
12. The computer-implemented method of claim 11 , wherein the ontology comprises a user-defined ontology, wherein the user-defined ontology comprises a plurality of user defined terms describing the data within the document database.
13. The computer-implemented method of claim 12 , wherein the ontology comprises a merged ontology, wherein the merged ontology comprises the plurality of providerdefined terms and the plurality of user defined terms, and wherein, when a provider-defined term and a user defined term conflict, the merged ontology comprises the user defined term.
14. The computer-implemented method of claim 10 , wherein the data stored in the document database comprises at least one organic attribute, wherein the at least one organic attribute comprises a feature of an entity represented by the data stored in the document database, and wherein the organic attribute is included in the ontology.
15. The computer-implemented method of claim 14 , wherein the data stored in the document database comprises at least one derived attribute, wherein the at least one derived attribute comprises a calculated feature of the entity based on at least one organic attribute, and wherein the derived attribute is included in the ontology.
16. The computer-implemented method of claim 10 , comprising:
querying, by an application programming interface, the graph database based on at least one user provided term, wherein the user provided term matches at least one term of the ontology, and wherein the graph database returns the node IDs corresponding to at least one node matching the at least one user provided term; and
retrieving, by the application programming interface, at least one document from the document database, wherein the at least one documents comprises the node IDs returned by the graph database.
17. The computer-implemented method of claim 10 , comprising storing, by the memory unit, the document database and the graph database in a cell-based architecture, wherein the document database and the graph database are isolated from additional data stored by the memory unit.
18. The computer-implemented method of claim 10 , wherein the ontology comprises a hierarchical ontology, and wherein the plurality of terms comprises at least one parent-child relationship.
19. A non-transitory computer-readable medium comprising a plurality of instructions which may be executed by a computer processor, wherein the plurality of instructions causes the processor to:
store a document database in a memory unit, a two-tiered database architecture, comprising: a document database configured to store data without any foreknowledge of a structure of the data, wherein the data comprises an ontology comprising a plurality of terms describing the data; and
store a graph database in the memory unit, wherein the graph database is configured to index and query underlying data stored in the document database, wherein the graph database comprises a materialized view of the document database to facilitate navigation of the ontology in the document database, wherein the graph database provides connections between the ontology, and wherein the document database and the graph database share common entity profile universally unique identifier (UUID).
20. The non-transitory computer-readable medium of claim 19 , wherein the plurality of instructions further causes the processor to:
query the graph database based on at least one user provided term, wherein the user provided term matches at least one term of the ontology, and wherein the graph database returns the node IDs corresponding to at least one node matching the at least one user provided term; and
retrieve at least one document from the document database, wherein the at least one documents comprises the node IDs returned by the graph database.