Efficient data indexing across multiple ontology-based databases
A system for indexing changes to an ontology into multiple databases and related methods are disclosed. The system is programmed to receive original data from data sources, transform the original data to ontology data, represent the ontology data in multiple forms respectively in the multiple databases, and process requests to access the ontology data from user accounts using the multiple databases. The system is programmed to subsequently merge changes to the ontology data based on updates from the data sources and edits from user accounts, create index data for the merged changes, and transmit the index data to the multiple databases.
1 . A computer-implemented method of indexing data changes to ontology-based databases, comprising:
receiving a set of data source updates for one or more datasets from one or more data sources;
receiving one or more sets of user edits to an ontology,
the ontology including a definition for each ontology entity type of a plurality of ontology entity types,
the ontology including a plurality of ontology entities instantiated from the plurality of ontology entity types,
at least one ontology entity of the plurality of ontology entities being represented in multiple forms respectively in multiple object databases of a plurality of object databases;
transforming the set of data source updates to a list of updates to the ontology based on a first mapping between the one or more datasets and the plurality of ontology entity types;
merging the list of updates to the ontology with the one or more sets of user edits into a merged dataset;
generating index data from the merged dataset for one or more object databases of the plurality of object databases based on a second mapping between the plurality of ontology entity types and the plurality of object databases;
transmitting the index data to the one or more object databases,
wherein the method is performed using one or more processors.
2 . The computer-implemented method of claim 1 , further comprising:
representing a data source of the one or more data sources by a dataset of the one or more datasets;
adding a record to the first mapping that associates the dataset with one or more ontology entity types of the plurality of ontology entity types;
creating a transformation between a schema for the dataset and the one or more definitions for the one or more ontology entity types;
generating one or more ontology entities from the dataset using the transformation.
3 . The computer-implemented method of claim 1 , multiple datasets of the one or more datasets being mapped to a common ontology entity type in the first mapping.
4 . The computer-implemented method of claim 1 , the plurality of object databases including a graph database and a relational database respectively associated with different data representations and query operations.
5 . The computer-implemented method of claim 1 , further comprising:
representing a data source of the one or more data sources by a dataset of the one or more datasets;
receiving original data from the data source;
determining changes to the dataset from the original data;
adding the changes to a changelog dataset.
6 . The computer-implemented method of claim 5 , the merging comprising:
joining the changelog dataset and the one or more sets of user edits based on a primary key of a certain ontology entity;
storing the merged dataset in a memory, thereby materializing the one or more sets of user edits.
7 . The computer-implemented method of claim 1 , the merging being performed when a size of the list of updates or a size of the one or more sets of user edits has exceeded a threshold or when a certain amount of time has passed.
8 . The computer-implemented method of claim 1 , the second mapping associating each ontology entity type with at least one object database of the plurality of object databases.
9 . The computer-implemented method of claim 8 ,
the merged dataset including changes to one or more ontology entities of a set of ontology entity types of the plurality of ontology entity types,
the generating comprising, for a specific object database of the plurality of object databases, creating one or more index files from the changes to the one or more ontology entities for the set of ontology entity types that are associated with the specific object database,
the one or more index files being in a specific format required by the specific object database.
10 . The computer-implemented method of claim 1 ,
each ontology entity type of the plurality of ontology entity types having one or more properties,
the index data associating a value for a property of an ontology entity type of the plurality of ontology entity types with a list of ontology entities of the plurality of ontology entities.
11 . The computer-implemented method of claim 1 ,
a user edit of the one or more sets of user edits being a change to an ontology entity of the plurality of ontology entities,
each set of user edits of the one or more sets of user edits to be applied as a single transaction to a user.
12 . The computer-implemented method of claim 1 , an ontology entity being an object or a link between two objects.
13 . A system for indexing data changes to ontology-based databases, comprising:
a memory;
one or more processors coupled to the memory and configured to perform:
receiving a set of data source updates for one or more datasets from one or more data sources;
receiving one or more sets of user edits to an ontology,
the ontology including a definition for each ontology entity type of a plurality of ontology entity types,
the ontology including a plurality of ontology entities instantiated from the plurality of ontology entity types,
at least one ontology entity of the plurality of ontology entities being represented in multiple forms respectively in multiple object databases of a plurality of object databases;
transforming the set of data source updates to a list of updates to the ontology based on a first mapping between the one or more datasets and the plurality of ontology entity types;
merging the list of updates to the ontology with the one or more sets of user edits into a merged dataset;
generating index data from the merged dataset for one or more object databases of the plurality of object databases based on a second mapping between the plurality of ontology entity types and the plurality of object databases;
transmitting the index data to the one or more object databases.
14 . The system of claim 13 , the plurality of object databases including a graph database and a relational database respectively associated with different data representations and query operations.
15 . The system of claim 13 , the one or more processors further configured to perform:
representing a data source of the one or more data sources by a dataset of the one or more datasets;
receiving original data from the data source;
determining changes to the dataset from the original data;
adding the changes to a changelog dataset.
16 . The system of claim 15 , the merging comprising:
joining the changelog dataset and the one or more sets of user edits based on a primary key of a certain ontology entity;
storing the merged dataset in a memory, thereby materializing the one or more sets of user edits.
17 . The system of claim 13 , the second mapping associating each ontology entity type with at least one object database of the plurality of object databases.
18 . The system of claim 17 ,
the merged dataset including changes to one or more ontology entities of a set of ontology entity types of the plurality of ontology entity types,
the generating comprising, for a specific object database of the plurality of object databases, creating one or more index files from the changes to the one or more ontology entities for the set of ontology entity types that are associated with the specific object database,
the one or more index files being in a specific format required by the specific object database.
19 . The system of claim 13 ,
a user edit of the one or more sets of user edits being a change to an ontology entity of the plurality of ontology entities,
each set of user edits of the one or more sets of user edits to be applied as a single transaction to a user.
20 . A non-transitory, computer-readable storage medium storing one or more sequences of instructions which when executed cause one or more processors to perform:
receiving a set of data source updates for one or more datasets from one or more data sources;
receiving one or more sets of user edits to an ontology,
the ontology including a definition for each ontology entity type of a plurality of ontology entity types,
the ontology including a plurality of ontology entities instantiated from the plurality of ontology entity types,
at least one ontology entity of the plurality of ontology entities being represented in multiple forms respectively in multiple object databases of a plurality of object databases;
transforming the set of data source updates to a list of updates to the ontology based on a first mapping between the one or more datasets and the plurality of ontology entity types;
merging the list of updates to the ontology with the one or more sets of user edits into a merged dataset;
generating index data from the merged dataset for one or more object databases of the plurality of object databases based on a second mapping between the plurality of ontology entity types and the plurality of object databases;
transmitting the index data to the one or more object databases.