Hybrid data integration platform
In general, embodiments of the present invention provide systems and computer readable media for implementing a single data integration platform that supports multiple data access interfaces to a single corpus of stored dynamic data collected from multiple data sources. In embodiments, the data integration platform includes a record tables layer that stores a group of data records and supports a CRUD interface for accessing the data records; a resolution mapping layer that stores a set of entities generated by a many-to-one mapping of data records to entities using entity resolution; and an entities layer that stores resolved entities which may be accessed via either a search interface based on search criteria or a hybrid search interface that supports “get via record id” queries.
1. A system comprising one or more computers and one or more storage devices for implementing a data integration platform that supports multiple data access interfaces to a single corpus of stored dynamic data collected from multiple data sources, the system comprising:
the single corpus of stored dynamic data; and
the data integration platform for accessing the single corpus of stored dynamic data via a plurality of data access interfaces, wherein the data integration platform comprises:
a records layer for storing a group of data records,
wherein each data record includes a permanent record identifier and a data payload associated with the record identifier, and
wherein the records layer implements a creating, reading, updating, and deleting (CRUD) interface for accessing the data records;
a resolution mapping layer for generating a group of entity records using the group of data records,
wherein each entity record includes an entity identifier and a set of record identifiers representing a set of the data records that have been determined, using an entity resolution system, to represent a unique entity;
an entities layer for storing a group of resolved entities derived from the entity records,
wherein each resolved entity includes a resolved entity identifier and a resolution payload that includes data from the data payload of at least one of the data records, and
wherein the entities layer implements a search interface for receiving search queries to resolve using the resolved entities; and
a hybrid search interface, implemented by the entities layer, for providing access to each of a plurality of hybrid clients to resolved data entities that include data created by the accessing hybrid client, receiving an update to one or more resolved data entities, wherein the receiving of the update to the one or more resolved data entities is a trigger event, and causing, upon determination of the trigger event, a new set of resolved data entities to be generated.
2. The system of claim 1 , wherein the records layer implements an operation specifying CRUD semantics to be applied to at least one of the data records in response to receiving a request from the CRUD interface.
3. The system of claim 2 , wherein the group of entity records is updated by the entity resolution system in response to a notification of the implementation of the operation specifying CRUD semantics by the records layer.
4. The system of claim 1 , wherein generating the group of entity records using the group of data records comprises data quality operations including at least one of de-duplication, enrichment, and normalization of the group of data records.
5. The system of claim 1 , wherein determining that a set of data records represents a unique entity comprises generating a new entity record by merging at least two of the group of entity records.
6. The system of claim 5 , wherein the new entity record is associated with each of the entity identifiers respectively associated with each of the merged entity records.
7. The system of claim 1 , wherein determining that a set of data records represents a unique entity comprises generating at least two new entity records by splitting one of the group of entity records.
8. The system of claim 7 , wherein one of the new entity records is determined, based on a heuristic, to be most representative of the entity record that was split, and the most representative new entity record is associated with the entity identifier of the entity record that was split.
9. The system of claim 1 , wherein, in response to a hybrid search query including a particular record identifier received from the hybrid search interface, the entities layer generates a search result including a resolved entity including data from the data payload of a data record associated with the particular record identifier.
10. The system of claim 1 , wherein, in response to a search query including a set of search criteria, the entities layer generates a search result including a set of the resolved entities, each resolved entity including attributes determined to match the set of search criteria.
11. The system of claim 1 , wherein the search interface includes a search engine.
12. A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed on one or more computers cause the one or more computers to perform operations implementing a data integration platform that supports multiple data access interfaces to a single corpus of stored dynamic data collected from multiple data sources comprising:
the single corpus of stored dynamic data; and
the data integration platform for accessing the single corpus of stored dynamic data via a plurality of data access interfaces, wherein the data integration platform comprises:
a records layer for storing a group of data records,
wherein each data record includes a permanent record identifier and a data payload associated with the record identifier, and
wherein the records layer implements a creating, reading, updating, and deleting (CRUD) interface for accessing the data records;
a resolution mapping layer for generating a group of entity records using the group of data records,
wherein each entity record includes an entity identifier and a set of record identifiers representing a set of the data records that have been determined, using an entity resolution system, to represent a unique entity; and
an entities layer for storing a group of resolved entities derived from the entity records,
wherein each resolved entity includes a resolved entity identifier and a resolution payload that includes data from the data payload of at least one of the data records, and wherein the entities layer implements a search interface for receiving search queries to resolve using the resolved entities; and
a hybrid search interface, implemented by the entities layer, for providing access to each of a plurality of hybrid clients to resolved data entities that include data created by the accessing hybrid client, receiving an update to one or more resolved data entities, wherein the receiving of the update to the one or more resolved data entities is a trigger event, and causing, upon determination of the trigger event, a new set of resolved data entities to be generated.
13. The computer program product of claim 12 , wherein the records layer implements an operation specifying CRUD semantics to be applied to at least one of the data records in response to receiving a request from the CRUD interface.
14. The computer program product of claim 13 , wherein the group of entity records is updated by the entity resolution system in response to a notification of the implementation of the operation specifying CRUD semantics by the records layer.
15. The computer program product of claim 12 , wherein generating the group of entity records using the group of data records comprises data quality operations including at least one of de-duplication, enrichment, and normalization of the group of data records.
16. The computer program product of claim 12 , wherein determining that a set of data records represents a unique entity comprises generating a new entity record by merging at least two of the group of entity records.
17. The computer program product of claim 16 , wherein the new entity record is associated with each of the entity identifiers respectively associated with each of the merged entity records.
18. The computer program product of claim 12 , wherein determining that a set of data records represents a unique entity comprises generating at least two new entity records by splitting one of the group of entity records.
19. The computer program product of claim 18 , wherein one of the new entity records is determined, based on a heuristic, to be most representative of the entity record that was split, and the most representative new entity record is associated with the entity identifier of the entity record that was split.
20. The computer program product of claim 12 , wherein, in response to a hybrid search query including a particular record identifier received from the hybrid search interface, the entities layer generates a search result including a resolved entity including data from the data payload of a data record associated with the particular record identifier.
21. The computer program product of claim 12 , wherein, in response to a search query including a set of search criteria, the entities layer generates a search result including a set of the resolved entities, each resolved entity including attributes determined to match the set of search criteria.
22. The computer program product of claim 12 , wherein the search interface includes a search engine.