Systems and methods for machine learning models for entity resolution
Methods, systems, and computer-readable media for linking multiple data entities. The method collects a snapshot of data from one or more data sources and converts it into a canonical representation of records expressing relationships between data elements in the records. The method next cleans the records to generate output data of entities by grouping chunks of records using a machine learning model. The method next ingests the output data of entities to generate a versioned data store of the entities and optimizes versioned data store for real-time data lookup. The method then receives a request for data pertaining to a real-world entity and presenting relevant data from the versioned data store of entities.
1. A non-transitory computer readable medium including instructions that are executable by one or more processors to cause a system to perform operations for linking multiple data entities, the operations comprising:
indexing entity instances;
mapping the entity instances to blocking key values by applying blocking functions;
generating versioned datasets for the entity instances;
identifying levels of evidence of relationship between the entity instances using a machine learning model and the versioned datasets;
indexing entity identifiers under the blocking functions by generating mapping tables, for updating the levels of evidence of relationship between the entity instances; and
persisting one or more mappings in the mapping tables to the entity instances based on the levels of evidence of relationship.
2. The non-transitory computer readable medium of claim 1 , wherein the entity instances comprise one or more shallow entity instances, wherein the shallow entity instances comprise a plurality of vector fields mapping to at most one data element.
3. The non-transitory computer readable medium of claim 2 , wherein the entity instances further comprise deep entity instances generated by coalescing a plurality of the shallow entity instances identified by coarse and fine identifiers.
4. The non-transitory computer readable medium of claim 1 , wherein the blocking key values are generated by stringifying the entity instances.
5. The non-transitory computer readable medium of claim 1 , wherein indexing the entity identifiers under the blocking functions comprise: using a blocking function key and a version number as an index to the mapping tables.
6. The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
generating entity specific tables by grouping a set of the entity identifiers and the entity instances identified by the entity identifiers.
7. The non-transitory computer readable medium of claim 6 , wherein persisting the one or more mappings in the mapping tables to the entity instances comprises:
storing the mapping tables and the entity specific tables to create a mapping between the blocking functions and the entity instances.
8. A method for linking multiple data entities, the method performed by one or more processors and comprising:
indexing entity instances;
mapping the entity instances to blocking key values by applying blocking functions;
generating versioned datasets for the entity instances;
identifying levels of evidence of relationship between the entity instances using a machine learning model and the versioned datasets;
indexing entity identifiers under the blocking functions by generating mapping tables, for updating the levels of evidence of relationship between the entity instances; and
persisting one or more mappings in the mapping table to the entity instances based on the levels of evidence of relationship.
9. The method of claim 8 , wherein the entity instances comprise one or more shallow entity instances, wherein the shallow entity instances comprise a plurality of vector fields mapping to at most one data element.
10. The method of claim 9 , wherein the entity instances further comprise deep entity instances generated by coalescing a plurality of the shallow entity instances identified by coarse and fine identifiers.
11. The method of claim 8 , wherein the blocking key values are generated by stringifying the entity instances.
12. The method of claim 8 , wherein indexing the entity identifiers under the blocking functions comprise: using a blocking function key and a version number as an index to the mapping tables.
13. The method of claim 8 , further comprising:
generating entity specific tables by grouping a set of the entity identifiers and the entity instances identified by the entity identifiers.
14. The method of claim 13 , wherein persisting the one or more mappings in the mapping tables to the entity instances comprises:
storing the mapping tables and the entity specific tables to create a mapping between the blocking functions and the entity instances.
15. A computer-implemented system for linking multiple data entities, the system comprising:
at least one non-transitory computer-readable medium configured to store instructions; and
at least one processor configured to execute the instructions to cause the system to perform operations comprising:
indexing entity instances;
mapping the entity instances to blocking key values by applying blocking functions;
generating versioned datasets for the entity instances;
identifying levels of evidence of relationship between the entity instances using a machine learning model and the versioned datasets;
indexing entity identifiers under the blocking functions by generating mapping tables for updating the levels of evidence of relationship between the entity instances; and
persisting one or more mappings in the mapping table to the entity instances based on the levels of evidence of relationship.
16. The system of claim 15 , wherein the entity instances comprise one or more shallow entity instances, wherein the shallow entity instances comprise a plurality of vector fields mapping to at most one data element.
17. The system of claim 15 , wherein the blocking key values are generated by stringifying the entity instances.
18. The system of claim 15 , wherein indexing the entity identifiers under the blocking functions comprise: using a blocking function key and a version number as an index to the mapping tables.
19. The system of claim 15 , wherein the operations further comprise:
generating entity specific tables by grouping a set of the entity identifiers and the entity instances identified by the entity identifiers.
20. The system of claim 19 , wherein persisting the one or more mappings in the mapping tables to the entity instances comprises:
storing the mapping tables and the entity specific tables to create a mapping between the blocking functions and the entity instances.