Identity resolution in big, noisy, and/or unstructured data
In an environment containing big data, noisy data, and/or unstructured data, it is desirable to identify an entity referenced by input data. The entity can be identified by generating records corresponding to characteristics of the entity based on the input data. These records can be merged when it is determined that more than one record corresponds to the same entity. By doing so it is possible to more easily identify and classify information related to an entity, though such information may have been obtained in a manner that might otherwise be deemed unstructured or noisy. The method can be applied across large sets of data (“big data”) to obtain meaning from data that may otherwise be unclassifiable to a human observer.
1. A method of managing data relating to an entity, the method comprising:
identifying, by at least one device comprising a processor, an entity referenced by a plurality of input text sources;
generating, by the at least one device, a first node record corresponding to the entity in a database;
identifying a second entity referenced by one or more of the plurality of input text sources;
generating a second node record in the database corresponding to the second entity;
populating, by the at least one device, the first node record with first characteristics of the first entity extracted from at least one of the plurality of input text sources, wherein the first characteristics comprise connection characteristics regarding a social connection between the first entity and the second entity, employment information regarding the first entity, and one or more reliability scores regarding reliability of the input data;
populating, by the at least one device, the second node record with first characteristics of the second entity extracted from at least one of the plurality of input text sources;
calculating, by the at least one device, a numeric connection strength representative of a strength of the social connection between the first entity and the second entity taking into account at least one reliability score regarding reliability of the input data;
generating a link record in the database corresponding to a connection between the first entity and the second entity;
assigning, by the at least one device, the numeric connection strength to the link record, wherein the numeric connection strength is based upon the social connection between the first entity and the second entity;
automatically comparing, by the at least one device, characteristics populated in the first node record with respective characteristics populated in a plurality of respective other node records of the database; and
automatically merging, by the at least one device, the first node record with a third record in the database in response to a result of determining through the comparing that the first node record and the third node record both correspond to the first entity, resulting in a merged node record corresponding to the first entity.
2. The method of claim 1 , further comprising:
calculating, by the at least one device, respective identity probabilities for the plurality of respective other node records of the database s based on the comparing, the identity probabilities corresponding to probability of the entity corresponding to the first entity being a same entity as respective entities corresponding to the respective other node records of the database,
wherein the merging comprises merging the first node record with one or more second node records in response to an identity probability for the one or more second node records being higher than a threshold.
3. The method of claim 2 , wherein the merging further comprises including an identity probability for the second node record in the merged node record.
4. The method of claim 2 , wherein the calculating comprises:
assigning reliability scores to a plurality of respective characteristics populated in the first node record; and
weighting the plurality of respective characteristics populated in the first record according to the reliability scores.
5. The method of claim 4 , wherein the assigning comprises assigning the reliability scores based on one or more of an age of the plurality of input text sources, a reliability of the plurality of input text sources, frequency of a corresponding characteristic appearing in the plurality of input text sources, or a type of the corresponding characteristic.
6. The method of claim 1 , wherein the comparing comprises:
comparing at least one of the connection characteristics regarding a social connection populated in the first node record with at least one respective connection characteristic regarding a social connection populated in at least one of the plurality of respective other node records of the database; and
comparing the numeric connection strength with at least one corresponding numeric connection strength populated in at least one of the plurality of respective other node records of the database.
7. The method of claim 1 , wherein the computing a connection strength of a social connection comprises computing the connection strength based on at least one of length of the social connection, mutual connection between parties to the social connection, nature of interactions between the parties to the social connection, or frequency of the social connection being referenced in the input text source.
8. The method of claim 1 , wherein:
the first entity is a person, and
the characteristics of the first entity comprise at least one of biographical information, employment information, or family information.
9. The method of claim 1 , wherein:
The first entity is an organization, and
the characteristics of the first entity comprise at least one of operating information, business information, membership information, or characteristics of one or more members of the organization.
10. The method of claim 1 , wherein the merging comprises:
updating the second node record with one or more characteristics from the first node record, and discarding the first node record.
11. A system, comprising:
a database comprising a plurality of records corresponding to respective entities, wherein respective ones of the records are populated with characteristics of a respectively corresponding entity;
at least one processor; and
a memory that stores processor-executable instructions, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
identifying a first entity referenced by a plurality of input text sources;
generating a first node record in the database corresponding to the first entity;
identifying a second entity referenced by one or more of the plurality of input text sources;
generating a second node record in the database corresponding to the second entity;
populating the first node record with first characteristics of the first entity extracted from at least one of the plurality of input text sources, wherein the first characteristics comprise connection characteristics regarding a social connection between the first entity and the second entity, employment information regarding the first entity, and one or more reliability scores regarding reliability of the input data;
populating the second node record with first characteristics of the second entity extracted from at least one of the plurality of input text sources;
calculating a numeric connection strength representative of a strength of the social connection between the first entity and the second entity taking into account at least one reliability score regarding reliability of the input data;
generating a link record in the database corresponding to a connection between the first entity and the second entity;
assigning a numeric connection strength to the link record, wherein the numeric connection strength is based upon the social connection between the first entity and the second entity;
comparing characteristics populated in the first node record with respective characteristics populated in respective other node records of the database; and
merging the first node record with a third node record in the database in response to a result of determining through the comparing that the first node record and the third node record both correspond to the first entity, resulting in a merged node record corresponding to the first entity.
12. The system of claim 11 , wherein the operations further comprise:
calculating respective identity probabilities for the respective other node records of the database based on the comparing, the identity probabilities corresponding to probability of the first entity being a same entity as respective entities corresponding to the respective other node records of the database; and
merging the first node record with one or more second node records in response to an identity probability for the one or more second node records being higher than a threshold.
13. The system of claim 11 , wherein the operations further comprise:
assigning reliability scores to a plurality of respective characteristics populated in the first record; and
weighting the plurality of respective characteristics populated in the first node record according to the reliability scores.
14. The system of claim 11 , wherein the operations further comprise:
comparing the characteristics populated in the first node record with the respective characteristics populated in the respective other node records of the database at least in part by comparing the respective social connection and corresponding connection strength populated in the first node record with respective social connection and corresponding connection strength populated in the respective other node records of the database.
15. The system of claim 11 , wherein the operations further comprise:
computing the connection strength of a social connection based on at least one of length of the social connection, mutual connections between parties to the social connection, nature of interactions between the parties to the social connection, or frequency of the social connection being referenced in the input text source.
16. A non-transitory computer-readable medium comprising processor-executable instructions which, when executed by at least one processor, facilitate performance of operations, the operations comprising:
identifying a first entity referenced by at least one of a plurality of input data sources;
generating a first node record corresponding to the first entity in a database comprising a plurality of records;
identifying a second entity referenced by one or more of the plurality of input text sources;
generating a second node record in the database corresponding to the second entity;
populating, by the at least one device, the first node record with first characteristics of the entity extracted from at least one of the plurality of input text sources, wherein the first characteristics comprise connection characteristics regarding a social connection between the first entity and the second entity, employment information regarding the first entity, and one or more reliability scores regarding reliability of the input data;
populating the second node record with first characteristics of the second entity extracted from at least one of the plurality of input text sources;
calculating a numeric connection strength representative of a strength of the social connection between the first entity and the second entity taking into account at least one reliability score regarding reliability of the input data;
generating a link record in the database corresponding to a connection between the first entity and the second entity;
assigning a numeric connection strength to the link record, wherein the numeric connection strength is based upon the social connection between the first entity and the second entity;
comparing, by the at least one device, characteristics populated in the first node record with respective characteristics populated in respective other node records of the database; and
merging the first node record with a third node record in the database in response to a result determining through the comparing that the first node record and the third node record both correspond to the first entity, resulting in a merged node record corresponding to the first entity.
17. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
calculating respective identity probabilities for the respective other node records of the database based on the comparing, the identity probabilities corresponding to probability of the first entity being a same entity as respective entities corresponding to the respective other node records of the database; and
merging the first node record with one or more second node records in response to an identity probability for the one or more second node records being higher than a threshold.
18. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
assigning reliability scores to a plurality of respective characteristics populated in the first record; and
weighting the plurality of respective characteristics populated in the first record according to the reliability scores.
19. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
comparing the characteristics populated in the first node record with the respective characteristics populated in the respective other node records of the database at least in part by comparing the respective social connection and corresponding connection strength populated in the first node record with respective social connection and corresponding connection strength populated in the respective other node records of the database.