Systems and methods for large scale global entity resolution

1. A computer-implemented method, comprising:

ingesting text data from a plurality of documents containing a plurality of mentions;

locating, from the text data, for each of a selected plurality of chains of coreferent mentions, a particular context-based name from the respective chain, wherein the coreferent mentions correspond to entities and the context-based name is a longest name in the respective chain, a last name in the respective chain, or a most frequently occurring name in the respective chain;

determining an entity category for each respective one of the plurality of chains;

determining one or more entity attributes from structured data and unstructured data;

based on the located particular context-based name, the entity category, and the one or more attributes, assigning high-probability coreferent chains to high-confidence buckets, such as to produce a power law probability distribution having a head region and a tail region; and

resolving, based at least in part on the power law probability distribution, the coreferent mentions to identify corresponding real-world entities.

2. The method of claim 1 , wherein the power law probability distribution is produced such that the respective rank of a particular chain corresponding to a context-based name is inversely proportional to the frequency of the context-based name across a document corpus.

3. The method of claim 1 , wherein assigning the high-probability chains to the high-confidence buckets comprises:

grouping the plurality of chains based on the respective context-based name and category such that chains having a same context-based name and same category are grouped together into a respective partition;

within the respective partition, grouping chains that correspond to the same concept into sub-entities; and

grouping together the sub-entities, across and within partitions, that correspond to the same concept.

4. The method of claim 1 , further comprising:

assigning one or more low-frequency sub-entities into the head region; and

assigning one or more high-frequency sub-entities into the tail region, wherein the low-frequency sub-entities correspond to mentions that occur less frequently across a document corpus than the mentions corresponding to the high-frequency sub-entities.

5. The method of claim 4 , further comprising:

performing iterative global coreference resolution functions on the sub-entities assigned to the tail region, based on similarity comparison functions;

performing, in a fully distributed computing environment, shuffling functions on the sub-entities assigned to the tail region such as to improve the rate of coreference resolution;

applying a high-pass filter to generate only high-confidence resolved entities at the end of each iteration of performing the iterative global coreference functions; and

performing feature aggregation to produce representations of entities from coreferent sub-entities.

6. The method of claim 5 , further comprising:

selecting iterative random combinations of sub-entities based on similarity comparisons between sub-entities of the head region and based on the entities created from the coreferent sub-entities of the tail region.

7. The method of claim 1 , further comprising:

determining overlap of incoming data with data in an existing coreference hierarchy by assigning the high-probability coreferent chains to the high-confidence buckets;

performing computations on an overlapping region of the distribution that corresponds to the determined overlap, and restoring the computations from the previous state for the remainder of the head region and tail region; and

performing best entity guesses for the newly-computed sub-entities in the overlapping region to aid resolution, the best entity guesses corresponding to an estimation of an entity to which each of the newly-computed sub-entities corresponds.

8. The method of claim 1 , further comprising:

persisting the entity data structures;

performing random sampling algorithm functions based on predetermined similarity comparisons and intelligent shuffling between new incoming data and the persisted entities from previous runs; and

creating new entities on an as-needed basis for further runs.

9. The method of claim 1 , further comprising:

computing, using one or more similarity functions, one or more similarity metrics corresponding to at least one of contextual similarity, name similarity, title similarity, and attribute similarity simultaneously to determine coreference of a pair of sub-entities or a pair of entities.

10. The method of claim 1 , further comprising:

generating, based on the assigning of the high-probability coreferent chains to high-confidence buckets, candidates for structured entities;

determining overlap between identified high-confidence buckets of both structured data and unstructured data; and

for non-zero overlap, allotting structured entities to the tail region and, for trivial overlap, allotting structured entities in the head region such as to enhance a rate of the unstructured-structured data resolution.

11. A system, comprising:

one or more processors;

a memory device operatively coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the system to perform functions that comprise:

ingesting text data from a plurality of documents containing a plurality of mentions;

locating, from the text data, for each of a selected plurality of chains of coreferent mentions, a particular context-based name from the respective chain, wherein the coreferent mentions correspond to entities and the context-based name is a longest name in the respective chain, a last name in the respective chain, or a most frequently occurring name in the respective chain;

determining an entity category for each respective one of the plurality of chains;

determining one or more entity attributes from structured data and unstructured data;

based on the located particular context-based name, the entity category, and the one or more attributes, assigning high-probability coreferent chains to high-confidence buckets, such as to produce a power law probability distribution having a head region and a tail region; and

resolving, based at least in part on the power law probability distribution, the coreferent mentions to identify corresponding real-world entities.

12. The system of claim 11 , wherein the power law probability distribution is produced such that the respective rank of a particular chain corresponding to a context-based name is inversely proportional to the frequency of the context-based name across a document corpus.

13. The system of claim 11 , wherein assigning the high-probability chains to the high-confidence buckets comprises:

grouping the plurality of chains based on the respective context-based name and category such that chains having a same context-based name and same category are grouped together into a respective partition;

within the respective partition, grouping chains that correspond to the same concept into sub-entities; and

grouping together, across and within partitions, the sub-entities that correspond to the same concept.

14. The system of claim 11 , wherein the stored instructions, when executed by the one or more processors, further cause the system to perform functions that comprise:

assigning one or more low-frequency sub-entities into the head region; and

assigning one or more high-frequency sub-entities into the tail region, wherein the low-frequency sub-entities correspond to mentions that occur less frequently across a document corpus than the mentions corresponding to the high-frequency sub-entities.

15. The system of claim 14 , wherein the stored instructions, when executed by the one or more processors, further cause to the system to perform functions that comprise:

performing iterative global coreference resolution functions on the sub-entities assigned to the tail region, based on similarity comparison functions;

performing, in a fully distributed computing environment, shuffling functions on the sub-entities assigned to the tail region such as to improve the rate of coreference resolution;

applying a high-pass filter to generate only high-confidence resolved entities at the end of each iteration of performing the iterative global coreference functions; and

performing feature aggregation to produce representations of entities from coreferent sub-entities.

16. The system of claim 15 , wherein the stored instructions, when executed by the one or more processors, further cause to the system to perform functions that comprise:

selecting iterative random combinations of sub-entities based on similarity comparisons between sub-entities of the head region and based on the entities created from the coreferent sub-entities of the tail region.

17. The system of claim 11 , wherein the stored instructions, when executed by the one or more processors, further cause to the system to perform functions that comprise:

determining overlap of incoming data with data in an existing coreference hierarchy by assigning the high-probability coreferent chains to the high-confidence buckets;

performing computations on an overlapping region of the distribution that corresponds to the determined overlap, and restoring the computations from the previous state for the remainder of the head region and tail region; and

performing best entity guesses for the newly-computed sub-entities in the overlapping region to aid resolution, the best entity guesses corresponding to an estimation of an entity to which each of the newly-computed sub-entities corresponds.

18. The system of claim 11 , wherein the stored instructions, when executed by the one or more processors, further cause to the system to perform functions that comprise:

persisting the entity data structures;

performing random sampling algorithm functions based on predetermined similarity comparisons and intelligent shuffling between new incoming data and the persisted entities from previous runs; and

creating new entities on an as-needed basis for further runs.

19. The system of claim 11 , wherein the stored instructions, when executed by the one or more processors, further cause to the system to perform functions that comprise:

computing, using one or more similarity functions, one or more similarity metrics corresponding to at least one of contextual similarity, name similarity, title similarity, and attribute similarity simultaneously to determine coreference of a pair of sub-entities or a pair of entities.

20. The system of claim 11 , wherein the stored instructions, when executed by the one or more processors, further cause to the system to perform functions that comprise:

generating, based on the assigning of the high-probability coreferent chains to high-confidence buckets, candidates for structured entities;

determining overlap between identified high-confidence buckets of both structured data and unstructured data; and

for non-zero overlap, allotting structured entities to the tail region and, for trivial overlap, allotting structured entities in the head region such as to enhance a rate of the unstructured-structured data resolution.

21. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause a computer to perform functions that comprise:

ingesting text data from a plurality of documents containing a plurality of mentions;

locating, from the text data, for each of a selected plurality of chains of coreferent mentions, a particular context-based name from the respective chain, wherein the coreferent mentions correspond to entities and the context-based name is a longest name in the respective chain, a last name in the respective chain, or a most frequently occurring name in the respective chain;

determining an entity category for each respective one of the plurality of chains;

determining one or more entity attributes from structured data and unstructured data;

based on the located particular context-based name, the entity category, and the one or more attributes, assigning high-probability coreferent chains to high-confidence buckets, such as to produce a power law probability distribution having a head region and a tail region; and

resolving, based at least in part on the power law probability distribution, the coreferent mentions to identify corresponding real-world entities.

22. The non-transitory computer-readable medium of claim 21 , wherein the power law probability distribution is produced such that the respective rank of a particular chain corresponding to a context-based name is inversely proportional to the frequency of the context-based name across a document corpus.

23. The non-transitory computer-readable medium of claim 21 , wherein assigning the high-probability chains to the high-confidence buckets comprises:

grouping the plurality of chains based on the respective context-based name and category such that chains having a same context-based name and same category are grouped together into a respective partition;

within the respective partition, grouping chains that correspond to the same concept into sub-entities; and

grouping together, across and within partitions, the sub-entities that correspond to the same concept.

24. The non-transitory computer-readable medium of claim 21 , wherein the stored instructions, when executed by the one or more processors, further cause the computer to perform functions that comprise:

assigning one or more low-frequency sub-entities into the head region; and

assigning one or more high-frequency sub-entities into the tail region, wherein the low-frequency sub-entities correspond to mentions that occur less frequently across a document corpus than the mentions corresponding to the high-frequency sub-entities.

25. The non-transitory computer-readable medium of claim 24 , wherein the stored instructions, when executed by the one or more processors, further cause to the computer to perform functions that comprise:

performing iterative global coreference resolution functions on the sub-entities assigned to the tail region, based on similarity comparison functions;

performing, in a fully distributed computing environment, shuffling functions on the sub-entities assigned to the tail region such as to improve the rate of coreference resolution;

applying a high-pass filter to generate only high-confidence resolved entities at the end of each iteration of performing the iterative global coreference functions; and

performing feature aggregation to produce representations of entities from coreferent sub-entities.

26. The non-transitory computer-readable medium of claim 25 , wherein the stored instructions, when executed by the one or more processors, further cause to the computer to perform functions that comprise:

selecting iterative random combinations of sub-entities based on similarity comparisons between sub-entities of the head region and based on the entities created from the coreferent sub-entities of the tail region.

27. The non-transitory computer-readable medium of claim 21 , wherein the stored instructions, when executed by the one or more processors, further cause to the computer to perform functions that comprise:

determining overlap of incoming data with data in an existing coreference hierarchy by assigning the high-probability coreferent chains to the high-confidence buckets;

performing computations on an overlapping region of the distribution that corresponds to the determined overlap, and restoring the computations from the previous state for the remainder of the head region and tail region; and

performing best entity guesses for the newly-computed sub-entities in the overlapping region to aid resolution, the best entity guesses corresponding to an estimation of an entity to which each of the newly-computed sub-entities corresponds.

28. The non-transitory computer-readable medium of claim 21 , wherein the stored instructions, when executed by the one or more processors, further cause to the computer to perform functions that comprise:

persisting the entity data structures;

performing random sampling algorithm functions based on predetermined similarity comparisons and intelligent shuffling between new incoming data and the persisted entities from previous runs; and

creating new entities on an as-needed basis for further runs.

29. The non-transitory computer-readable medium of claim 21 , wherein the stored instructions, when executed by the one or more processors, further cause to the computer to perform functions that comprise:

computing, using one or more similarity functions, one or more similarity metrics corresponding to at least one of contextual similarity, name similarity, title similarity, and attribute similarity simultaneously to determine coreference of a pair of sub-entities or a pair of entities.

30. The non-transitory computer-readable medium of claim 11 , wherein the stored instructions, when executed by the one or more processors, further cause to the computer to perform functions that comprise:

generating, based on the assigning of the high-probability coreferent chains to high-confidence buckets, candidates for structured entities;

determining overlap between identified high-confidence buckets of both structured data and unstructured data; and

for non-zero overlap, allotting structured entities to the tail region and, for trivial overlap, allotting structured entities in the head region such as to enhance a rate of the unstructured-structured data resolution.