Clustering of data records with hierarchical cluster IDs
The present disclosure relates clustering similar data records together in a hierarchical clustering scheme. Each tier in a cluster corresponds to a minimal match score, which reflects a degree of confidence. A hierarchical cluster ID is generated for respective data records. The hierarchical cluster ID may be made up of a series of values, wherein each value reflects a tier within the hierarchical clustering scheme. A user may enter a partial hierarchical cluster ID to select clusters associated with a lower confidence. Thus, in some embodiments, the hierarchical cluster ID is variable in length in a manner that corresponds to the tiers in the hierarchical clustering scheme.
1. A system comprising:
a database that stores at least one database table comprising a plurality of records; and
a memory coupled to a processor; the memory comprising a plurality of instructions that cause the processor to:
access a subset of records derived from at least one database table;
generate a hierarchical clustering for the subset of records by iteratively performing ordinal classifications on a plurality of pairwise comparisons over a plurality of iterations, the hierarchical clustering comprising a plurality of tiers, wherein each tier corresponds to a respective degree of confidence among the records within the tier, wherein the ordinal classifications are performed by an ordinal classifier that generates a non-binary output based on training data comprising ordinal, non-binary labels;
assign a respective hierarchical cluster ID to each record in the subset of records according to the hierarchical clustering, the hierarchical cluster ID comprising a series of values, wherein each value reflects a respective tier among the plurality of tiers; and
receive a hierarchical cluster ID from a client device to allow a selection of a cluster corresponding to one tier among the plurality of tiers of the hierarchical clustering according to the hierarchical cluster ID.
2. The system of claim 1 , wherein the hierarchical cluster ID received from the client device is a partial hierarchical cluster ID that specifies a lower confidence tier.
3. The system of claim 1 , wherein the hierarchical cluster ID received from the client device is a complete hierarchical cluster ID that specifies a higher confidence tier.
4. The system of claim 1 , wherein the hierarchical cluster ID is variable in length such that the length corresponds to a respective tier.
5. The system of claim 1 , wherein the plurality of instructions further cause the processor to generate a processed database table that includes a field for the respective hierarchical cluster IDs, wherein the processed database table is generated from de-duplicating the at least one database table.
6. The system of claim 1 , wherein the ordinal classifier is trained using labels that indicate at least one of a Strong-Match, a Weak-Match, an Unknown, and a Hard-Conflict.
7. The system of claim 1 , wherein at least one of the pairwise comparisons comprises a comparison of a first input and a second input, the first input being a single record and the second input being a plurality of merged records.
8. A computer implemented method comprising:
accessing a subset of records derived from at least one database table;
generating a hierarchical clustering for the subset of records by iteratively performing ordinal classifications on a plurality of pairwise comparisons over a plurality of iterations, wherein the ordinal classifications are performed by an ordinal classifier that generates a non-binary output based on training data comprising ordinal, non-binary labels, the hierarchical clustering comprising a plurality of tiers, wherein each tier corresponds to a respective degree of confidence among the records within the tier;
assigning a respective hierarchical cluster ID to each record in the connected component according to the hierarchical clustering, the hierarchical cluster ID comprising a series of values, wherein each value reflects a respective tier among the plurality of tiers; and
receiving a hierarchical cluster ID from a client device to allow a selection of a cluster corresponding to one tier among the plurality of tiers of the hierarchical clustering according to the hierarchical cluster ID.
9. The method of claim 8 , wherein the hierarchical cluster ID received from the client device is a partial hierarchical cluster ID that specifies a lower confidence tier.
10. The method of claim 9 , wherein the hierarchical cluster ID received from the client device is a complete hierarchical cluster ID that specifies a higher confidence tier.
11. The method of claim 8 , wherein the hierarchical cluster ID is variable in length such that the length corresponds to a respective tier.
12. The method of claim 8 , further comprising:
generating a processed database table that includes a field for the respective hierarchical cluster IDs, wherein the processed database table is generated from de-duplicating the at least one database table.
13. The method of claim 8 , wherein the ordinal classifier is trained using labels that indicate at least one of a Strong-Match, a Weak-Match, a No-Conflict, and a Hard-Conflict.
14. The method of claim 8 , wherein at least one of the pairwise comparisons comprises a comparison of a first input and a second input, the first input being a single record and the second input being a plurality of merged records.
15. One or more non-transitory computer-readable, nor volatile storage memory comprising stored instructions that are executable and, responsive to execution by a computing device, the computing device performs operations comprising:
accessing a subset of records derived from at least one database table;
generating a hierarchical clustering for the subset of records by iteratively performing ordinal classifications on a plurality of pairwise comparisons over a plurality of iterations, wherein the ordinal classifications are performed by an ordinal classifier that generates a non-binary output based on training data comprising ordinal, non-binary labels, the hierarchical clustering comprising a plurality of tiers, wherein each tier corresponds to a respective degree of confidence among the records within the tier;
assigning a respective hierarchical cluster ID to each record in the subset of records according to the hierarchical clustering, the hierarchical cluster ID comprising a series of values, wherein each value reflects a respective tier among the plurality of tiers; and
receiving a hierarchical cluster ID from a client device to allow a selection of a cluster corresponding one tie among a plurality of tiers of the hierarchical clustering according to the hierarchical cluster ID.
16. The one or more non-transitory computer-readable, non-volatile storage memory of claim 15 , wherein each ordinal, non-binary label corresponds to a respective tier.
17. The one or more non-transitory computer-readable, non-volatile storage memory of claim 15 , wherein the ordinal classifier is trained using labels that indicate at least one of No-Conflict and a Hard-Conflict.
18. The one or more non-transitory computer-readable, non-volatile storage memory of claim 15 , wherein the hierarchical cluster ID is variable in length such that the length corresponds to a respective tier.
19. The one or more non-transitory computer-readable, non-volatile storage memory of claim 16 , wherein, in response to execution by the computing device, the computing device performs an operation of:
generating a processed database table that includes a field for the respective hierarchical cluster IDs, wherein the processed database table is generated from de-duplicating the at least one database table.
20. The one or more non-transitory computer-readable, non-volatile storage memory of claim 15 , wherein at least one of the pairwise comparisons comprises a comparison of a first input and a second input, the first input being a single record and the second input being a plurality of merged records.