Techniques for database entries de-duplication

1. A method for data entries deduplication, comprising:

indexing an input data set, wherein the input data set is in a tabular format and the indexing includes providing a unique Row identifier (RowID), wherein rows are the data entries;

standardizing the input data set into a predefined and unified format;

segmenting the standardized input data set, wherein each segment includes a subset of the rows included in the input data set;

indexing each segment using a text search engine;

computing attribute similarity for each column across each pair of rows;

computing, for each pair of rows, row-to-row similarity as a weighted sum of attribute similarities;

clustering pairs of rows based on their row-to-row similarities;

determine clusters that are substantially related based on a cluster signature;

iteratively merging clusters that are substantially related;

providing an output data set including at least the clustered pairs of rows; and

wherein the output data set further includes the input data set, a cluster identification indicating the deduplicated group to which a corresponding row belongs, a cluster anchor information including the RowID, and a confidence score indicating a confidence or likelihood that the row belongs to the cluster.

2. The method of claim 1 , wherein computing attribute similarity further comprises:

utilizing a comparator based on a type of an attribute to compute the attribute similarity, wherein the comparator is any one of: exact matching and fuzzy matching.

3. The method of claim 2 , wherein row-to-row similarity demonstrates pairs of rows are similar, and wherein the weights are determined based on a machine learning model.

4. The method of claim 1 , wherein clustering the pairs of rows further comprises:

generating a graph including nodes and edges, wherein the nodes represent rows and edges represent the row-to-row similarities; and

applying a greedy algorithm on the graph to cluster rows, wherein each cluster includes at two similar data entries.

5. The method of claim 4 , wherein the clustering results in isolated rows, wherein each isolated row is individually clustered.

6. The method of claim 4 , wherein the format of the output data set is any one of: a table and a graph.

7. The method of claim 6 , wherein the input data set is sourced from a plurality of data sources.

8. The method of claim 4 , wherein the cluster signature is a combination of a cluster anchor, a cluster ID, and a neighbor vector, where the cluster anchor is the row that has the most connections with other rows in a particular cluster, and where the neighbor vector includes rows that are part of the particular cluster in addition to rows that are “N” hops away, “N” being a non-negative integer, in the generated graph, from rows within the particular cluster.

9. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to perform a process method for data entries deduplication, the process comprising:

indexing an input data set, wherein the input data set is in a tabular format and the indexing includes providing a unique row identifier (ID);

standardizing the input data set into a predefined and unified format;

segmenting the standardized input data set, wherein each segment includes a subset of the rows included in the input data set;

indexing each segment using a text search engine;

computing attribute similarity for each column across each pair of rows;

computing, for each pair of rows, row-to-row similarity as a weighted sum of attribute similarities;

clustering pairs of rows based on their row-to-row similarities;

determine clusters that are substantially related based on a cluster signature;

iteratively merging clusters that are substantially related;

providing an output data set including at least the clustered data entries and clustered data entries; and

wherein the output data set further includes the input data set, a cluster identification indicating the deduplicated group to which a corresponding row belongs, a cluster anchor information including the RowID, and a confidence score indicating a confidence or likelihood that the row belongs to the cluster.

10. The non-transitory computer readable medium according to claim 9 ,

wherein the process further comprises:

generating a graph including nodes and edges, wherein the nodes represent rows and edges represent the row-to-row similarities; and

applying a greedy algorithm on the graph to cluster rows, wherein each cluster includes at two similar data entries,

wherein the cluster signature is a combination of a cluster anchor, a cluster ID, and a neighbor vector, where the cluster anchor is the row that has the most connections with other rows in a particular cluster, and where the neighbor vector includes rows that are part of the particular cluster in addition to rows that are “N” hops away, “N” being a non-negative integer, in the generated graph, from rows within the particular cluster.

11. A system for data entries deduplication, comprising:

a processing circuitry; and

a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:

index an input data set, wherein the input data set is in a tabular formant and the indexing includes providing a unique row identifier (ID);

standardize the input data set into a predefined and unified format;

segment the standardized input data set, wherein each segment includes a subset of the rows included in the input data set;

index each segment using a text search engine;

compute attribute similarity for each column across each pair of rows;

compute, for each pair of rows, row-to-row similarity as a weighted sum of attribute similarities;

cluster pairs of rows based on their row-to-row similarities;

determine clusters that are substantially related based on a cluster signature;

iteratively merge clusters that are substantially related;

provide an output data set including at least the clustered data entries and clustered data entries; and

wherein the output data set further includes the input data set, a cluster identification indicating the deduplicated group to which a corresponding row belongs, a cluster anchor information including the RowID, and a confidence score indicating a confidence or likelihood that the row belongs to the cluster.

12. The system of claim 10 , wherein the system is further configured to:

utilize a comparator based on a type of an attribute to compute the attribute similarity, wherein the comparator is any one of: exact matching and fuzzy matching.

13. The system of claim 12 , wherein row-to-row similarity demonstrates pairs of rows are similar, and wherein the weights are determined based on a machine learning model.

14. The system of claim 11 , wherein the system is further configured to:

generate a graph including nodes and edges, wherein the nodes represent rows and edges represent the row-to-row similarities; and

apply a greedy algorithm on the graph to cluster rows, wherein each cluster includes at two similar data entries.

15. The system of claim 14 , wherein the format of the output data set is any one of: a table and a graph.

16. The system of claim 15 , wherein the input data set is sourced from a plurality of data sources.

17. The system of claim 14 , wherein the cluster signature is a combination of a cluster anchor, a cluster ID, and a neighbor vector, where the cluster anchor is the row that has the most connections with other rows in a particular cluster, and where the neighbor vector includes rows that are part of the particular cluster in addition to rows that are “N” hops away, “N” being a non-negative integer, in the generated graph, from rows within the particular cluster.