Domain agnostic similarity detection

1. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

evaluating, by a database management system, contents of each field of a plurality of fields in a table, wherein the evaluating comprises determining a scoring measure reflective of a uniqueness of content contained within each field, and wherein the scoring measure is based on a total number of distinct values contained within each field, a total number of occurrences for each of the distinct values contained within each field, and a total number of records in the database table;

assigning, by the database management system, the respective scoring measure to each field of the plurality of fields in the table;

sorting, by the database management system, each field of the plurality of fields based on the assigned scoring measures;

automatically selecting, by the database management system and based on the assigned scoring measures, two or more fields of the sorted plurality of fields by at least determining that each of the respective scoring measures assigned to the two or more fields is within a defined range of acceptable scoring measures;

receiving, at the database management system, an indication of a first level of fuzziness for performing a fuzzy search;

executing, by the database management system, a first fuzzy search based on the selected two or more fields and the first level of fuzziness to obtain a first search result, wherein the execution of the first fuzzy search comprises concatenating a plurality of the selected two or more fields;

automatically generating, by the database management system based on a number of records in the first search result, a second level of fuzziness by at least

in response to determining that the number of records in the first search result is less than a lower threshold, assigning, to the second level of fuzziness, a value that is greater than the first level of fuzziness, and

in response to determining that the number of records in the first search result is greater than an upper threshold, assigning, to the second level of fuzziness, a value that is less than the first level of fuzziness; and

executing, by the database management system, a second fuzzy search based on the selected two or more fields and the second level of fuzziness, wherein the execution of the second fuzzy search comprises concatenating the plurality of the selected two or more fields.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the first level of fuzziness and the second level of fuzziness are based on a sliding scale indicating levels of fuzziness, and wherein the operations further comprise:

receiving, by the database management system, an identifier of a table comprising a plurality of records having a plurality of fields.

3. The non-transitory computer-readable storage medium of claim 1 , wherein the assigned scoring measures are calculated based on:

S

j

=

-

∑

i

=

1

N

⁢

⁢

p

i

⁢

log

⁢

⁢

p

i

.

4. The non-transitory computer-readable storage medium of claim 1 , wherein the operations further comprise:

outputting, by the database management system, a second search result of the second fuzzy search, wherein the second search result comprises at least two records including at least one field having similar content.

5. The non-transitory computer-readable storage medium of claim 1 , wherein to automatically select the two or more fields, the operations further comprise:

identifying, by the database management system, at least one first field having a higher scoring measure compared to at least one second other field; and

selecting, by the database management system, one or more of the at least one first field having the higher scoring measure.

6. A method for implementation by one or more data processors forming part of at least one computing system, the method comprising:

evaluating, by a database management system, contents of each field of a plurality of fields in a table, wherein the evaluating comprises determining a scoring measure reflective of a uniqueness of content contained within each field, and wherein the scoring measure is based on a total number of distinct values contained within each field, a total number of occurrences for each of the distinct values contained within each field, and a total number of records in the database table;

assigning, by the database management system, a respective scoring measure to each field of the plurality of fields in the table;

sorting, by the database management system, each field of the plurality of fields based on the assigned scoring measures;

automatically selecting, by the database management system and based on the assigned scoring measures, two or more fields of the sorted plurality of fields by at least determining that each of the respective scoring measures assigned to the two or more fields is within a defined range of acceptable scoring measures;

receiving, at the database management system, an indication of a first level of fuzziness for performing a fuzzy search;

executing, by the database management system, a first fuzzy search based on the selected two or more fields and the first level of fuzziness to obtain a first search result, wherein the execution of the first fuzzy search comprises concatenating a plurality of the selected two or more fields;

automatically generating, by the database management system based on a number of records in the first search result, a second level of fuzziness by at least

in response to determining that the number of records in the first search result is less than a lower threshold, assigning, to the second level of fuzziness, a value that is greater than the first level of fuzziness, and

in response to determining that the number of records in the first search result is greater than an upper threshold, assigning, to the second level of fuzziness, a value that is less than the first level of fuzziness; and

executing, by the database management system, a second fuzzy search based on the selected two or more fields and the second level of fuzziness, wherein the execution of the second fuzzy search comprises concatenating the plurality of the selected two or more fields.

7. The method of claim 6 , wherein the first level of fuzziness and the second level of fuzziness are based on a sliding scale indicating levels of fuzziness, and wherein the method further comprises receiving, by the database management system, an identifier of a table comprising a plurality of records having a plurality of fields.

8. The method of claim 6 , wherein the assigned scoring measures are calculated based on:

S

j

=

-

∑

i

=

1

N

⁢

⁢

p

i

⁢

log

⁢

⁢

p

i

.

9. The method of claim 6 , further comprising:

outputting, by the database management system, a second search result of the second fuzzy search, wherein the second search result comprises at least two records including at least one field having similar content.

10. The method of claim 6 , wherein to automatically select the two or more fields, the method further comprises:

identifying, by the database management system, at least one first field having a higher scoring measure compared to at least one second other field; and

selecting, by the database management system, one or more of the at least one first field having the higher scoring measure.

11. A system comprising:

computer hardware comprising at least one programmable processor configured to perform operations comprising:

evaluating, by a database management system, contents of each field of a plurality of fields in a table, wherein the evaluating comprises determining a scoring measure reflective of a uniqueness of content contained within each field, wherein the scoring measure is based on a total number of distinct values contained within each field, a total number of occurrences for each of the distinct values contained within each field, and a total number of records in the database table;

assigning, by the database management system, the respective scoring measure to each field of the plurality of fields in the table;

sorting, by the database management system, each field of the plurality of fields based on the assigned scoring measures;

automatically selecting, by the database management system and based on the assigned scoring measures, two or more fields of the sorted plurality of fields by at least determining that each of the respective scoring measures assigned to the two or more fields is within a defined range of acceptable scoring measures;

receiving, at the database management system, an indication of a first level of fuzziness for performing a first fuzzy search;

executing, by the database management system, the first fuzzy search based on the selected two or more fields and the first level of fuzziness to obtain a first search result, wherein the execution of the first fuzzy search comprises concatenating a plurality of the selected two or more fields;

automatically generating, by the database management system based on a number of records in the first search result, a second level of fuzziness by at least

in response to determining that the number of records in the first search result is less than a lower threshold, assigning, to the second level of fuzziness, a value that is greater than the first level of fuzziness, and

in response to determining that the number of records in the first search result is greater than an upper threshold, assigning, to the second level of fuzziness, a value that is less than the first level of fuzziness; and

executing, by the database management system, a second fuzzy search based on the selected two or more fields and the second level of fuzziness, wherein the execution of the second fuzzy search comprises concatenating the plurality of the selected two or more fields.

12. The system of claim 11 , wherein the first level of fuzziness and the second level of fuzziness are based on a sliding scale indicating levels of fuzziness, and wherein the operations further comprise:

receiving, by the database management system, an identifier of a table comprising a plurality of records having a plurality of fields.

13. The system of claim 11 , wherein the assigned scoring measures are calculated based on:

S

j

=

-

∑

i

=

1

N

⁢

p

i

⁢

log

⁢

⁢

p

i

.

14. The system of claim 11 , wherein the operations further comprise:

outputting, by the database management system, a second search result of the second fuzzy search, wherein the second search result comprises at least two records including at least one field having similar content.

15. The system of claim 11 , wherein to select the two or more fields, the operations further comprise:

identifying, by the database management system, at least one first field having a higher scoring measure compared to at least one second other field; and

selecting, by the database management system, one or more of the at least one first field having the higher scoring measure.