MACHINE LEARNING ENHANCED CLASSIFIER
The presently disclosed subject matter includes a computerized method and system that provide the ability to train and execute a unique machine learning (ML) model specifically configured to enhance classifier (e.g., RegEx) output by identifying and removing false positive results from the classifiers output. Classifier output, comprising a collection of data-subsets (e.g., columns in a relational database) of one or more structured or semi-structured data sources (e.g., tables of a relational database), are transformed to be represented by a plurality of numerical vectors. The numerical vectors are used during a training phase (as well as the execution phase) for training a machine learning model to enhance the classifier output and reduce false positives.
1 . A non-transitory program storage device readable by a computer, tangibly embodying a program of instructions executable by the computer to perform a method of identifying matching data-subsets from a plurality of data-subsets of one or more structured or semi-structured data sources, each data-subset comprising data-values; the method comprising:
before machine learning (ML) model execution:
obtaining data indicative of one or more matching data-subsets, each matching data-subset includes matching data-values determined by applying on the data-subset a classifier dedicated for identifying data-values that match a certain pattern; and
for each matching data-subset of the one or more matching data-subsets
generating at least two respective features vectors, comprising:
dividing the matching data-subset into a first subgroup comprising matching data-values identified as matching by the classifier and a second subgroup comprising non-matching data-values identified as non-matching by the classifier; and
generating a first features vector from the data-values in the first subgroup and a second features vector from the data-values in the second subgroup; wherein each feature in the first features vector is indicative of an aggregated frequency of a respective n-gram-position pair in the first subgroup, and each feature in the second features vector is indicative of an aggregated frequency of a respective n-gram-position pair in the second subgroup; wherein each n-gram-position pair indicates the position of a respective n-gram in a data value in the first subgroup or the second subgroup;
during ML model execution:
providing the at least two respective features vectors, of the one or more matching data-subsets, as input to the ML model; and
using the ML model for processing the at least two respective features vectors and for determining, based on both the first features vector generated from the matching data-values and the second features vector generated from the non-matching data-values, if exist at least one data-subset from the one or more matching data-subsets, which represents a false positive matching data-subset, thereby enhancing the classifier output.
2 . (canceled)
3 . The non-transitory program storage device of claim 1 , wherein the n-gram-position pairs include unigrams-position pairs and/or bigram-position pairs.
4 . The non-transitory program storage device of claim 1 , wherein the generation of the first features vector and the second features vector comprises:
dividing each data-value in the first subgroup and the second subgroup into n-grams;
assigning each n-grams with a respective value indicating its position within a respective data-value string, giving rise to a respective n-grams-position pair; and
for each one of the first subgroup and second subgroup, aggregating frequency of occurrence of each n-grams position pair in a respective vector, giving rise to a first features vector and second features vector.
5 . The non-transitory program storage device of claim 1 , wherein the method further comprising calculating a hash value for each feature in the first features vector and second features vector.
6 . The non-transitory program storage device of claim 1 , wherein the method further comprising generating, for each matching data-subset, one or more additional features vectors using respective data related to the data-subset and/or data-source comprising the data-subset, wherein the one or more additional features vectors are also used as input to the ML model.
7 . The non-transitory program storage device of claim 6 , wherein the one or more additional features vectors include at least one word vector representing the respective data.
8 . The non-transitory program storage device of claim 1 , wherein the method further comprising:
generating two additional name-features-vectors to be used as input to the ML model; wherein a first name-features-vector comprises a collection of features, each feature being indicative of a respective n-gram in a name of the data-subset and a second name-features-vector comprises a collection of features, each feature is a respective n-gram in a processed version of the name of the data-subset.
9 . The non-transitory program storage device of claim 1 , wherein the method further comprising applying the classifier on the one or more data-subsets, thereby obtaining the classifier data output.
10 . The non-transitory program storage device of claim 1 , wherein the method further comprising implementing a ML training phase, comprising:
applying the classifier on a plurality of data-subsets to thereby classify each data-value in each data-subset as matching or non-matching the classifier;
generating a sample dataset comprising multiple matching data-subsets and multiple non-matching data-subsets, wherein classification of a data-subset as matching or non-matching is according to a relative portion of matching and non-matching data-values in each respective data-subset;
for each data-subset in the sample dataset:
dividing the data-subset into a first subgroup comprising matching data-values, and a second subgroup comprising non-matching data-values;
generating a first features vector from the data-values in the first subgroup, and a second features vector from the data-values in the second subgroup;
receiving user input indicative of whether classification of each data-subset in the sample dataset to matching or non-matching, is true or false, thereby obtaining an annotated sample dataset;
using the first features vector and the second features vectors of each one of the data-subsets in the sample dataset as a training set for training the ML model for identifying false detections made by the classifier.
11 . The non-transitory program storage device of claim 10 , wherein the first features vector comprises a first collection of features, each feature being indicative of an aggregated frequency of a respective n-gram-position pair in the first subgroup, and the second features vector comprises a second collection of features, each feature being indicative of an aggregated frequency of a respective n-gram-position pair in the second subgroup; wherein each n-gram-position pair indicates the position of a respective n-gram in a data value in the first subgroup or the second subgroup.
12 . The non-transitory program storage device of claim 11 , wherein generating the first features vector and the second features vectors further comprises calculating, for each feature, a respective hash value.
13 . The non-transitory program storage device of claim 11 , wherein n-gram-position pairs in the first collection and the second collection includes unigrams-position pairs and/or bigram-position pairs.
14 . The non-transitory program storage device of claim 1 , wherein the classifier is a regular expression (RegEx).
15 . The non-transitory program storage device of claim 10 , wherein the method further comprising:
generating for each data-subset in the sample dataset, one or more additional features vectors using respective data related to the data-subset and/or data-source, wherein the one or more additional features vectors are also used as input to the ML model.
16 . The non-transitory program storage device of claim 11 , wherein the method further comprising:
generating for each data-subset in the sample dataset, two additional name-features-vector to be used as input to the ML model; wherein a first name-features-vector comprises a collection of features, each feature is a respective n-gram in a name of the data-subset and a second name-features-vector comprises a collection of features, each feature is a respective n-gram in a processed version of the name of the data-subset.
17 . A computerized system of identifying matching data-subsets from a plurality of data-subsets of one or more structured or semi-structured data sources, each data-subset comprising data-values; the system comprises a processing circuitry configured to:
before machine learning (ML) model execution:
obtain data indicative of one or more matching data-subsets, each data-subset includes matching data-values and is determined by applying on the data-subset a classifier dedicated for identifying data-values that match a certain pattern; and
for each matching data-subset of the one or more data-subsets
generate at least two respective features vectors, comprising:
divide the matching data-subset into a first subgroup comprising matching data-values identified as matching by the classifier and a second subgroup comprising non-matching data-values identified as non-matching by the classifier; and
generate at least two features vectors including a first features vector from the data-values in the first subgroup and a second features vector from the data-values in the second subgroup; wherein the first features vector comprises a first collection of features, each feature being indicative of an aggregated frequency of a respective n-gram-position pair in the first subgroup, and the second features vector comprises a second collection of features, each feature being indicative of an aggregated frequency of a respective n-gram-position pair in the second subgroup; wherein each n-gram-position pair indicates the position of a respective n-gram in a data value in the first subgroup or the second subgroup;
during ML model execution
provide the at least two features vectors of the one or more matching data-subsets, as input to the ML model; and
use the ML model for processing the at least two respective features vectors and for determining, based on both the a first features vector generated from the matching data-values and the second features vector generated from the non-matching data-values, if exists at least one data-subset from the one or more matching data-subsets, which represents a false positive matching data-subsets, thereby enhancing classifier output.
18 . (canceled)
19 . The computerized system of claim 17 , wherein n-gram-position pairs in the first collection and the second collection include unigrams-position pairs and/or bigram-position pairs.
20 . The computerized system of claim 17 , wherein the processing circuitry is configured for generation of the first features vector and the second features vector, to:
divide each data-value in the first subgroup and the second subgroup into n-grams;
assign each n-grams with a respective value indicating its position within a respective data-value string, giving rise to a respective n-grams-position pair; and
for each one of the first subgroup and second subgroup, aggregate frequency of occurrence of each n-grams position pair in a respective vector, giving rise to a first features vector and second features vector.
21 . The computerized system of claim 17 , wherein the processing circuitry is further configured to calculate a hash value for each feature in the first features vector and second features vector.
22 . The computerized system of claim 17 , wherein the classifier is a regular expression (RegEx).
23 . The computerized system of claim 17 , wherein the processing circuitry is further configured to generate one or more additional features vectors using respective data related to the data-subset and/or data-source, wherein the one or more additional features vectors are also used as input to the ML model.
24 . The computerized system of claim 17 , wherein the processing circuitry is further configured to:
generate two additional name-features-vectors to be used as input to the ML model, wherein a first name-features-vector comprises a collection of features, each feature is indicative of a respective n-gram in a name of the data-subset and a second name-features-vector comprises a collection of features, each feature is indicative of a respective n-gram in a processed version of the name of the data-subset.
25 . The computerized system of claim 17 , wherein the processing circuitry is further configured to apply the classifier on the one or more data-subsets to thereby obtain the classifier data output.
26 . A non-transitory program storage device readable by a computer, tangibly embodying a program of instructions executable by the computer to perform a method of training a machine learning (ML) model dedicated for detecting false detections in classifier output applied on data-subsets of one or more structured or semi-structured data sources, each data-subset comprising data-values; the method comprising using a processing circuitry for
generating a sample dataset comprising multiple matching data-subsets and multiple non-matching data-subsets, wherein classification of a data-subset as matching or non-matching is according to a relative portion of matching and non-matching data-values in each respective data-subset; wherein data-values in each data-subset are classified as matching or non-matching by applying the classifier on the data-values;
for each data-subset in the sample dataset:
generating at least two respective features vectors, comprising:
dividing the data-subset into a first subgroup comprising the matching data-values, identified as matching by the classifier, and a second subgroup comprising the non-matching data-values identified as non-matching by the classifier;
generating a first features vector from the data-values in the first subgroup and a second features vector from the data-values in the second subgroup; wherein the first features vector comprises a first collection of features, each feature being indicative of an aggregated frequency of a respective n-gram-position pair in the first subgroup, and the second features vector comprises a second collection of features, each feature being indicative of an aggregated frequency of a respective n-gram-position pair in the second subgroup; wherein each n-gram-position pair indicates the position of a respective n-gram in a data value in the first subgroup or the second subgroup;
receiving user input indicative of whether classification of each data-subset in the sample dataset to matching or non-matching, is true or false, thereby obtaining an annotated sample dataset;
using both the first features vector, generated based on the matching data-values, and the second features vector, generated based on the non-matching data-values of each one of the data-subsets in the sample dataset as a training set for training the ML model for identifying false positive detections made based on the classifier.
27 . (canceled)
28 . The non-transitory program storage device of claim 26 , wherein n-gram-position pairs in the first collection and the second collection includes unigrams-position pairs and/or bigram-position pairs.
29 . The non-transitory program storage device of claim 26 , wherein the method further comprising:
generating for each data-subset in the sample dataset, one or more additional features vectors using respective data related to the data-subset and/or data-source, wherein the one or more additional features vectors are also used as input to the ML model.
30 . The non-transitory program storage device of claim 26 , wherein the method further comprising:
generating for each data-subset in the sample dataset, two name-features-vectors that are used as input for the ML model, wherein a first name-features-vector comprises a collection of features, each feature being indicative of a respective n-gram in a name of the data-subset, and a second name-features-vector comprises a collection of features, each feature being indicative of a respective n-gram in a processed version of the name of the data-subset.
31 . A method of identifying matching data-subsets from a plurality of data-subsets of one or more structured or semi-structured data sources, each data-subset comprising data-values; the method comprising:
before machine learning (ML) model execution:
obtaining data indicative of one or more matching data-subsets, each matching data-subset includes matching data-values and is determined by applying on the data-subset a classifier dedicated for identifying data-values that match a certain pattern; and
for each matching data-subset of the one or more matching data-subsets
generating at least two respective features vectors, comprising:
dividing the matching data-subset into a first subgroup comprising matching data-values identified as matching by the classifier and a second subgroup comprising non-matching data-values identified as non-matching by the classifier; and
generating a first features vector from the data-values in the first subgroup and a second features vector from the data-values in the second subgroup; wherein each feature in the first features vector is indicative of an aggregated frequency of a respective n-gram-position pair in the first subgroup, and each feature in the second features vector is indicative of an aggregated frequency of a respective n-gram-position pair in the second subgroup; wherein each n-gram-position pair indicates the position of a respective n-gram in a data value in the first subgroup or the second subgroup;
during ML model execution:
providing the at least two respective features vectors, of the one or more matching data-subsets, as input to the ML model; and
using the ML model for processing the at least two respective features vectors and for determining, based on both the first features vector generated from the matching data-values and the second features vector generated from the non-matching data-values, if exist at least one data-subset from the one or more matching data-subsets, which represents a false positive matching data-subset, thereby enhancing the classifier output.
32 . The method of claim 31 , further comprising:
generating two additional name-features-vectors to be used as input to the ML model; wherein a first name-features-vector comprises a collection of features, each feature being indicative of a respective n-gram in a name of the data-subset and a second name-features-vector comprises a collection of features, each feature being indicative of a respective n-gram in a processed version of the name of the data-subset.
33 . The non-transitory program storage device of claim 1 , wherein the data-subsets are columns, and the structures or semi structured data resource is a relational database.