Machine learning engine for determining data similarity
System and methods for training and using a machine-learning similarity framework are provided. During training, the similarity framework generates an ensemble of tress. The trees have different properties at each node. The similarity framework uses the ensemble of trees to determine similarity between objects. The objects are propagated through nodes of each tree in the ensemble of trees until the objects reach leaf nodes. The objects are propagated by comparing the properties at each node of the tree to the features of the objects until the objects reach the leaf nodes. The similarity framework determines a similarity score for every two objects in each tree and adjusts the similarity score by tree importance. The object similarity score is determined by combining the similarity scores from the trees in the ensemble of trees. The similarity framework generates a similarity matrix with the object similarity scores of the every two objects.
1 . A method for determining similarity between objects, the method comprising:
generating, using a processor and a machine learning similarity framework, an ensemble of binary trees using training features of a training dataset, wherein the generating the ensemble of binary trees further comprises:
randomly generating a plurality of hyperparameters;
generating a plurality of trees using the plurality of hyperparameters and training features;
determining a score for each tree in the plurality of trees using validation features in a validation dataset;
determining a score for each hyperparameter from the plurality of hyperparameters based on scores for a subset of trees from the plurality of trees associated with the each hyperparameter;
selecting a hyperparameter from the plurality of hyperparameters based on scores for the plurality of hyperparameters; and
selecting the subset of trees corresponding to the selected hyperparameter into the ensemble of binary trees;
for each tree in the ensemble of binary trees:
propagating the objects through the each tree from a top node and until the objects reach leaf nodes of the each tree, wherein a propagating path through the each tree is based on comparing at least one feature associated with each object in the objects to at least one property of at least one node associated with the each tree; and
determining a similarity score for each pair of objects from the objects in the leaf nodes of the each tree, wherein determining the similarity score further comprises:
determining a deepest common node between a first object and a second object in the each pair of objects in the each tree; and
determining a depth of the each tree, wherein determining the similarity score is based on the deepest common node and the depth of the each tree; and
for the each pair of objects, combining the similarity score from the each tree in the ensemble of binary trees into an object similarity score, wherein the object similarity score is indicative of a similarity between the each pair of objects; and
identifying, using object similarity scores that are below a similarity threshold, an object that is different from other objects, wherein the object that is different from other objects includes erroneous data in one or more features.
2 . The method of claim 1 , further comprising:
generating a similarity matrix, the similarity matrix including the object similarity scores for pairs of objects propagated through the ensemble of trees, wherein each row and each column in the similarity matrix represents the objects and entries representing the object similarity scores corresponding to the each of the pairs of objects.
3 . The method of claim 1 , wherein determining the similarity score for the each pair of objects comprises:
assigning the similarity score of one when the first object and the second object in the each pair of objects are associated with the same leaf node in the leaf nodes of the each tree; or
assigning the similarity score of less than one when the first object and the second object in the each pair of objects are associated with different leaf nodes in the leaf nodes of the each tree.
4 . The method of claim 1 , wherein the generating the ensemble of binary trees comprises:
determining a tree importance for the each tree in the ensemble of binary trees; and
wherein determining the similarity score for the each of the pair of objects is based on the tree importance for the each tree.
5 . The method of claim 4 , further comprising:
determining the tree importance of the each tree in the ensemble of trees by calculating an error of the ensemble of trees before and after adding the each tree to the ensemble of trees.
6 . The method of claim 1 , wherein the generating the ensemble of trees uses a loss function and a gradient of the loss function.
7 . A system for determining similarity between objects, the system comprising:
at least one memory storing a similarity framework with an ensemble of binary trees; and
at least one processor coupled to the memory and configured to read instructions associated with the similarity framework to cause the system to perform operations, the operations comprising:
generating, using a processor and a machine learning similarity framework, an ensemble of binary trees using training features of a training dataset, wherein the generating the ensemble of binary trees further comprises:
generating a plurality of trees using a plurality of hyperparameters and training features;
determining a score for each tree in the plurality of trees using validation features in a validation dataset;
determining a score for each hyperparameter from the plurality of hyperparameters based on scores for a subset of trees from the plurality of trees associated with the each hyperparameter;
selecting a hyperparameter from the plurality of hyperparameters based on scores for the plurality of hyperparameters; and
selecting the subset of trees corresponding to the selected hyperparameter into the ensemble of binary trees;
for each tree in the ensemble of binary trees:
propagating the objects through the each tree until the objects reach leaf nodes of the each tree, wherein the propagating compares at least one feature associated with each object in the objects to at least one property of at least one node associated with the each tree; and
determining a similarity score for each of two objects from the objects in the leaf nodes, wherein determining the similarity score further comprises:
determining a deepest common node between a first object and a second object in the two objects in the each tree; and
determining a depth of the each tree, wherein determining the similarity score is based on the deepest common node and the depth of the each tree; and
for each of the two objects, combining similarity scores from the each tree in the ensemble of trees into an object similarity score; and
identifying, using object similarity scores and a similarity threshold, a pair of similar objects.
8 . The system of claim 7 , wherein the operations further comprise:
generating a similarity matrix, the similarity matrix including object similarity scores for the objects, the similarity matrix including rows and columns, wherein each row and each column represents the objects and entries representing the object similarity scores corresponding to the each of the two objects.
9 . The system of claim 7 , wherein the operations further comprise:
assigning the similarity score of one when the each of the two objects are associated with the same leaf node in the leaf nodes of the each tree; or
assigning the similarity score of less than one when the each of the two objects are associated with different leaf nodes in the leaf nodes of the each tree.
10 . The system of claim 7 , wherein the operations further comprise:
determining a tree importance for the each tree in the ensemble of trees; and
wherein the similarity score for the each of the two objects is based on the tree importance for the each tree.
11 . The system of claim 10 , wherein the operations further comprise:
determining the tree importance of the each tree in the ensemble of binary trees by calculating an error of the ensemble of binary trees before and after adding the each tree to the ensemble of trees.
12 . The system of claim 7 , wherein the similarity score identifies similarity in the at least one feature of the two objects.
13 . The system of claim 12 , wherein the first object in the each of the two objects is a first transaction and the second object in the each of the two objects is a second transaction, and the similarity score identifies the first transaction as a fraudulent transaction.
14 . The system of claim 7 , wherein the first object in the each of the two objects is an object from a training dataset and the second object in the each of the two objects is a new object, and the similarity score identifies whether the new object is an outlier to the object in the training dataset.
15 . The system of claim 7 , wherein the similarity framework further comprises an encoder; and the operations further comprise:
encoding the at least one feature associated with each object into an encoded feature; and
comparing the at least one encoded feature to the at least one property of the at least one node associated with the each tree.
16 . A non-transitory computer readable medium having instructions stored thereon, that when executed by a processor, cause the processor to perform operations, the operations comprising:
generating, using a machine learning similarity framework, an ensemble of binary trees;
for each tree in the ensemble of binary trees:
propagating objects that include images through the each tree until the objects reach leaf nodes of the each tree, wherein the propagating compares at least one feature associated with each object in the objects to at least one property of at least one node associated with the each tree; and
determining a similarity score for each of two objects from the objects in the leaf nodes, wherein determining the similarity score further comprises:
determining a deepest common node between a first object and a second object in each of the two objects in the each tree; and
determining a depth of the each tree, wherein determining the similarity score is based on the deepest common node and the depth of the each tree; and
for each of the two objects, combining similarity scores from the each tree in the ensemble of trees into an object similarity score, wherein the object similarity score indicates a similarity between the two objects;
identifying using object similarity scores and a similarity threshold, an object with an image similar to other objects with other images; and
substituting, the object with the image in place of the other objects in an image recognition system.
17 . The non-transitory computer readable medium of claim 16 , wherein generating the ensemble of trees further uses at least one hyperparameter, wherein the at least one hyperparameter controls depth of the each tree in the ensemble of binary trees and a number of leaf nodes in the at least one tree in the ensemble of binary trees.
18 . The non-transitory computer readable medium of claim 16 , wherein the operations further comprise:
generating a similarity matrix, the similarity matrix including entries storing object similarity scores for each of the two objects in the objects, wherein each row and each column in the similarity matrix represents the objects.
19 . The non-transitory computer readable medium of claim 16 , wherein the generating the ensemble of binary trees comprises:
determining a tree importance for the each tree in the ensemble of binary trees; and
wherein determining the similarity score for the each of the pair of objects is based on the tree importance for the each tree.
20 . The non-transitory computer readable medium of claim 19 , further comprising:
determining the tree importance of the each tree in the ensemble of trees by calculating an error of the ensemble of trees before and after adding the each tree to the ensemble of trees.