Decision tree with compensation for previously unseen data
View Patent ↗A computer-implemented method is disclosed for efficiently processing records with unseen data. In the method, a computer system may obtain a plurality of records and a decision tree generated in a learning process. The decision tree may include a distinction node having multiple paths extending therefrom. After arriving at the distinction node with one or more records, the computer system may determine that the one or more records correspond to data of a type not seen by the distinction node in the learning process. Thereafter, the computer system may depart the distinction node via each of the multiple paths and eventually reach multiple leaf nodes of the decision tree. Each of the multiple leaf nodes may correspond to a probability distribution. Accordingly, the computer system may combine the probability distribution of each of the multiple leaf nodes to obtain a hybrid probability distribution corresponding to the one or more records.
1. A computer-implemented method for efficiently processing records with unseen data, the method comprising:
obtaining, by a computer system, a plurality of records;
obtaining, by the computer system, a decision tree built in a learning process; and
processing, by the computer system, the plurality of records through the decision tree, the processing comprising:
arriving at a distinction node of the decision tree with one or more records of the plurality of records, the distinction node having multiple paths extending therefrom;
determining, by the computer system after the arriving, that the one or more records correspond to data of a type not seen by the distinction node in the learning process;
departing, by the computer system after the determining, the distinction node via each of the multiple paths;
reaching, by the computer system after the departing, multiple leaf nodes of the decision tree, each of the multiple leaf nodes corresponding to a probability distribution; and
combining the probability distribution of each of the multiple leaf nodes to obtain a hybrid probability distribution corresponding to the one or more records,
wherein:
the learning process comprises a first number of passes through a first path of the multiple paths and a second number of passes through a second path of the multiple paths;
the combining comprises combining a probability distribution corresponding to the first path weighted in proportion to the first number and a probability distribution corresponding to the second path weighted in proportion to the second number; and
the decision tree is a probability estimation tree.
2. The method of claim 1 , wherein the decision tree is programmed to perform record linkage.
3. The method of claim 2 , wherein each record of the plurality of records comprises a customer profile.
4. The method of claim 3 , wherein the decision tree is programmed to identify one or more first records within the plurality of records that are likely to correspond to a common customer or household.
5. The method of claim 4 , wherein the data comprises an attribute of a record of the one or more records.
6. The method of claim 4 , wherein the data comprises a result of a computation based at least in part on one or more attributes of the one or more records.
7. The method of claim 4 , wherein the hybrid probability distribution comprises a weighted combination of the probability distribution of each of the multiple leaf nodes.
8. The method of claim 1 , wherein the computer system provides a parallel computing environment.
9. The method of claim 8 , wherein the computer system comprises a plurality of worker nodes.
10. The method of claim 9 , wherein the processing is conducted by the plurality of worker nodes.
11. A computer-implemented method for efficiently processing records with unseen data, the method comprising:
obtaining, by a computer system, a plurality of records, each record comprising a customer profile;
obtaining, by the computer system, a decision tree created in a learning process; and
processing, by the computer system, the plurality of records through the decision tree, the processing comprising:
arriving at a distinction node of the decision tree with one or more records of the plurality of records, the distinction node having multiple paths extending therefrom;
determining, by the computer system after the arriving, that the one or more records correspond to data of a type not seen by the distinction node in the learning process;
departing, by the computer system after the determining, the distinction node via each of the multiple paths;
reaching, by the computer system after the departing, multiple leaf nodes of the decision tree, each of the multiple leaf nodes corresponding to a probability distribution; and
combining, by the computer system, the probability distribution of each of the multiple leaf nodes to obtain a hybrid probability distribution corresponding to the one or more records, and
wherein:
the decision tree is a probability estimation tree;
the learning process comprises a first number of passes through a first path of the multiple paths and a second number of passes through a second path of the multiple paths; and
the combining comprises combining a probability distribution corresponding to the first path weighted in proportion to the first number and a probability distribution corresponding to the second path weighted in proportion to the second number.
12. A computer system comprising:
a plurality of processors;
one or more memory devices operably connected to one or more processors of the plurality of processors, the one or more memory devices collectively storing a plurality of records; and
a plurality of comparison modules, each programmed to process records of the plurality of records through a decision tree comprising a distinction node, the distinction node having multiple paths extending therefrom, the plurality of comparison modules each being further programmed to depart the distinction node via all of the multiple paths when the distinction node is confronted with data of a type not seen by the distinction node in the learning process, and the plurality of comparison modules each being further programmed to combine probability distributions from each of multiple leaf nodes reached via the multiple paths to obtain a hybrid probability distribution for one or more records of the plurality of records corresponding to the data, and
wherein:
the decision tree is a probability estimation tree;
the learning process comprises a first number of passes through a first path of the multiple paths and a second number of passes through a second path of the multiple paths; and
the plurality of comparison modules each being further programmed to combine comprises the plurality of comparison modules each being further programmed to combine a probability distribution corresponding to the first path weighted in proportion to the first number and a probability distribution corresponding to the second path weighted in proportion to the second number.
13. The method Of claim 1 , wherein:
the decision tree is programmed to perform record linkage;
each record of the plurality of records comprises a customer profile;
the decision tree is programmed to identify one or more records within the plurality of records that are likely to correspond to a common customer or household; and
the computer system provides a parallel computing environment.