METHODS AND SYSTEMS FOR TRAINING A DECISION-TREE BASED MACHINE LEARNING ALGORITHM (MLA)
Methods and processors for training a decision-tree based Machine Learning Algorithm (MLA) are disclosed. During a first training iteration, the processor generates a first tree, which includes generating a first tree structure with leaf nodes. The at least one from the plurality of training objects falling in one leaf node and none falling in an other node. The leaf values being based on a first noise-inducing function such that they are non-null leaf values. During a second training iteration of the decision-tree based MLA, the processor generates a second tree with a second tree structure with a third leaf node. A third leaf value is based on an estimated gradient value of a loss function for at least one training object falling in the third leaf node. The processor is configured to store the first and the second tree of the decision-tree based MLA in a storage.
1 . A method of training a decision-tree based Machine Learning Algorithm (MLA), the method executable by a processor having access to a training dataset, the training dataset comprising a plurality of training objects and a plurality of target values for respective ones from the plurality of training objects, the method comprising:
during a first training iteration of the decision-tree based MLA:
generating, by the processor, a first tree using the plurality of training objects, the generating including:
generating a first tree structure with a first leaf node and a second leaf node, at least one from the plurality of training objects falling in the first leaf node and none falling in the second leaf node; and
generating a first leaf value for the first leaf node and a second leaf value for the second leaf node, the first and second leaf values being based on a first noise-inducing function such that the first leaf value and the second leaf value are non-null leaf values;
during a second training iteration of the decision-tree based MLA:
generating, by the processor, a second tree using the training dataset, the generating including:
generating a second tree structure with a third leaf node; and
generating a third leaf value to the third leaf node, the third leaf value being based on an estimated gradient value of a loss function for at least one training object falling in the third leaf node;
storing, by the processor, the first and the second tree of the decision-tree based MLA in a storage.
2 . The method of claim 1 , wherein the first tree structure has a plurality of leaf nodes including the first leaf node and the second lead node, and wherein all leaf values assigned to the plurality of lead nodes using the first noise-inducing function are exclusively non-null leaf values.
3 . The method of claim 1 , wherein the first tree structure is a uniformly-distributed tree structure.
4 . The method of claim 1 , wherein the second tree structure is generated using a second noise-inducing function.
5 . The method of claim 1 , wherein the generating the second tree comprises generating, by the server, the second tree structure using a Gradient Boosting (GB) technique.
6 . The method of claim 5 , wherein the GB technique includes a randomized tree generation process.
7 . The method of claim 1 , wherein the method comprises generating, by the processor, a plurality of first trees during a plurality of first training iterations, the first training iteration being one from the plurality of first training iterations, the first tree being one from the plurality of first trees.
8 . The method of claim 1 , wherein the method comprises generating, by the processor, a plurality of second trees during a plurality of second training iterations, the second training iteration being one from the plurality of second training iterations, the second tree being one from the plurality of second trees.
9 . The method of claim 1 , wherein the decision-tree based MLA is being trained for performing a regression task during an in-use phase of the decision-tree based MLA.
10 . The method of claim 1 , wherein the decision-tree based MLA is being trained for performing a classification task during an in-use phase of the decision-tree based MLA.
11 . The method of claim 1 , wherein the first noise-inducing function is a function having a null average and a finite distribution.
12 . The method of claim 1 , wherein the loss function is at least one of a 0-1 loss, Normalized Discounted Cumulative Gain (NDCG), and PFound.
13 . The method of claim 1 , wherein the loss function is at least one of a hinge loss, logistic loss, and squared error loss.
14 . A processor for training a decision-tree based Machine Learning Algorithm (MLA), the processor having access to a training dataset, the training dataset comprising a plurality of training objects and a plurality of target values for respective ones from the plurality of training objects, the processor being configured to:
during a first training iteration of the decision-tree based MLA:
generate a first tree using the plurality of training objects, the generating including:
generate a first tree structure with a first leaf node and a second leaf node, at least one from the plurality of training objects falling in the first leaf node and none falling in the second leaf node; and
generate a first leaf value for the first leaf node and a second leaf value for the second leaf node, the first and second leaf values being based on a first noise-inducing function such that the first leaf value and the second leaf value are non-null leaf values;
during a second training iteration of the decision-tree based MLA:
generate a second tree using the training dataset, the generating including:
generate a second tree structure with a third leaf node; and
generate a third leaf value to the third leaf node, the third leaf value being based on an estimated gradient value of a loss function for at least one training object falling in the third leaf node;
store the first and the second tree of the decision-tree based MLA in a storage.
15 . The processor of claim 14 , wherein the first tree structure has a plurality of leaf nodes including the first leaf node and the second lead node, and wherein all leaf values are generated for the plurality of lead nodes using the first noise-inducing function are exclusively non-null leaf values.
16 . The processor of claim 14 , wherein the first tree structure is a uniformly-distributed tree structure.
17 . The processor of claim 14 , wherein the second tree structure is generated using a second noise-inducing function.
18 . The processor of claim 14 , wherein the processor is configured to generate a plurality of first trees during a plurality of first training iterations, the first training iteration being one from the plurality of first training iterations, the first tree being one from the plurality of first trees.
19 . The processor of claim 14 , wherein the processor is configured to generate a plurality of second trees during a plurality of second training iterations, the second training iteration being one from the plurality of second training iterations, the second tree being one from the plurality of second trees.
20 . The processor of claim 14 , wherein the first noise-inducing function is a function having a null average and a finite distribution.