System and method for ensemble expert diversification via bidding
The present teaching relates to method, system, medium, and implementations for machine learning. A check is performed on a level of available bidding currency for bidding a training sample that is used to train a model via machine learning. A bid in an amount within the available bidding currency is sent, to a source of the training sample, for the training sample. The training sample is received from the source when the bid is successful. A prediction is then generated in accordance with the training sample based on one or more parameters associated with the model and is sent to the source.
1. A method implemented on at least one machine including at least one processor, memory, and communication platform capable of connecting to a network for machine learning, the method comprising:
checking a level of available bidding currency for bidding a training sample used for training a model via machine learning, wherein the available bidding currency is dynamically allocated based on a confidence level in prediction generated during the training;
sending, to a source of the training sample, a bid for the training sample in an amount within the available bidding currency;
receiving the training sample from the source when the bid is successful; and
sending, to the source, a prediction generated in accordance with the training sample based on one or more parameters associated with the model.
2. The method of claim 1 , wherein the source determines how to distribute the training sample to achieve expert diversification.
3. The method of claim 1 , wherein the available bidding currency is
initialized by the source; and
is deducted by the amount upon sending the bid to the source.
4. The method of claim 1 , further comprising
computing, with respect to the prediction, a metric characterizing the prediction; and
sending, to the source, the metric together with the prediction to facilitate the source to determine whether a ground truth label for the training sample is to be provided.
5. The method of claim 4 , wherein the metric includes a confidence score indicative of the level of confidence in the prediction.
6. The method of claim 1 , further comprising
receiving, from the source, a ground truth label corresponding to the training sample; and
updating the one or more parameters of the model based on the prediction and the ground truth label.
7. The method of claim 6 , wherein the step of updating comprises:
computing a discrepancy between the prediction and the ground truth label;
determining an adjustment to the one or more parameters of the model based on the discrepancy; and
adjusting the one or more parameters in accordance with the adjustment.
8. Machine readable non-transitory medium having information recorded thereon for machine learning, wherein the information, once read by a machine, causes the machine to perform:
checking a level of available bidding currency for bidding a training sample used for training a model via machine learning, wherein the available bidding currency is dynamically allocated based on a confidence level in prediction generated during the training;
sending, to a source of the training sample, a bid for the training sample in an amount within the available bidding currency;
receiving the training sample from the source when the bid is successful; and
sending, to the source, a prediction generated in accordance with the training sample based on one or more parameters associated with the model.
9. The medium of claim 8 , wherein the source determines how to distribute the training sample to achieve expert diversification.
10. The medium of claim 8 , wherein the available bidding currency is
initialized by the source; and
is deducted by the amount upon sending the bid to the source.
11. The medium of claim 8 , wherein the information, when read by the machine, further causes the machine to perform:
computing, with respect to the prediction, a metric characterizing the prediction; and
sending, to the source, the metric together with the prediction to facilitate the source to determine whether a ground truth label for the training sample is to be provided.
12. The medium of claim 11 , wherein the metric includes a confidence score indicative of the level of confidence in the prediction.
13. The medium of claim 4 - 8 , wherein the information, once read by the machine, further causes the machine to perform:
receiving, from the source, a ground truth label corresponding to the training sample; and
updating the one or more parameters of the model based on the prediction and the ground truth label.
14. The medium of claim 13 , wherein the step of updating comprises:
computing a discrepancy between the prediction and the ground truth label;
determining an adjustment to the one or more parameters of the model based on the discrepancy; and
adjusting the one or more parameters in accordance with the adjustment.
15. A system for machine learning, comprising:
a currency assessment unit implemented with a processor and configured for checking a level of available bidding currency for bidding a training sample used for training a model via machine learning, wherein the available bidding currency is dynamically allocated based on a confidence level in prediction generated during the training;
a training data bidding unit implemented with the processor and configured for sending, to a source of the training sample, a bid for the training sample in an amount within the available bidding currency;
a training data processing unit implemented with the processor and configured for receiving the training sample from the source when the bid is successful; and
a training unit implemented with the processor and configured for sending, to the source, a prediction generated in accordance with the training sample based on one or more parameters associated with the model.
16. The system of claim 15 , wherein the source determines how to distribute the training sample to achieve expert diversification.
17. The system of claim 15 , wherein the available bidding currency is
initialized by the source; and
is deducted by the amount upon sending the bid to the source.
18. The system of claim 15 , further comprising a confidence assessment unit implemented with the processor and configured for:
computing, with respect to the prediction, a metric characterizing the prediction; and
sending, to the source, the metric together with the prediction to facilitate the source to determine whether a ground truth label for the training sample is to be provided, wherein
the metric includes a confidence score indicative of the level of confidence in the prediction.
19. The system of claim 15 , wherein the training unit is further configured for:
receiving, from the source, a ground truth label corresponding to the training sample; and
updating the one or more parameters of the model based on the prediction and the ground truth label.
20. The system of claim 19 , wherein the step of updating comprises:
computing a discrepancy between the prediction and the ground truth label;
determining an adjustment to the one or more parameters of the model based on the discrepancy; and
adjusting the one or more parameters in accordance with the adjustment.