System and method for ensemble expert diversification

1. A method implemented on at least one machine including at least one processor, memory, and communication platform capable of connecting to a network for machine learning, the method comprising:

receiving, by an expert trainer, from a source, based on a level of available bidding currency, a training sample for learning;

generating, by the expert trainer, a prediction in accordance with the training sample and based on one or more parameters associated with a model;

determining a metric characterizing a level of confidence of the prediction, wherein the level of available bidding currency is dynamically allocated to the expert trainer by a controller based on the level of confidence of the prediction;

transmitting the prediction and the metric to the source, wherein the metric is to be used by the source to reach a determination on whether a ground truth label for the training sample is to be provided;

receiving, if the source determined that the metric of the prediction satisfies a certain condition, the ground truth label from the source; and

enhancing the model by updating the one or more parameters of the model based on the prediction and the ground truth label.

2. The method of claim 1 , wherein the metric includes a confidence score indicative of the level of confidence in the prediction, and

wherein the receiving the ground truth label comprises receiving, if the source determined that the level of confidence exceeds a predetermined level, the ground truth label from the source.

3. The method of claim 2 , wherein the determination of whether the ground truth label is to be provided is based on whether the metric satisfies a certain criterion.

4. The method of claim 1 , wherein the step of receiving the training sample comprises:

checking the level of available bidding currency to be used for bidding the training sample;

sending a bid for the training sample in an amount within the available bidding currency; and

obtaining the training sample when the bid is successful.

5. The method of claim 4 , further comprising receiving an update to the level of available bidding currency.

6. The method of claim 5 , wherein the update to the level of available bidding currency is determined based on at least one of the amount of the bid and the metric.

7. The method of claim 1 , wherein the step of updating comprises:

computing a discrepancy between the prediction and the ground truth label;

determining an adjustment to the one or more parameters of the model based on the discrepancy; and

adjusting the one or more parameters in accordance with the adjustment.

8. Machine readable and non-transitory medium having information recorded thereon for machine learning, wherein the information, once read by the machine, causes a machine to perform:

receiving, by an expert trainer, from a source, based on a level of available bidding currency, a training sample for learning;

generating, by the expert trainer, a prediction in accordance with the training sample and based on one or more parameters associated with a model;

determining a metric characterizing a level of confidence of the prediction, wherein the level of available bidding currency is dynamically allocated to the expert trainer by a controller based on the level of confidence of the prediction;

transmitting the prediction and the metric to the source, wherein the metric is to be used by the source to reach a determination on whether a ground truth label for the training sample is to be provided;

receiving, if the source determined that the metric of the prediction satisfies a certain condition, the ground truth label from the source; and

enhancing the model by updating the one or more parameters of the model based on the prediction and the ground truth label.

9. The medium of claim 8 , wherein the metric includes a confidence score indicative of the level of confidence in the prediction, and

wherein the receiving the ground truth label comprises receiving, if the source determined that the level of confidence exceeds a predetermined level, the ground truth label from the source.

10. The medium of claim 9 , wherein the determination of whether the ground truth label is to be provided is based on whether the metric satisfies a certain criterion.

11. The medium of claim 8 , wherein the step of receiving the training sample comprises:

checking the level of available bidding currency to be used for bidding the training sample;

sending a bid for the training sample in an amount within the available bidding currency; and

obtaining the training sample when the bid is successful.

12. The medium of claim 11 , wherein the information, when read by the machine, further causes the machine to perform receiving an update to the level of available bidding currency.

13. The medium of claim 12 , wherein the update to the level of available bidding currency is determined based on at least one of the amount of the bid and the metric.

14. The medium of claim 8 , wherein the step of updating comprises:

computing a discrepancy between the prediction and the ground truth label;

determining an adjustment to the one or more parameters of the model based on the discrepancy; and

adjusting the one or more parameters in accordance with the adjustment.

15. A system for machine learning, comprising:

a training data processing unit of an expert trainer implemented with a processor and configured for receiving, from a source, a training sample for learning; and

a training unit of the expert trainer implemented with the processor and configured for generating a prediction in accordance with the training sample and based on one or more parameters associated with a model; and

a confidence assessment unit of the expert trainer implemented with the processor and configured for

determining a metric characterizing a level of confidence of the prediction, wherein the level of available bidding currency is dynamically allocated to the expert trainer by a controller based on the level of confidence of the prediction, and

transmitting the prediction and the metric to the source, wherein the metric is to be used by the source to reach a determination on whether a ground truth label for the training sample is to be provided, wherein the training unit is further configured for

receiving, if the source determined that the metric of the prediction satisfies a certain condition, the ground truth label from the source, and

enhancing the model by updating, upon receiving the ground truth label, the one or more parameters of the model based on the prediction and the ground truth label.

16. The system of claim 15 , wherein the metric includes a confidence score indicative of the level of confidence in the prediction, and

wherein the receiving the ground truth label comprises receiving, if the source determined that the level of confidence exceeds a predetermined level, the ground truth label from the source.

17. The system of claim 16 , wherein the determination of whether the ground truth label is to be provided is based on whether the metric satisfies a certain criterion.

18. The system of claim 15 , further comprising:

a currency assessment unit implemented with the processor and configured for checking a level of available bidding currency to be used for bidding the training sample; and

a training data bidding unit implemented with the processor and configured for sending a bid for the training sample in an amount within the available bidding currency, wherein

the training data processing unit is further configured for obtaining the training sample when the bid is successful.

19. The system of claim 18 , wherein the level of available bidding currency is updated.

20. The system of claim 19 , wherein the level of available bidding currency is updated based on at least one of the amount of the bid and the metric.