System and Method for Ensemble Expert Diversification and Control Thereof

1 . A method implemented on at least one machine including at least one processor, memory, and communication platform capable of connecting to a network for machine learning, the method comprising:

sending a training sample to an expert for training a model representative of the expert;

receiving a prediction generated by the expert in accordance with the training sample and based on one or more parameters associated with the model;

analyzing a metric with respect to the prediction characterizing the prediction;

sending, when the metric satisfies a first criterion, a ground truth label associated with the training sample to the expert to facilitate the training.

2 . The method of claim 1 , wherein the metric includes a confidence score indicative of a level of confidence of the expert in the prediction.

3 . The method of claim 1 , wherein the one or more parameters of the model are updated by the expert during the training based on the prediction and the ground truth label.

4 . The method of claim 1 , wherein the step of sending comprises:

receiving, from the expert, a bid for the training sample in a bidding amount determined according to a level of available bidding currency associated with the expert; and

determining whether the expert is among one or more winners of bids from one or more experts based on at least one condition;

transmitting the training sample to the expert when the expert is among the one or more winners.

5 . The method of claim 4 , further comprising updating the level of available bidding currency associated with the expert when

the expert is among the one or more winners; and

a second criterion is satisfied.

6 . The method of claim 5 , wherein the second criterion is evaluated with respect to the metric with respect to the prediction.

7 . The method of claim 5 , wherein the update to the level of available bidding currency is determined based on the amount of the bid.

8 . Machine readable and non-transitory medium having information stored thereon for machine learning, wherein the information, when read by the machine, causes the machine to perform:

sending a training sample to an expert for training a model representative of the expert;

receiving a prediction generated by the expert in accordance with the training sample and based on one or more parameters associated with the model;

analyzing a metric with respect to the prediction characterizing the prediction;

sending, when the metric satisfies a first criterion, a ground truth label associated with the training sample to the expert to facilitate the training.

9 . The medium of claim 8 , wherein the metric includes a confidence score indicative of a level of confidence of the expert in the prediction.

10 . The medium of claim 8 , wherein the one or more parameters of the model are updated by the expert during the training based on the prediction and the ground truth label.

11 . The medium of claim 8 , wherein the step of sending comprises:

receiving, from the expert, a bid for the training sample in a bidding amount determined according to a level of available bidding currency associated with the expert; and

determining whether the expert is among one or more winners of bids from one or more experts based on at least one condition;

transmitting the training sample to the expert when the expert is among the one or more winners.

12 . The medium of claim 11 , wherein the information, when read by the machine, further causes the machine to perform updating the level of available bidding currency associated with the expert when

the expert is among the one or more winners; and

a second criterion is satisfied.

13 . The medium of claim 12 , wherein the second criterion is evaluated with respect to the metric with respect to the prediction.

14 . The medium of claim 12 , wherein the update to the level of available bidding currency is determined based on the amount of the bid.

15 . A system for machine learning, comprising:

a training data distribution unit configured for sending a training sample to an expert for training a model representative of the expert; and

a ground truth allocation unit configured for

receiving a prediction generated by the expert in accordance with the training sample and based on one or more parameters associated with the model,

analyzing a metric with respect to the prediction characterizing the prediction, and

sending, when the metric satisfies a first criterion, a ground truth label associated with the training sample to the expert to facilitate the training, wherein

the one or more parameters of the model are updated by the expert during the training based on the prediction and the ground truth label.

16 . The system of claim 15 , wherein the metric includes a confidence score indicative of a level of confidence of the expert in the prediction.

17 . The system of claim 15 , wherein the training data distribution unit is further configured for:

receiving, from the expert, a bid for the training sample in a bidding amount determined according to a level of available bidding currency associated with the expert; and

transmitting the training sample to the expert if the expert is among one or more winners of bids received from one or more experts.

18 . The system of claim 17 , further comprising a bidding winner selector configured for:

analyzing bids from the one or more experts; and

selecting the one or more winners based on the bids.

19 . The system of claim 18 , further comprising a currency allocation updater configured for updating the level of available bidding currency associated with the expert when the expert is among the one or more winners and a second criterion is satisfied.

20 . The system of claim 19 , wherein

the second criterion is evaluated with respect to the metric with respect to the prediction; and

the update to the level of available bidding currency is determined based on the amount of the bid.