Mixed intelligence data labeling system for machine learning
View Patent ↗A method of hybrid data labeling for machine learning, including receiving multiple unlabeled objects forming an unlabeled data set, pre-labeling the unlabeled data set by a machine learning system to output a pending label data pool, bifurcating the pending label data pool by the machine learning system into high and low confidence sets, dispatching the high confidence set to a machine labeler, dispatching the low confidence set to a human labeler, merging the label sets to return a pre-review label data pool, determining a difference between the pending label data pool and the pre-review label data pool, review labeling the data objects, if the determined difference of the data objects is greater than a predefined error threshold and storing the data objects to a reviewed pool if the determined difference of the data objects is less than and equal to the predefined error threshold.
1. A method of hybrid data labeling for machine learning, comprising:
receiving a plurality of data objects that are unlabeled, wherein the unlabeled data objects form an unlabeled data set;
pre-labeling the unlabeled data set by a machine learning system to output a pending label data pool;
bifurcating the pending label data pool by the machine learning system into a high confidence set and a low confidence set;
dispatching the high confidence set to a machine labeler to return a machine labeled set;
dispatching the low confidence set to at least one human labeler to return a human defined label set;
merging the machine labeled set and the human defined label set to return a pre-review label data pool;
determining a difference between the pending label data pool and the pre-review label data pool;
review labeling of the at least one of the plurality of data objects, if the determined difference of at least one of the plurality of data objects is greater than a predefined error threshold; and
storing the at least one of the plurality of data objects to a reviewed pool if the determined difference of at least one of the plurality of data objects is at least one of less than and equal to the predefined error threshold.
2. The method of hybrid data labeling of claim 1 , further comprising:
merging the reviewed pool and the review labeled at least one of the plurality of data objects into an acceptable label result pool; and
storing the acceptable label result pool to the pending label data pool.
3. The method of hybrid data labeling of claim 2 , further comprising adding metadata comments to the review labeled at least one of the plurality of data objects.
4. The method of hybrid data labeling of claim 3 , further comprising resetting a portion of the high confidence set to the low confidence set when the determined difference is greater than the predefined error threshold.
5. The method of hybrid data labeling of claim 4 , further comprising providing feedback to the at least one human labeler when the determined difference is greater than the predefined error threshold.
6. The method of hybrid data labeling of claim 5 , further comprising dispatching a mis-labeled object from the low confidence set by the at least one human labeler to another of the at least one human labeler.
7. The method of hybrid data labeling of claim 1 , further comprising training the machine labeler based on the review labeling.
8. The method of hybrid data labeling of claim 1 , wherein the dispatching of the low confidence set is based on at least one of pre-computed quality, customer provided quantity, bidding price and registered human labor availability.
9. The method of hybrid data labeling of claim 1 , wherein the determining the difference is based on intersection of union of labels.
10. The method of hybrid data labeling of claim 1 , wherein the determining the difference is based on matching bonding boxes.
11. The method of hybrid data labeling of claim 1 , wherein the determining the difference is based on free space differences.