Labeler scoring
Apparatuses, systems, and techniques to assess labeling of objects within an image. In at least one embodiment, objects are labeled in an image and then said labeling is assessed based, at least in part, on one or more prior assessments of the one or more unlabeled objects.
1 . A processor, comprising:
one or more circuits to:
cause an evaluation of labeling of one or more unlabeled objects within one or more images, at least in part, on one or more prior of labelings of the one or more unlabeled objects;
cause a comparison of a time for the labeling to an expected time for the labeling based, at least in part, on timing information from one or more labelings of a same labeling job type; and
generate one or more labeling performance scores based, at least in part, on the evaluation and the comparison.
2 . The processor of claim 1 , wherein the one or more circuits are to generate one or more redundant labeling tasks and distributing the one or more redundant labeling tasks to one or more labelers.
3 . The processor of claim 1 , wherein the one or more circuits are to assess the labeling of the one or more images by at least determining a consensus label associated with the one or more prior of labelings of the one or more unlabeled objects.
4 . The processor of claim 1 , wherein the one or more circuits are to assess the labeling of the one or more images by at least determining an average time to perform the one or more prior labelings of the one or more unlabeled objects.
5 . The processor of claim 1 , wherein the one or more circuits are to generate one or more key performance indicators while at least a portion of a dataset comprising the one or more images is unlabeled.
6 . The processor of claim 1 , wherein the one or more circuits are to store information indicative of one or more key performance indicators associated with one or more labelers performing the labeling.
7 . The processor of claim 1 , wherein the one or more circuits are to generate one or more reports indicative of labeling performance, the one or more reports generated based, at least in part, on one or more key performance indicators determined based, at least in part, on a consensus label and an average time spent to generate a label.
8 . A system, comprising:
one or more processors to:
cause an evaluation of labeling of one or more unlabeled objects within one or more images based, at least in part, on one or more prior labelings of the one or more unlabeled objects;
cause a comparison of a time for the labeling to an expected time for the labeling based, at least in part, on timing information from one or more labelings of a same labeling job type; and
generate one or more labeling performance scores based, at least in part, on the evaluation and the comparison.
9 . The system of claim 8 , wherein the one or more processors are to generate one or more redundant labeling tasks and distributing the one or more redundant labeling tasks to one or more labelers.
10 . The system of claim 8 , wherein the one or more processors are to assess the labeling of the one or more images by at least determining a consensus label associated with the one or more prior labelings of the one or more unlabeled objects.
11 . The system of claim 8 , wherein the one or more processors are to assess the labeling of the one or more images by at least determining an average time to perform the one or more prior labelings of the one or more unlabeled objects.
12 . The system of claim 8 , wherein the one or more processors are to generate one or more key performance indicators while at least a portion of a dataset comprising the one or more images is unlabeled.
13 . The system of claim 8 , wherein the one or more processors are to store information indicative of one or more key performance indicators associated with one or more labelers performing the labeling.
14 . The system of claim 8 , wherein the one or more processors are to generate one or more reports indicative of labeling performance, the one or more reports generated based, at least in part, on one or more key performance indicators determined based, at least in part, on a consensus label and an average time spent to generate a label.
15 . A method, comprising:
causing an evaluation of labeling of one or more unlabeled objects within one or more images based, at least in part, on one or more prior labelings of the one or more unlabeled objects;
causing a comparison of a time for the labeling to an expected time for the labeling based, at least in part, on timing information from one or more labelings of a same labeling job type; and
generate one or more labeling performance scores based, at least in part, on the evaluation and the comparison.
16 . The method of claim 15 , further comprising generating one or more redundant labeling tasks and distributing the one or more redundant labeling tasks to one or more labelers.
17 . The method of claim 15 , further comprising assessing the labeling of the one or more images by at least determining a consensus label associated with the one or more prior labelings of the one or more unlabeled objects.
18 . The method of claim 15 , further comprising assessing the labeling of the one or more images by at least determining an average time to perform the one or more prior labelings of the one or more unlabeled objects.
19 . The method of claim 15 , further comprising storing information indicative of one or more key performance indicators associated with one or more labelers performing the labeling.
20 . The method of claim 15 , further comprising generating one or more reports indicative of labeling performance, the one or more reports generated based, at least in part, on one or more key performance indicators determined based, at least in part, on a consensus label and an average time spent to generate a label.