Targeted crowd sourcing for metadata management across data sets
A system includes: a memory operable to store a predictive model; a first processor communicatively coupled to the memory, the first processor operable to execute the predictive model to perform operations including generating knowledge score metrics based on a set of attributes for individuals included in a specified population, where the knowledge score metrics quantify a prediction of a capability of an individual for performing metadata labeling; a second processor communicatively coupled to the memory and the first processor, the second processor is operable to perform operations including comparing the knowledge score metrics to a specified threshold, and identifying attributes of individuals from a specified population having knowledge score metrics exceeding the specified threshold as attributes of individuals capable of performing metadata labeling.
1. A method for generating seed input for an artificial intelligence system, the method comprising:
collecting information on a set of attributes for each individual in an initial set of individuals;
assessing a quality of metadata labels produced during a metadata labeling task performed by each individual in the initial set of individuals;
evaluating, by a processor of the artificial intelligence system, the set of attributes and the quality of metadata labels produced during the metadata labeling task;
based on results of the evaluating, associating quality metrics with the information on the set of attributes for each individual in the initial set of individuals;
comparing, by the processor of the artificial intelligence system, the quality metrics to a threshold;
in response to determining that a first quality metric of the quality metrics exceeds the threshold, identifying, by the processor of the artificial intelligence system, a subset of the set of attributes which is associated with the first quality metric as being attributes of an expert with respect to a type of data in a given data set; and
inputting the subset of the set of attributes to a neural network processor to seed the artificial intelligence system to identify experts.
2. The method of claim 1 , further comprising:
in response to determining that a second quality metric of the quality metrics does not exceed the threshold, inputting, to the neural network processor, a subset of the set of attributes which is associated with the second quality metric in a hold out set of training data for the artificial intelligence system.
3. The method of claim 1 , wherein the quality of the metadata labels produced during the metadata labeling task comprises an assessed degree of accuracy in defining an operational definition of elements of a given data set.
4. The method of claim 1 , wherein the set of attributes comprises one or more of: demographic data, education data, or data regarding a role within a company, and
wherein the set of attributes indicates a degree of familiarity with a given data set.
5. The method of claim 1 , wherein at least some of the metadata labels produced during the metadata labeling task identify a technical description of data in a given data set.
6. The method of claim 1 , wherein at least some of the metadata labels produced during the metadata labeling task identify an operational description of data in a given data set.
7. The method of claim 1 , further comprising identifying training data for a plurality of different machine learning models, wherein each machine learning model of the plurality of different machine learning models is trained based on training data identified for a different type of data set.
8. A non-transitory computer readable medium storing instructions which, when executed by a processor of an artificial intelligence system, cause the processor to perform operations, the operations comprising:
collecting information on a set of attributes for each individual in an initial set of individuals;
assessing a quality of metadata labels produced during a metadata labeling task performed by each individual in the initial set of individuals;
evaluating the set of attributes and the quality of metadata labels produced during the metadata labeling task;
based on results of the evaluating, associating quality metrics with the information on the set of attributes for each individual in the initial set of individuals;
comparing the quality metrics to a threshold;
in response to determining that a first quality metric of the quality metrics exceeds the threshold, identifying a subset of the set of attributes which is associated with the first quality metric as being attributes of an expert with respect to a type of data in a given data set; and
inputting the subset of the set of attributes to a neural network processor to seed the artificial intelligence system to identify experts.
9. The non-transitory computer readable medium of claim 8 , wherein the operations further comprise:
in response to determining that a second quality metric of the quality metrics does not exceed the threshold, inputting, to the neural network processor, a subset of the set of attributes which is associated with the second quality metric in a hold out set of training data for the artificial intelligence system.
10. The non-transitory computer readable medium of claim 8 , wherein the quality of the metadata labels produced during the metadata labeling task comprises an assessed degree of accuracy in defining an operational definition of elements of a given data set.
11. The non-transitory computer readable medium of claim 8 , wherein the set of attributes comprises one or more of: demographic data, education data, or data regarding a role within a company, and
wherein the set of attributes indicates a degree of familiarity with a given data set.
12. The non-transitory computer readable medium of claim 8 , wherein at least some of the metadata labels produced during the metadata labeling task identify a technical description of data in a given data set.
13. The non-transitory computer readable medium of claim 8 , wherein at least some of the metadata labels produced during the metadata labeling task identify an operational description of data in a given data set.
14. The non-transitory computer readable medium of claim 8 , wherein the operations further comprise:
identifying training data for a plurality of different machine learning models, wherein each machine learning model of the plurality of different machine learning models is trained based on training data identified for a different type of data set.
15. An artificial intelligence system comprising:
a processor; and
a non-transitory computer readable medium storing instructions which, when executed by the processor, cause the processor to perform operations, the operations comprising:
collecting information on a set of attributes for each individual in an initial set of individuals;
assessing a quality of metadata labels produced during a metadata labeling task performed by each individual in the initial set of individuals;
evaluating the set of attributes and the quality of metadata labels produced during the metadata labeling task;
based on results of the evaluating, associating quality metrics with the information on the set of attributes for each individual in the initial set of individuals;
comparing the quality metrics to a threshold;
in response to determining that a first quality metric of the quality metrics exceeds the threshold, identifying a subset of the set of attributes which is associated with the first quality metric as being attributes of an expert with respect to a type of data in a given data set; and
inputting the subset of the set of attributes to a neural network processor to seed the artificial intelligence system to identify experts.
16. The artificial intelligence system of claim 15 , wherein the operations further comprise:
in response to determining that a second quality metric of the quality metrics does not exceed the threshold, inputting, to the neural network processor, a subset of the set of attributes which is associated with the second quality metric in a hold out set of training data for the artificial intelligence system.
17. The artificial intelligence system of claim 15 , wherein the quality of the metadata labels produced during the metadata labeling task comprises an assessed degree of accuracy in defining an operational definition of elements of a given data set.
18. The artificial intelligence system of claim 15 , wherein the set of attributes comprises one or more of: demographic data, education data, or data regarding a role within a company, and
wherein the set of attributes indicates a degree of familiarity with a given data set.
19. The artificial intelligence system of claim 15 , wherein at least some of the metadata labels produced during the metadata labeling task identify a technical description of data in a given data set or an operational description of data in a given data set.
20. The artificial intelligence system of claim 15 , wherein the operations further comprise:
identifying training data for a plurality of different machine learning models, wherein each machine learning model of the plurality of different machine learning models is trained based on training data identified for a different type of data set.