Targeted crowd sourcing for metadata management across data sets
A system includes: a memory operable to store a predictive model; a first processor communicatively coupled to the memory, the first processor operable to execute the predictive model to perform operations including generating knowledge score metrics based on a set of attributes for individuals included in a specified population, where the knowledge score metrics quantify a prediction of a capability of an individual for performing metadata labeling; a second processor communicatively coupled to the memory and the first processor, the second processor is operable to perform operations including comparing the knowledge score metrics to a specified threshold, and identifying attributes of individuals from a specified population having knowledge score metrics exceeding the specified threshold as attributes of individuals capable of performing metadata labeling.
1. A method comprising:
forming a machine learning model operable on a neural network processor in a computing environment, the machine learning model configured to analyze input data comprising an initial set of attributes to predict analytical capabilities of a set of individuals for performing a specific task;
operating the neural network processor to execute the machine learning model, wherein the machine learning model generates knowledge score metrics for the set of individuals based at least in part on the initial set of attributes, and wherein the knowledge score metrics quantify a prediction of an analytic capability of each individual in the set of individuals for performing the specific task;
comparing, by a second processor in the computing environment, the knowledge score metrics to a specified threshold;
identifying, by the second processor, an additional set of attributes of individuals from the set of individuals having knowledge score metrics exceeding the specified threshold, wherein the additional set of attributes specifically identifies attributes of individuals capable of performing the specific task, wherein at least one attribute of the additional set of attributes is not included in the initial set of attributes used to generate the knowledge score metrics; and
providing, as additional training input to the machine learning model, the additional set of attributes.
2. The method of claim 1 , wherein the specific task comprises:
analyzing a data set; and
based on analyzing the data set, generating metadata labels for data in the data set.
3. The method of claim 1 , wherein the initial set of attributes comprises one or more of: demographic data, education data, or data regarding a role within a company, and wherein the initial set of attributes indicates a degree of familiarity with a given data set.
4. The method of claim 1 , wherein the set of individuals comprises people within a company that meet a specified minimum criterion.
5. The method of claim 1 , wherein the specified threshold for the knowledge score metrics is based on the knowledge score metrics of a seed set of experts.
6. The method of claim 1 , further comprising:
requesting individuals having the additional set of attributes to perform metadata labeling of metadata for data in a data set;
assessing an accuracy of the metadata labeling performed by the individuals; and
updating the metadata when the accuracy of the metadata labeling is assessed as accurate.
7. The method of claim 1 , further comprising:
requesting individuals having the additional set of attributes to perform metadata labeling;
assessing an accuracy of the metadata labeling performed by the individuals; and
updating the machine learning model for the individuals having the additional set of attributes, based on the accuracy of the metadata labeling.
8. The method of claim 7 , further comprising:
updating the machine learning model by adjusting weights of attributes of at least one of: the initial set of attributes or the additional set of attributes which are determined to be beneficial or less relevant for performing metadata labeling.
9. A non-transitory computer readable medium storing instructions which, when executed by at least one processor, cause the at least one processor to perform operations, the operations comprising:
forming a machine learning model in a computing environment, the machine learning model configured to analyze input data comprising an initial set of attributes to predict analytical capabilities of a set of individuals for performing a specific task;
executing the machine learning model, wherein the machine learning model generates knowledge score metrics for the set of individuals based at least in part on the initial set of attributes, and wherein the knowledge score metrics quantify a prediction of an analytic capability of each individual in the set of individuals for performing the specific task;
comparing, in the computing environment, the knowledge score metrics to a specified threshold;
identifying an additional set of attributes of individuals from the set of individuals having knowledge score metrics exceeding the specified threshold, wherein the additional set of attributes specifically identifies attributes of individuals capable of performing the specific task, wherein at least one attribute of the additional set of attributes is not included in the initial set of attributes used to generate the knowledge score metrics; and
providing, as additional training input to the machine learning model, the additional set of attributes.
10. The non-transitory computer readable medium of claim 9 , wherein the specific task comprises:
analyzing a data set; and
based on analyzing the data set, generating metadata labels for data in the data set.
11. The non-transitory computer readable medium of claim 9 , wherein the initial set of attributes comprises one or more of: demographic data, education data, or data regarding a role within a company, and wherein the initial set of attributes indicates a degree of familiarity with a given data set.
12. The non-transitory computer readable medium of claim 9 , wherein the set of individuals comprises people within a company that meet a specified minimum criterion.
13. The non-transitory computer readable medium of claim 9 , wherein the specified threshold for the knowledge score metrics is based on the knowledge score metrics of a seed set of experts.
14. The non-transitory computer readable medium of claim 9 , the operations further comprising:
requesting individuals having the additional set of attributes to perform metadata labeling of metadata for data in a data set;
assessing an accuracy of the metadata labeling performed by the individuals; and
updating the metadata when the accuracy of the metadata labeling is assessed as accurate.
15. The non-transitory computer readable medium of claim 9 , the operations further comprising:
requesting individuals having the additional set of attributes to perform metadata labeling;
assessing an accuracy of the metadata labeling performed by the individuals; and
updating the machine learning model for the individuals having the additional set of attributes based on the accuracy of the metadata labeling.
16. A system comprising:
a memory operable to store a predictive model;
a first processor communicatively coupled to the memory, wherein the first processor is operable to execute the predictive model to perform operations, the operations comprising:
generating knowledge score metrics based on an initial set of attributes for individuals included in a specified population, wherein the knowledge score metrics quantify a prediction of a capability of an individual for performing metadata labeling; and
a second processor communicatively coupled to the memory and the first processor, wherein the second processor is operable to perform second operations, the second operations comprising:
comparing the knowledge score metrics to a specified threshold; and
identifying an additional set of attributes of individuals from a specified population having knowledge score metrics exceeding the specified threshold, wherein the additional set of attributes specifically identifies attributes of individuals capable of performing metadata labeling, wherein at least one attribute of the additional set of attributes is not included in the initial set of attributes on which the knowledge score metrics are based,
wherein the first processor is further for updating the predictive model using the additional set of attributes as new training input.
17. The system of claim 16 , wherein the initial set of attributes comprises one or more of: demographic data, education data, or data regarding a role within a company, and wherein the initial set of attributes indicates a degree of familiarity with a given data set.
18. The system of claim 16 , wherein the specified threshold for the knowledge score metrics is based on knowledge score metrics of a seed set of experts.
19. The system of claim 16 ,
wherein the second processor is further operable to perform second operations comprising:
generating requests to individuals having the additional set of attributes to perform metadata labeling; and
wherein the first processor is further operable to perform operations comprising:
receiving as input an assessment of an accuracy of the metadata labeling performed by the individuals; and
updating the predictive model for the individuals having the additional set of attributes based on the accuracy of the metadata labeling.
20. The system of claim 19 , wherein the second processor is further operable to perform second operations comprising:
updating the predictive model by adjusting weights of attributes of at least one of: the initial set of attributes or the additional set of attributes which are determined by the predictive model to be beneficial or less relevant for performing metadata labeling.