Inferring confidence and need for natural language processing of input data

1. A method comprising:

receiving a data set comprising a plurality of records, wherein each respective record of the plurality of records corresponds to a respective patient of a plurality of patients contains a respective one or more fields;

identifying a first group of fields in the plurality of records, wherein the first group of fields comprises at least one field from each record from the plurality of records, and wherein the first group of fields is identified based on determining that each field in the first group of fields has a common metadata attribute, wherein the common metadata attribute comprises a label;

determining a first plurality of metrics for the first group of fields based on metadata associated with each field in the first group of fields, wherein the first plurality of metrics describe metadata aspects of the first group of fields, wherein the first plurality of metrics comprise: (i) a percentage of records in the plurality of records that includes a field associated with the first group of fields and (ii) a number of distinct values of fields in the first group of fields;

assigning one or more weight values to each of the first plurality of metrics;

generating a natural language processing (NLP) measure and a discreteness measure for the first group of fields, based on the first plurality of metrics and the one or more assigned weight values;

selecting a first processing workflow of a plurality of processing workflows to use when ingesting data from the first group of fields into a corpus, based on comparing the NLP measure and the discreteness measure to one or more predefined thresholds; and

processing each of the fields in the first group of fields using the first processing workflow.

2. The method of claim 1 , wherein the NLP measure indicates a level of necessity for performing a NLP operation when processing each of the first group of fields, and wherein the discreteness measure indicates a level of confidence for values in the first group of fields.

3. The method of claim 1 , wherein the metadata describing each of the respective one or more fields comprises at least one of: (i) a type of the respective field, (ii) a name of the respective field, (iii) one or more values of the respective field, (iv) a data type of the respective field, and (v) one or more NLP normalized values of the respective field.

4. The method of claim 3 , wherein the first plurality of metrics includes at least one of: (i) an average number of words in each field in the first group of fields; (ii) a number of words in a largest field of the first group of fields; (iii) a percentage of the first group of fields where a normalized value of the field matches an original value of the field; and (iv) a percentage of the first group of fields where the field value cannot be normalized.

5. The method of claim 1 , wherein the first processing workflow comprises:

directly extracting a value from each of the first subset of fields; and

ingesting the values into the corpus.

6. The method of claim 1 , wherein the first processing workflow comprises:

processing each of the fields in the first group of fields with one or more NLP models to determine a corresponding value; and

ingesting the values into the corpus.

7. The method of claim 1 , wherein the first processing workflow comprises:

flagging the first group of fields for additional review; and

ingesting the values from the first group of fields based on the additional review.

8. The method of claim 1 , wherein the first processing workflow comprises refraining from ingesting values corresponding to the first group of fields into the corpus.

9. The method of claim 1 , wherein assigning a weight value to each of the first plurality of metrics comprises processing each of the first plurality of metrics using a machine learning model, wherein the machine learning model was trained based on historical fields with a known NLP measure and discreteness measure.

10. The method of claim 1 , the method further comprising determining a monetary value of the data set based on the NLP measure and the discreteness measure.

11. A computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:

receiving a data set comprising a plurality of records, wherein each respective record of the plurality of records corresponds to a respective patient of a plurality of patients contains a respective one or more fields;

identifying a first group of fields in the plurality of records, wherein the first group of fields comprises at least one field from each record from the plurality of records, and wherein the first group of fields is identified based on determining that each field in the first group of fields has a common metadata attribute, wherein the common metadata attribute comprises a label;

determining a first plurality of metrics for the first group of fields based on metadata associated with each field in the first group of fields, wherein the first plurality of metrics describe metadata aspects of the first group of fields, wherein the first plurality of metrics comprise: (i) a percentage of records in the plurality of records that includes a field associated with the first group of fields and (ii) a number of distinct values of fields in the first group of fields;

assigning one or more weight values to each of the first plurality of metrics;

generating a natural language processing (NLP) measure and a discreteness measure for the first group of fields based on the first plurality of metrics and the one or more assigned weight values;

selecting a first processing workflow of a plurality of processing workflows to use when ingesting data from the first group of fields into a corpus, based on comparing the NLP measure and the discreteness measure to one or more predefined thresholds; and

processing each of the fields in the first group of fields using the first processing workflow.

12. The computer program product of claim 11 , wherein the NLP measure indicates a level of necessity for performing a NLP operation when processing each of the first group of fields, and wherein the discreteness measure indicates a level of confidence for values in the first group of fields.

13. The computer program product of claim 11 , wherein the metadata describing each of the respective one or more fields comprises at least one of: (i) a type of the respective field, (ii) a name of the respective field, (iii) one or more values of the respective field, (iv) a data type of the respective field, and (v) one or more NLP normalized values of the respective field.

14. The computer program product of claim 13 , wherein the first plurality of metrics includes at least one of: (i) an average number of words in each field in the first group of fields; (ii) a number of words in a largest field of the first group of fields; (iii) a percentage of the first group of fields where a normalized value of the field matches an original value of the field; and (iv) a percentage of the first group of fields where the field value cannot be normalized.

15. The computer program product of claim 11 , wherein assigning a weight value to each of the first plurality of metrics comprises processing each of the first plurality of metrics using a machine learning model, wherein the machine learning model was trained based on historical fields with a known NLP measure and discreteness measure.

16. A system comprising:

one or more computer processors; and

a memory containing a program which when executed by the one or more computer processors performs an operation, the operation comprising:

receiving a data set comprising a plurality of records, wherein each respective record of the plurality of records corresponds to a respective patient of a plurality of patients contains a respective one or more fields;

identifying a first group of fields in the plurality of records, wherein the first group of fields comprises at least one field from each record from the plurality of records, and wherein the first group of fields is identified based on determining that each field in the first group of fields has a common metadata attribute, wherein the common metadata attribute comprises a label;

determining a first plurality of metrics for the first group of fields based on metadata associated with each field in the first group of fields, wherein the first plurality of metrics describe metadata aspects of the first group of fields, wherein the first plurality of metrics comprise: (i) a percentage of records in the plurality of records that includes a field associated with the first group of fields and (ii) a number of distinct values of fields in the first group of fields;

assigning one or more weight values to each of the first plurality of metrics;

generating a natural language processing (NLP) measure and a discreteness measure for the first group of fields, based on the first plurality of metrics and the one or more assigned weight values;

selecting a first processing workflow of a plurality of processing workflows to use when ingesting data from the first group of fields into a corpus, based on comparing the NLP measure and the discreteness measure to one or more predefined thresholds; and

processing each of the fields in the first group of fields using the first processing workflow.

17. The system of claim 16 , wherein the NLP measure indicates a level of necessity for performing a NLP operation when processing each of the first group of fields, and wherein the discreteness measure indicates a level of confidence for values in the first group of fields.

18. The system of claim 16 , wherein the metadata describing each of the respective one or more fields comprises at least one of: (i) a type of the respective field, (ii) a name of the respective field, (iii) one or more values of the respective field, (iv) a data type of the respective field, and (v) one or more NLP normalized values of the respective field.

19. The system of claim 18 , wherein the first plurality of metrics includes at least one of: (i) an average number of words in each field in the first group of fields; (ii) a number of words in a largest field of the first group of fields; (iii) a percentage of the first group of fields where a normalized value of the field matches an original value of the field; and (iv) a percentage of the first group of fields where the field value cannot be normalized.

20. The system of claim 16 , wherein assigning a weight value to each of the first plurality of metrics comprises processing each of the first plurality of metrics using a machine learning model, wherein the machine learning model was trained based on historical fields with a known NLP measure and discreteness measure.