Efficient data processing to identify information and reformat data files, and applications thereof
The present disclosure is directed to systems and methods for identifying demographic information in a data file. The method may include: receiving the data file containing a plurality of fields of demographic information from a third-party, the data file having inconsistent or mislabeled nomenclatures for one or more fields of the plurality of fields or spurious demographic information; analyzing the data file using a machine learning model trained according to other data files to distinguish between each of the plurality of fields of demographic information, the machine learning model being based on a plurality of machine learning algorithms to identify different types demographic information; generating a score indicating a probability that each of the plurality of fields of demographic information was identified correctly; and generating a revised data file labeling each of the plurality of fields of demographic information based on the identified type.
1. A computer-implemented method of identifying demographic information in a data file, comprising:
training a machine learning model according to labeled, sampled training sets to identify a heading based at least in part on structure and content of the data file from a data source with information describing medical providers, the machine learning model being based on a plurality of machine learning algorithms to identify different types of demographic information;
receiving, by a processor, data files containing a plurality of fields of demographic information from a plurality of third-party sources, the data files having inconsistent or mislabeled nomenclatures with one another for one or more fields of the plurality of fields of demographic information;
analyzing, by the processor, the heading identifying a plurality of strings representing a field of demographic information in the data files using the machine learning model, wherein the analyzing is performed across two or more fields of demographic information to identify a relationship between headings, wherein the identification of the relationship further comprises:
determining types of demographic information;
determining whether the headings should be grouped as pairs; and
determining whether entries of each of the headings describe a same medical provider; and
generating, by the processor, a score indicating a probability that each of the plurality of fields of demographic information was identified correctly, wherein the generating the score further comprises:
generating a baseline score for each of the plurality of fields of demographic information; and
adjusting the baseline score to increase the score when the heading and content of the field of demographic information match, or to decrease the score when the heading and the content of the field of demographic information do not match;
generating, by the processor, a revised data file labeling each of the plurality of fields of demographic information based on the identified type; and
inserting, by the processor and in the revised data file, missing fields of demographic information based on the identified type of demographic information.
2. The method of claim 1 , wherein analyzing the data file comprises analyzing semantic content of each of the plurality of fields of demographic information to identify the different types of demographic information.
3. The method of claim 1 , wherein analyzing the data file comprises analyzing a shape of each of the plurality of fields of demographic information to identify the different types of demographic information.
4. The method of claim 1 , wherein analyzing the data file comprises analyzing metadata of each of the plurality of fields of demographic information to identify the different types of demographic information.
5. The method of claim 4 , wherein the metadata includes each nomenclature of each of the plurality of fields of demographic information.
6. The method of claim 1 , wherein, in response to identifying different ones of the plurality of fields of demographic information, the method further comprises cross-checking at least one of the plurality of fields of demographic information against known demographic information.
7. The method of claim 1 , further comprising transmitting the revised data file to a third-party.
8. The method of claim 1 , wherein the analyzing comprises distinguishing between each of the plurality of fields of demographic information based on a position of the heading in the data files with respect to another position of another heading in the data files.
9. A system for identifying demographic information in a data file, comprising:
a memory that stores instructions for identifying the demographic information in the data file; and
a processor configured to execute the instructions that cause the processor to:
train a machine learning model according to labeled, sampled training sets to identify a heading based at least in part on structure and content of the data file from a data source with information describing medical providers, the machine learning model being based on a plurality of machine learning algorithms to identify different types of demographic information;
receive, by the processor, data files containing a plurality of fields of demographic information from a plurality of third-party sources, the data files having inconsistent or mislabeled nomenclatures with one another for one or more fields of the plurality of fields of demographic information;
analyze, by the processor, the heading identifying a plurality of strings representing a field of demographic information in the data files using the machine learning model, wherein the analyze is performed across two or more fields of demographic information to identify a relationship between headings, wherein the identification of the relationship further comprises the instructions that cause the processor to:
determine types of demographic information;
determine whether the headings should be grouped as pairs;
determine whether entries of each of the headings describe a same medical provider; and
generate, by the processor, a score indicating a probability that each of the plurality of fields of demographic information was identified correctly, wherein the generate the score further comprises the instructions that cause the processor to:
generate a baseline score for each of the plurality of fields of demographic information; and
adjust the baseline score to increase the score when the heading and content of the field of demographic information match, or to decrease the score when the heading and the content of the field of demographic information do not match;
generate, by the processor, a revised data file labeling each of the plurality of fields of demographic information based on the identified type; and
insert, by the processor and in the revised data file, missing fields of demographic information based on the identified type of demographic information.
10. The system of claim 9 , wherein analyzing the data file comprises analyzing semantic content of each of the plurality of fields of demographic information to identify the different types of demographic information.
11. The system of claim 9 , wherein analyzing the data file comprises analyzing metadata of each of the plurality of fields of demographic information to identify the different types of demographic information.
12. The system of claim 11 , wherein the metadata includes each nomenclature of each of the plurality of fields of demographic information.
13. The system of claim 9 , wherein analyzing the data file comprises analyzing each nomenclature to identify the different types of demographic information.
14. The system of claim 9 , wherein, in response to identifying different ones of the plurality of fields of demographic information, the instructions further cause the processor to cross-check at least one of the plurality of fields of demographic information against known demographic information.
15. The system of claim 9 , wherein the instructions further cause the processor to transmit the revised data file to a third-party.
16. A non-transitory program storage device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations, the operations comprising:
training a machine learning model according to labeled, sampled training sets to identify a heading based at least in part on structure and content of a data file from a data source with information describing medical providers, the machine learning model being based on a plurality of machine learning algorithms to identify different types of demographic information;
receiving data files containing a plurality of fields of demographic information from a plurality of third-party sources, the data files having inconsistent or mislabeled nomenclatures with one another for one or more fields of the plurality of fields of demographic information;
analyzing the heading identifying a plurality of strings representing a field of demographic information in the data files using the machine learning model, wherein the analyzing is performed across two or more fields of demographic information to identify a relationship between headings, wherein the identification of the relationship further comprises:
determining types of demographic information;
determining whether the headings should be grouped as pairs; and
determining whether entries of each of the headings describe a same medical provider; and
generating a score indicating a probability that each of the plurality of fields of demographic information was identified correctly, wherein the generating the score further comprises:
generating a baseline score for each of the plurality of fields of demographic information; and
adjusting the baseline score to increase the score when the heading and content of the field of demographic information match, or to decrease the score when the heading and the content of the field of demographic information do not match;
generating a revised data file labeling each of the plurality of fields of demographic information based on the identified type; and
inserting in the revised data file, missing fields of demographic information based on the identified type of demographic information.
17. The non-transitory program storage device of claim 16 , wherein analyzing the data file comprises analyzing semantic content of each of the plurality of fields of demographic information to identify the different types of demographic information.
18. The non-transitory program storage device of claim 16 , wherein analyzing the data file comprises analyzing a shape of each of the plurality of fields of demographic information to identify the different types of demographic information.
19. The non-transitory program storage device of claim 16 , wherein analyzing the data file comprises analyzing metadata of each of the plurality of fields of demographic information to identify the different types of demographic information.
20. The non-transitory program storage device of claim 19 , wherein the metadata includes each nomenclature of each of the plurality of fields of demographic information.
21. The non-transitory program storage device of claim 16 , wherein, in response to identifying different ones of the plurality of fields of demographic information, the operations further comprise cross-checking at least one of the plurality of fields of demographic information against known demographic information.