SYSTEMS AND METHODS OF USING CONFIGURABLE FUNCTIONS TO HARMONIZE DATA FROM DISPARATE SOURCES
Example techniques for using configurable functions to harmonize data from disparate sources may include retrieving first and second datasets including data records with values for respective first and second sets of fields; analyzing the first and second sets of fields to identify a third set of fields included in both the first and second sets of fields; identifying data records from the first and second plurality of data records having matching values for the third set of fields; stitching the identified data records from the first and second pluralities of data records with one another in order to generate a third dataset including a third plurality of data records having values for both the first and second set of fields; applying one or more functions to the third plurality of data records of the third dataset to produce an output dataset; and displaying the output dataset via a user interface.
1 . A computer-implemented method for using configurable functions to harmonize data from disparate sources, comprising:
retrieving, by one or more processors, a first dataset from a first external data source, the first dataset including a first plurality of data records having values for each of a first set of fields;
retrieving, by the one or more processors, a second dataset from a second external data source, distinct from the first external data source, the second dataset including a second plurality of data records having values for each of a second set of fields;
analyzing, by the one or more processors, the first set of fields and the second set of fields to identify a third set of fields, the third set of fields being fields included in both the first set of fields and the second set of fields;
identifying, by the one or more processors, one or more data records of the first plurality of data records, and one or more respective data records of the second plurality of data records, having matching values for fields of the third set of fields;
stitching, by the one or more processors, each identified data record of the first plurality of data records with each respective identified data record of the second plurality of data records in order to generate a third dataset including a third plurality of data records having values for each of the first set of fields and for each of the second set of fields;
applying, by the one or more processors, one or more functions to the third plurality of data records of the third dataset to produce an output dataset; and
displaying, by the one or more processors, the output dataset via a user interface.
2 . The method of claim 1 , further comprising:
analyzing, by the one or more processors, the first dataset in order to identify the first set of fields; and
analyzing, by the one or more processors, the second dataset in order to identify the second set of fields.
3 . The method of claim 2 , wherein analyzing the first dataset in order to identify the first set of fields includes analyzing the respective values of each field of the first set of fields in order to identify the first set of fields, and wherein analyzing the second dataset in order to identify the second set of fields includes analyzing the respective values of each field of the second set of fields in order to identify the second set of fields.
4 . The method of claim 1 , wherein identifying one or more data records of the first plurality of data records, and one or more respective data records of the second plurality of data records, having matching values for the third set of fields further comprises:
converting, by the one or more processors, one or more values for a field of the third set of fields in the first dataset from a first format associated with the first dataset to a second format associated with the second dataset; and
comparing, by the one or more processors, the converted one or more values for the field of the third set of fields in the first dataset to one or more values for the field of the third set of fields in the second dataset in order to identify the one or more data records of the first plurality of data records, and one or more respective data records of the second plurality of data records, having matching values for the third set of fields.
5 . The method of claim 1 , further comprising:
accessing, by the one or more processors, a library of pre-built functions; and
identifying, by the one or more processors, the one or more functions to be applied to the third dataset.
6 . The method of claim 5 , wherein identifying the one or more functions to be applied to the third dataset is based on the identified fields of the third set of fields.
7 . The method of claim 1 , wherein each data record of the third dataset is associated with an individual, and wherein the output dataset includes recommendations or predictions for the individuals associated with the data records of the third dataset.
8 . A computer system for comprising one or more processors, and one or more memories storing non-transitory computer-readable instructions for using configurable functions to harmonize data from disparate sources, that, when executed by one or more processors, cause the one or more processors to:
retrieve a first dataset from a first external data source, the first dataset including a first plurality of data records having values for each of a first set of fields;
retrieve a second dataset from a second external data source, distinct from the first external data source, the second dataset including a second plurality of data records having values for each of a second set of fields;
analyze the first set of fields and the second set of fields to identify a third set of fields, the third set of fields being fields included in both the first set of fields and the second set of fields;
identify one or more data records of the first plurality of data records, and one or more respective data records of the second plurality of data records, having matching values for the third set of fields;
stitch each identified data record of the first plurality of data records with each respective identified data record of the second plurality of data records in order to generate a third dataset including a third plurality of data records having values for each of the first set of fields and for each of the second set of fields;
apply one or more functions to the third plurality of data records of the third dataset to produce an output dataset; and
display the output dataset via a user interface.
9 . The computer system of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
analyze the first dataset in order to identify the first set of fields; and
analyze the second dataset in order to identify the second set of fields.
10 . The computer system of claim 9 , wherein analyzing the first dataset in order to identify the first set of fields includes analyzing the respective values of each field of the first set of fields in order to identify the first set of fields, and wherein analyzing the second dataset in order to identify the second set of fields includes analyzing the respective values of each field of the second set of fields in order to identify the second set of fields.
11 . The computer system of claim 8 , wherein identifying one or more data records of the first plurality of data records, and one or more respective data records of the second plurality of data records, having matching values for the third set of fields includes:
converting one or more values for a field of the third set of fields in the first dataset from a first format associated with the first dataset to a second format associated with the second dataset; and
comparing the converted one or more values for the field of the third set of fields in the first dataset to one or more values for the field of the third set of fields in the second dataset in order to identify the one or more data records of the first plurality of data records, and one or more respective data records of the second plurality of data records, having matching values for the third set of fields.
12 . The computer system of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
access a library of pre-built functions; and
identify the one or more functions to be applied to the third dataset.
13 . The computer system of claim 12 , wherein identifying the one or more functions to be applied to the third dataset is based on the identified fields of the third set of fields.
14 . The computer system of claim 8 , wherein each data record of the third dataset is associated with an individual, and wherein the output dataset includes recommendations or predictions for the individuals associated with the data records of the third dataset.
15 . A non-transitory computer-readable medium storing instructions for using configurable functions to harmonize data from disparate sources that, when executed by one or more processors, cause the one or more processors to:
retrieve a first dataset from a first external data source, the first dataset including a first plurality of data records having values for each of a first set of fields;
retrieve a second dataset from a second external data source, distinct from the first external data source, the second dataset including a second plurality of data records having values for each of a second set of fields;
analyze the first set of fields and the second set of fields to identify a third set of fields, the third set of fields being fields included in both the first set of fields and the second set of fields;
identify one or more data records of the first plurality of data records, and one or more respective data records of the second plurality of data records, having matching values for the third set of fields;
stitch each identified data record of the first plurality of data records with each respective identified data record of the second plurality of data records in order to generate a third dataset including a third plurality of data records having values for each of the first set of fields and for each of the second set of fields;
apply one or more functions to the third plurality of data records of the third dataset to produce an output dataset; and
display the output dataset via a user interface.
16 . The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
analyze the first dataset in order to identify the first set of fields; and
analyze the second dataset in order to identify the second set of fields.
17 . The non-transitory computer-readable medium of claim 16 , wherein analyzing the first dataset in order to identify the first set of fields includes analyzing the respective values of each field of the first set of fields in order to identify the first set of fields, and wherein analyzing the second dataset in order to identify the second set of fields includes analyzing the respective values of each field of the second set of fields in order to identify the second set of fields.
18 . The non-transitory computer-readable medium of claim 15 , wherein identifying one or more data records of the first plurality of data records, and one or more respective data records of the second plurality of data records, having matching values for the third set of fields includes:
converting one or more values for a field of the third set of fields in the first dataset from a first format associated with the first dataset to a second format associated with the second dataset; and
comparing the converted one or more values for the field of the third set of fields in the first dataset to one or more values for the field of the third set of fields in the second dataset in order to identify the one or more data records of the first plurality of data records, and one or more respective data records of the second plurality of data records, having matching values for the third set of fields.
19 . The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
access a library of pre-built functions; and
identify the one or more functions to be applied to the third dataset.
20 . The non-transitory computer-readable medium of claim 15 , wherein identifying the one or more functions to be applied to the third dataset is based on the identified fields of the third set of fields.