Training machine learning algorithms with temporally variant personal data, and applications thereof
To train models, training data is needed. As personal data changes over time, the training data can get stale, obviating its usefulness in training the model. Embodiments deal with this by developing a database with a running log specifying how each person's data changes at the time. When data is ingested, it may not be normalized. To deal with this, embodiments clean the data to ensure the ingested data fields are normalized. Finally, the various tasks needed to train the model and solve for accuracy of personal data can quickly become cumbersome to a computing device. They can conflict with one another and compete inefficiently for computing resources, such as processor power and memory capacity. To deal with these issues, a scheduler is employed to queue the various tasks involved.
1 . A computer-implemented method for linking ingested data, the method comprising:
accessing, by one or more computing devices, data records of individuals;
parsing, by the one or more computing devices, the data records to locate each individual's demographic information;
assigning, by the one or more computing devices, the demographic information into predetermined categories;
comparing, by the one or more computing devices, each record against first categorized records using a pair-wise function to determine if they are the same;
training, by the one or more computing devices, a neural network using a training set, wherein the training set is retrieved from an database at a particular time an accuracy of the database was verified;
calculating, using the trained neural network and by the one or more computing devices, a similarity score for each pair of records, wherein the similarity score is a ratio based on how many categories match for each data pair of records;
determining, by the one or more computing devices, whether the similarity score meets or exceeds a similarity score threshold;
based on determining the similarity score meets or exceeds the similarity score threshold, linking, by the one or more computing devices, the data pair in a group;
determining, by the one or more computing devices, a most prevalent identity within the group, comprising:
comparing each record against second categorized records to determine the most prevalent identify based on a clear prevalent identify not being included within the group, wherein the second categorized records comprise one or more additional categories than the first categorized records; and
modifying, by the one or more computing devices, the data records to match the most prevalent identity within the group.
2 . The method of claim 1 , further comprising normalizing, by the one or more computing devices, the data records to be consistent with a predetermined format.
3 . The method of claim 1 , wherein the comparing is performed using regular expression matching or fuzzy matching.
4 . The method of claim 3 , wherein the regular expression matching determines whether two values match when they both satisfy the same regular expression.
5 . The method of claim 3 , wherein the fuzzy matching determines whether two values match when two strings match a pattern approximately.
6 . The method of claim 1 , further comprising:
assigning, by the one or more computing devices, a weight to each of the predetermined categories; and
wherein the similarity score is determined based on the weight for each of the predetermined categories.
7 . The method of claim 1 , wherein modifying the data records to match the most prevalent identity within the group comprises standardizing, by the one or more computing devices, each individual's demographic information within the group to match the most prevalent identity within the group.
8 . A non-transitory computer readable medium having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations for linking ingested data, the operations comprising:
accessing, by one or more computing devices, data records of individuals;
parsing, by the one or more computing devices, the data records to locate each individual's demographic information;
assigning, by the one or more computing devices, the demographic information into predetermined categories;
comparing, by the one or more computing devices, each record against first categorized records using a pair-wise function to determine if they are the same;
training, by the one or more computing devices, a neural network using a training set, wherein the training set is retrieved from an database at a particular time an accuracy of the database was verified;
calculating, using the trained neural network and by the one or more computing devices, a similarity score for each pair of records, wherein the similarity score is a ratio based on how many categories match for each data pair of records;
determining, by the one or more computing devices, whether the similarity score meets or exceeds a similarity score threshold,
based on determining the similarity score meets or exceeds the similarity score threshold, linking, by the one or more computing devices, the data pair in a group;
determining, by the one or more computing devices, a most prevalent identity within the group, comprising:
comparing each record against second categorized records to determine the most prevalent identify based on a clear prevalent identify not being included within the group, wherein the second categorized records comprise one or more additional categories than the first categorized records; and
modifying, by the one or more computing devices, the data records to match the most prevalent identity within the group.
9 . The non-transitory computer readable medium of claim 8 , wherein the operations further comprise normalizing, by the one or more computing devices, the data records to be consistent with a predetermined format.
10 . The non-transitory computer readable medium of claim 8 , wherein the comparing is performed using regular expression matching or fuzzy matching.
11 . The non-transitory computer readable medium of claim 10 , wherein the regular expression matching determines whether two values match when they both satisfy the same regular expression.
12 . The non-transitory computer readable medium of claim 10 , wherein the fuzzy matching determines whether two values match when two strings match a pattern approximately.
13 . The non-transitory computer readable medium of claim 8 , wherein the operations further comprise:
assigning, by the one or more computing devices, a weight to each of the predetermined categories; and
wherein the similarity score is determined based on the weight for each of the predetermined categories.
14 . The non-transitory computer readable medium of claim 8 , wherein modifying the data records to match the most prevalent identity within the group comprises standardizing, by the one or more computing devices, each individual's demographic information within the group to match the most prevalent identity within the group.