Supervised machine learning of data de-duplication

1. A computer-implemented method of de-duplicating data from industry reports, the computer-implemented method having steps comprising:

receiving through a communications interface a set of industry reports from multiple industry sources;

comparing with circuitry one or more attributes of at least two trade lines to identify whether the at least two trade lines in the set of industry reports are duplicates;

characterizing with the circuitry as a binary indication whether the comparing indicates the one or more attributes of the at least two trade lines are a match and storing the binary indication in a memory;

displaying on a display a representation of the binary indication and receiving via a graphical user interface (GUI) a user-identified indication whether the at least two trade lines are duplicates;

training a classifier through circuitry with feedback from the user-identified indication, the classifier being a computer-implemented classifier;

recording in the memory the binary indication whether the at least two trade lines are duplicates and when the at least two trade lines are duplicates, removing at least one of the at least two trade lines from the set of industry reports so as to de-duplicate a portion of the at least two trade lines; and

running the classifier with a processor as an intelligent proxy for subsequent de-duplication decisions,

wherein the set of industry reports includes reports from one of electronic file archival reports, electronic indexing reports, census reporting reports, inventory recording reports, research organization reports, financial decision-making reports, and evaluation and achievement reporting reports.

2. The computer-implemented method of claim 1 , further comprising:

dividing data results of the comparing with other comparing steps to form a training set of data and a testing set of data.

3. The computer-implemented method of claim 2 , wherein the training set of data contains at least 60% of the data results and the testing set of data contains not more than 40% of the data results.

4. The computer-implemented method of claim 2 , further comprising:

applying the training set of data to fit the classifier via a k-fold cross-validation process; and

testing an accuracy of the classifier via the testing set of data.

5. The computer-implemented method of claim 2 , further comprising:

fitting with the circuitry a plurality of decision trees for an associated plurality of subsamples of the data results.

6. The computer-implemented method of claim 5 , wherein the fitting is executed via an Extra Trees Classifier implemented by the circuitry.

7. The computer-implemented method of claim 1 , wherein a positive comparison of the one or more attributes of the at least two trade lines comprises blending of data by triggering a de-duplication of the portion of the at least two trade lines.

8. The computer-implemented method of claim 1 , wherein a negative comparison of the one or more attributes of the at least two trade lines comprises keeping the at least two trade lines as distinct trade lines without blending of data.

9. The computer-implemented method of claim 1 , wherein the set of industry reports include credit reports from different credit bureaus.

10. A supervised machine learning de-duplication system, comprising:

a communications interface configured to receive a set of industry reports from multiple industry sources; and

processing circuitry configured to

compare one or more attributes of at least two trade lines to identify whether the at least two trade lines in the set of industry reports are duplicates,

characterize as a binary indication whether the comparing indicates the one or more attributes of the at least two trade lines are a match and store in a memory the binary indication,

display a representation of the binary indication and receive via a graphical user interface (GUI) a user-identified indication whether the at least two trade lines are duplicates,

train a classifier with feedback from the user-identified indication, the classifier being a computer-implemented classifier,

record in the memory the binary indication whether the at least two trade lines are duplicates and remove at least one of the at least two trade lines from the set of industry reports so as to de-duplicate a portion of the at least two trade lines, and

run the classifier with a processor as an intelligent proxy for subsequent de-duplication decisions,

wherein the binary indication includes one or more of a true positive indicator, a true negative indicator, a false positive indicator, and a false negative indicator, and

wherein an accuracy of the classifier is a function of a total sum of the true positive indicators and the true negative indicators, divided by a total sum of the true positive indicators, the false negative indicators, the false positive indicators, and the true negative indicators.

11. The system of claim 10 , wherein the processing circuitry is further configured to

divide comparison results into a training set of data and a testing set of data;

apply the training set of data to fit the classifier via a k-fold cross-validation process;

test an accuracy of the classifier via the testing set of data; and

fit a plurality of decision trees for an associated plurality of sub-samples of the data results.

12. The system of claim 10 , wherein a positive comparison indicates redundancy of trade lines and triggers a de-duplication of one of the trade lines.

13. The system of claim 10 , wherein a negative comparison indicates at most a partial redundancy and indicates the two trade lines should be kept without blending of data.

14. The system of claim 10 , wherein the set of industry reports includes credit reports from different credit bureaus.

15. The system of claim 14 , wherein the one or more attributes include one or more of an account number, an account type, a date of an account opening, a data repository source, a high credit amount, a lost or stolen indicator, a bureau identity, a subscriber information, a balance, a payment amount, and a credit limit.

16. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions that when executed by circuitry performs a method of de-duplicating data, the method having steps comprising:

receiving through a communications interface a set of industry reports from multiple industry sources;

comparing with circuitry one or more attributes of at least two trade lines to identify whether the at least two trade lines in the set of industry reports are duplicates;

characterizing with the circuitry as a binary indication whether the comparing indicates the one or more attributes of the at least two trade lines are a match and storing in a memory the binary indication;

displaying on a display a representation of the binary indication and receiving via a graphical user interface (GUI) a user-identified indication whether the at least two trade lines are duplicates;

training a classifier through circuitry with feedback from the user-identified indication, the classifier being a computer-implemented classifier;

recording in memory the binary indication whether the at least two trade lines are duplicates and removing at least one of the at least two trade lines from the set of industry reports so as to de-duplicate a portion of the at least two trade lines;

running the classifier with a processor as an intelligent proxy for subsequent de-duplication decisions;

applying the training set of data to fit the classifier via a k-fold cross-validation process;

testing an accuracy of the classifier via the testing set of data; and

fitting a plurality of decision trees for an associated plurality of sub-samples of the data results.