Method for real-time enhancement of a predictive algorithm by a novel measurement of concept drift using algorithmically-generated features
A predictive analytics system and method in the setting of multi-class classification are disclosed, for identifying systematic changes in an evaluation dataset processed by a fraud-detection model by examining the time series histories of an ensemble of entities such as accounts. The ensemble of entities is examined and processed both individually and in aggregate, via a set of features determined previously using a distinct training dataset. The specific set of features in question may be calculated from the entity's time series history, and may or may not be used by the model to perform the classification. Certain properties of the detected changes are measured and used to improve the efficacy of the predictive model.
1. A computer-implemented method embedded in a non-transitory machine-readable medium, the method executable by one or more processors running on a machine configured for classification of events, the method comprising:
partitioning at least one dataset having at least one input data record based on an attribute associated with one or more input data records fed to a predictive model that classifies one or more events;
generating a set of features based on one or more historical data records fed to the predictive model according to different class labels for known class memberships associated with the one or more historical data records;
determining a first set of features for the at least one input data record based on the generated set of features and the partitioning of the at least one dataset;
generating a second set of features based on the first set of features, the second set of features representing a concept drift associated with the at least one dataset, wherein the concept drift represents a systematic shift in statistical distribution properties of new data records input over time to train the predictive model, the concept drift resulting in less accurate event classification by the predictive model, and wherein the new data records and the one or more historical data records are mutually exclusive;
responsive to at least one of the first set of features and the second set of features, generating a first score and a second score,
the first score representing a likelihood of an extent or magnitude of the concept drift, and
the second score representing a likelihood that the at least one new input data record is subject to the extent or magnitude of the concept drift as represented by the first score; and
adjusting a class membership for the new input data record based on the second score and according to the extent or magnitude of the concept drift to improve accurate classification in the predictive model.
2. The method of claim 1 , wherein the adjusting is based on a degree with which a classified event associated with the input event is unusual as determined by the second score and the extent or magnitude of the concept drift.
3. The method of claim 1 , wherein the first set of features is microscopically derived in relation to the attribute.
4. The method of claim 1 , wherein the second set of features is macroscopically derived based on an aggregated set of microscopically derived features associated with the input data record and input data records fed to the predictive model prior to the input data record.
5. The method of claim 1 , wherein the first set of features is associated with an entity defined by the partitioning of the at least one dataset, the partitioning of the at least one data set generating a summary statistic of the historical data records.
6. The method of claim 5 , wherein the class membership adjustment is based on the partitioning of the at least one dataset for generating a summary statistic of the one or more historical data records prior to generating the set of features based on the one or more historical data records.
7. The method of claim 5 , wherein the class membership adjustment is based on the one or more historical data records.
8. The method of claim 5 , wherein the class membership adjustment includes a binary classification.
9. The method of claim 5 , wherein the class membership adjustment is based on a feed-forward neural network such that the set of features based on the historical one or more data records are hidden nodes of the feed-forward neural network.
10. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
partitioning at least one dataset having at least one input data record based on an attribute associated with one or more input data records fed to a predictive model;
generating a set of features based on one or more historical data records fed to the predictive model according to different class labels for known class memberships associated with the one or more historical data records;
determining a first set of features for the at least one input data record based on the partitioning of the at least one dataset and the generated set of features;
generating a second set of features based on the first set of features, the second set of features representing a concept drift associated with the at least one dataset, wherein the concept drift represents a systematic shift in statistical distribution properties of new data records input over time to train the predictive model, the concept drift resulting in less accurate event classification by the predictive model, and wherein the new data records and the one or more historical data records are mutually exclusive;
responsive to at least one of the first set of features and the second set of features, generating a first score and a second score,
the first score representing a likelihood of an extent and magnitude of the concept drift, and
the second score representing a likelihood that a new input data record is subject to the extent and magnitude of the concept drift as represented by the first score; and
adjusting a class membership for the new input data record based on the second score and according to a feed-forward neural network, wherein the set of features based on the historical one or more data records are hidden nodes of the feed-forward neural network, to improve fraud detection in the predictive model responsive to the magnitude of the concept drift.
11. The computer program product of claim 10 , wherein the adjusting is based on a degree with which a classified event associated with the input event is unusual as determined by the second score and the magnitude of the concept drift.
12. The computer program product of claim 10 , wherein the first set of features are microscopically derived in relation to the attribute.
13. The computer program product of claim 10 , wherein the second set of features are macroscopically derived based on an aggregated set of microscopically derived features associated with the input data record and input data records fed to the predictive model prior to the input data record.
14. The computer program product of claim 10 , wherein the first set of features are associated with an entity defined by the partitioning of the at least one dataset, partitioning the one or more data records includes generating a summary statistic of the historical data records prior to generating the set of features based on the historical data records.
15. A computer-implemented system comprising:
at least one programmable processor; and
a non-transitory, machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
partitioning at least one dataset having at least one input data record based on an attribute associated with one or more input data records fed to a predictive model;
generating a set of features based on one or more historical data records fed to the predictive model according to different class labels for known class memberships associated with the one or more historical data records;
determining a first set of features for the at least one input data record based on the partitioning of the at least one dataset and the generated set of features;
generating a second set of features based on the first set of features, the second set of features representing a concept drift associated with the at least one dataset, wherein the concept drift represents a systematic shift in statistical distribution properties of new data records input over time to train the predictive model, the concept drift resulting in less accurate event classification by the predictive model, and wherein the new data records and the one or more historical data records are mutually exclusive;
responsive to at least one of the first set of features and the second set of features, generating a first score and a second score,
the first score representing a likelihood of an extent and magnitude of the concept drift, and
the second score representing a likelihood that a new input data record is subject to the extent and magnitude of the concept drift as represented by the first score; and
adjusting a class membership for the new input data record based on the second score and according to the magnitude of the concept drift.
16. The system of claim 15 , wherein the adjusting is based on a degree with which a classified event associated with the input event is unusual as determined by the second score and the magnitude of the concept drift.
17. The system of claim 15 , wherein the first set of features are microscopically derived in relation to the attribute.
18. The system of claim 15 , wherein the second set of features are macroscopically derived based on an aggregated set of microscopically derived features associated with the input data record and input data records fed to the predictive model prior to the input data record.
19. The system of claim 15 , wherein the first set of features are associated with an entity defined by the partitioning of the at least one dataset, partitioning the one or more data records includes generating a summary statistic of the historical data records prior to generating the set of features based on the historical data records.
20. The system of claim 19 , wherein the class membership adjustment is based on the partitioning of the at least one dataset for generating a summary statistic of the one or more historical data records using a feed-forward neural network according to magnitude of the concept drift, the feed-forward neural network being trained using backpropagation and stochastic gradient decent on tagged input data records involving at least a first transaction and a second transaction, the second transaction being prior in time to the first transaction.