Systems and methods for classification of database records by mitigating bias using random data re-censoring
Methods and systems are described herein for improving data processing efficiency of classifying user files in a database. More particularly, methods and systems are described herein for improving data processing efficiency of classifying user files in a database in which the user files have a temporal element. The methods and systems described herein accomplish these improvements by mitigating bias using random data censoring.
1 . A system for classifying user files in a database into disparate states by mitigating bias using random data censoring, the system comprising:
one or more processors; and
a non-transitory, computer-readable medium comprising instructions that when executed by the one or more processors cause operations comprising:
receiving a request to determine respective probabilities that a user record for a user belongs to each of a plurality of user record states;
retrieving a user record history for the user record, wherein the user record history comprises periodic status checks, wherein each of the periodic status checks comprises a status check date;
determining that the user record history comprises a bias associated with the periodic status checks;
in response to determining that the user record history comprises a bias, generating a re-censored record history by randomly re-censoring the user record history based on a set build date, wherein generating the re-censored record history by re-censoring the user record history based on the set build date comprises:
determining the set build date based on an availability of data in the user record history;
determining a respective periodic status check proximity at each of the periodic status checks by:
retrieving a time stamp for each periodic status check of the periodic status checks; and
comparing the time stamp to the set build date;
determining a respective re-censoring probability at each of the periodic status checks based on the respective periodic status check proximity; and
determining whether a respective user record status at each of the periodic status checks is available based on the respective periodic status check proximity;
generating a feature input for the user record history based on the respective user record status at each of the periodic status checks that is available in the re-censored record history;
processing the feature input using a first model;
receiving a first output of the first model indicating the respective probabilities that the user record for the user belongs to each of the plurality of user record states; and
generating a recommendation based on the first output.
2 . A method of classifying user files in a database into disparate states by mitigating bias using random data censoring, the method comprising:
receiving a request to determine respective probabilities that a user record for a user belongs to each of a plurality of user record states;
retrieving a user record history for the user record, wherein the user record history comprises periodic status checks, wherein each of the periodic status checks comprises a status check date;
determining that the user record history comprises a bias associated with the periodic status checks;
in response to determining that the user record history comprises a bias, generating a re-censored record history by randomly re-censoring the user record history based on a set build date, wherein generating the re-censored record history by re-censoring the user record history based on the set build date comprises:
determining the set build date based on an availability of data in the user record history;
determining a respective periodic status check proximity at each of the periodic status checks by:
retrieving a time stamp for each periodic status check of the periodic status checks; and
comparing the time stamp to the set build date;
determining a respective re-censoring probability at each of the periodic status checks based on the respective periodic status check proximity; and
determining whether a respective user record status at each of the periodic status checks is available based on the respective periodic status check proximity;
generating a feature input for the user record history based on the respective user record status at each of the periodic status checks that is available in the re-censored record history;
processing the feature input using a first model;
receiving a first output of the first model indicating the respective probabilities that the user record for the user belongs to each of the plurality of user record states; and
generating a recommendation based on the first output.
3 . The method of claim 2 , wherein generating the re-censored record history by re-censoring the user record history based on the set build date further comprises:
receiving a user input of a data cutoff date; and
determining the set build date prior to the data cutoff date.
4 . The method of claim 2 , wherein generating the re-censored record history by re-censoring the user record history based on the set build date further comprises:
determining an availability of data in the user record history;
determining a data cutoff date based on the availability of data; and
determining the set build date prior to the data cutoff date.
5 . The method of claim 2 , wherein generating the re-censored record history by re-censoring the user record history based on the set build date further comprises:
determining a data cutoff date based on stationarity of data in the user record history;
determining the data cutoff date based on the stationarity of data; and
determining the set build date prior to the data cutoff date.
6 . The method of claim 2 , further comprising:
determining a number of the periodic status checks in the user record history; and
selecting the first model from a plurality of models based on the number of the periodic status checks in the user record history corresponding to a first range.
7 . The method of claim 2 , wherein generating the re-censored record history by re-censoring the user record history based on the set build date further comprises:
receiving a user input of a time-dependent re-censoring mechanism; and
applying the time-dependent re-censoring mechanism to the user record history.
8 . The method of claim 7 , wherein applying the time-dependent re-censoring mechanism to the user record history further comprises:
receiving a first datapoint in the user record history; and
setting an event status for the first datapoint based on the time-dependent re-censoring mechanism.
9 . The method of claim 2 , wherein processing the feature input using the first model further comprises:
applying a parametric survival analysis to the feature input; and
determining the first output based on the parametric survival analysis.
10 . The method of claim 2 , wherein processing the feature input using the first model further comprises:
applying a Kaplan-Meier curve analysis to the feature input; and
determining the first output based on the Kaplan-Meier curve analysis.
11 . The method of claim 2 , wherein processing the feature input using the first model further comprises:
applying a Cox proportional hazards regression to the feature input; and
determining the first output based on the Cox proportional hazards regression.
12 . The method of claim 2 , wherein the first model is a stochastic probability model comprising a Markov chain.
13 . The method of claim 2 , wherein the first model uses transition probability matrices to forecast a probability of an action related to the user record.
14 . The method of claim 2 , wherein the first output is a probability distribution row vector.
15 . The method of claim 2 , wherein the plurality of user record states comprises a user record state corresponding to a charge-off of the user record.
16 . One or more non-transitory, computer-readable mediums comprising instructions that, when executed by one or more processors, cause operations comprising:
receiving a request to determine respective probabilities that a user record for a user belongs to each of a plurality of user record states;
retrieving a user record history for the user record, wherein the user record history comprises periodic status checks, wherein each of the periodic status checks comprises a status check date;
determining that the user record history comprises a bias associated with the periodic status checks;
in response to determining that the user record history comprises a bias, generating a re-censored record history by re-censoring the user record history based on a set build date, wherein generating the re-censored record history by re-censoring the user record history based on the set build date comprises:
determining the set build date based on an availability of data in the user record history;
determining a respective periodic status check proximity at each of the periodic status checks by:
retrieving a time stamp for each periodic status check of the periodic status checks; and
comparing the time stamp to the set build date;
determining a respective re-censoring probability at each of the periodic status checks based on the respective periodic status check proximity; and
determining whether a respective user record status at each of the periodic status checks is available based on the respective periodic status check proximity;
generating a feature input for the user record history based on the respective user record status at each of the periodic status checks that is available in the re-censored record history;
processing the feature input using a first model;
receiving a first output of the first model indicating the respective probabilities that the user record for the user belongs to each of the plurality of user record states; and
generating a recommendation based on the first output.