System, method, and computer program product for testing a machine learning model based on simulations in an offline environment

1 . A system comprising at least one processor programmed or configured to:

receive historical transaction data, wherein the historical transaction data comprises a plurality of data records of history payment transactions, wherein each data record has a historical timestamp;

generate online simulation data based on the historical transaction data, wherein, when generating the online simulation data, the at least one processor is programmed or configured to:

modify the historical timestamp of each data record to provide the online simulation data, wherein when modifying the historical timestamp of each data record to provide the online simulation data, the at least one processor is programmed or configured to:

assign a timestamp to each data insertion action of a plurality of data insertion actions where the timestamp corresponds to a time of a simulated payment transaction of a plurality of simulated payment transactions;

determine a timeline for the plurality of data insertion actions and a plurality of data request actions based on the online simulation data, wherein the timeline represents a simulation of a real-time flow of data that is seen in a production environment, wherein the plurality of data request actions comprises a plurality of requests for an inference from a stateful machine learning (ML) model, and wherein, when determining the timeline for the plurality of data insertion actions and the plurality of data request actions, the at least one processor is programmed or configured to:

orchestrate the plurality of data insertion actions and the plurality of data request actions in the timeline; and

select data sources of the plurality of data insertion actions and the plurality of data request actions;

perform a simulation of online activities involving the stateful ML model in an offline environment using the timeline for the plurality of data insertion actions and the plurality of data request actions, wherein the stateful ML model is configured to provide an output, in response to an input that comprises a simulated payment transaction, that includes a prediction of a category of a plurality of categories of transactions to which the simulated payment transaction is assigned, and wherein, when performing the simulation of online activities involving the stateful ML model in the offline environment using the timeline for the plurality of data insertion actions and the plurality of data request actions, the at least one processor is programmed or configured to:

begin the simulation of online activities at a time equal to a start time of the timeline;

execute one or more instructions for each time step of a plurality of time steps based on the timeline; and

update a state of the stateful ML model after each data request action of the plurality of data request actions during the simulation of online activities; and

validate the stateful ML model based on the simulation of online activities.

2 . The system of claim 1 , wherein, when performing the simulation of online activities involving the stateful ML model, the at least one processor is programmed or configured to:

perform a feature engineering procedure based on the online simulation data to provide a plurality of features for the simulation of online activities; and

validate the plurality of features for the simulation of online activities based on the historical transaction data.

3 . The system of claim 1 , wherein the at least one processor is further programmed or configured to:

perform a feature engineering procedure based on the historical transaction data to provide a plurality of features for the historical transaction data; and

wherein, when validating the plurality of features for the simulation of online activities, the at least one processor is programmed or configured to:

compare the plurality of features for the simulation of online activities to the plurality of features for the historical transaction data.

4 . The system of claim 1 , wherein, when generating the online simulation data, the at least one processor is programmed or configured to:

perform one or more data conversion operations on the historical transaction data.

5 . The system of claim 1 , wherein, when modifying the historical timestamp of each data record, the at least one processor is programmed or configured to:

replace the historical timestamp of each data record with a relative timestamp.

6 . The system of claim 1 , wherein, when performing the simulation of online activities involving the stateful ML model, the at least one processor is programmed or configured to:

generate a simulation score of the stateful ML model based on the online simulation data; and

wherein, when validating the stateful ML model, the at least one processor is programmed or configured to:

compare the simulation score of the stateful ML model to a predetermined score.

7 . A computer-implemented method comprising:

receiving, with at least one processor, historical transaction data, wherein the historical transaction data comprises a plurality of data records of history payment transactions, wherein each data record has a historical timestamp;

generating, with at least one processor, online simulation data based on the historical transaction data, wherein generating the online simulation data comprises:

modifying the historical timestamp of each data record to provide the online simulation data, wherein modifying the historical timestamp of each data record to provide the online simulation data comprises:

assigning a timestamp to each data insertion action of a plurality of data insertion actions where the timestamp corresponds to a time of a simulated payment transaction of a plurality of simulated payment transactions;

determining, with at least one processor, a timeline for the plurality of data insertion actions and a plurality of data request actions based on the online simulation data, wherein the timeline represents a simulation of a real-time flow of data that is seen in a production environment, wherein the plurality of data request actions comprises a plurality of requests for an inference from a stateful machine learning (ML) model, and wherein determining the timeline for the plurality of data insertion actions and the plurality of data request actions comprises:

orchestrating the plurality of data insertion actions and the plurality of data request actions in the timeline; and

selecting data sources of the plurality of data insertion actions and the plurality of data request actions;

performing, with at least one processor, a simulation of online activities involving the stateful ML model using the timeline for the plurality of data insertion actions and the plurality of data request actions, wherein the stateful ML model is configured to provide an output, in response to an input that comprises a simulated payment transaction, that includes a prediction of a category of a plurality of categories of transactions to which the simulated payment transaction is assigned, and wherein performing the simulation of online activities involving the stateful ML model in the offline environment using the timeline for the plurality of data insertion actions and the plurality of data request actions comprises:

beginning the simulation of online activities at a time equal to a start time of the timeline;

executing one or more instructions for each time step of a plurality of time steps based on the timeline; and

updating a state of the stateful ML model after each data request action of the plurality of data request actions during the simulation of online activities; and

validating, with at least one processor, the stateful ML model based on the simulation of online activities.

8 . The computer-implemented method of claim 7 , wherein performing the simulation of online activities involving the stateful ML model comprises:

performing a feature engineering procedure based on the online simulation data to provide a plurality of features for the simulation of online activities; and

validating the plurality of features for the simulation of online activities based on the historical transaction data.

9 . The computer-implemented method of claim 7 , further comprising:

performing a feature engineering procedure based on the historical transaction data to provide a plurality of features for the historical transaction data,

wherein validating the plurality of features for the simulation of online activities comprises:

comparing the plurality of features for the simulation of online activities to the plurality of features for the historical transaction data.

10 . The computer-implemented method of claim 7 , wherein generating the online simulation data comprises:

performing one or more data conversion operations on the historical transaction data.

11 . The computer-implemented method of claim 7 , wherein modifying the historical timestamp of each data record comprises:

replacing the historical timestamp of each data record with a relative timestamp.

12 . The computer-implemented method of claim 7 , wherein performing the simulation of online activities involving the stateful ML model comprises:

generating a simulation score of the stateful ML model based on the online simulation data; and

wherein validating the stateful ML model comprises:

comparing the simulation score of the stateful ML model to a predetermined score.

13 . A computer program product comprising at least one non-transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to:

receive historical transaction data, wherein the historical transaction data comprises a plurality of data records of history payment transactions, wherein each data record has a historical timestamp;

generate online simulation data based on the historical transaction data, wherein the one or more instructions that cause the at least one processor to generate the online simulation data, cause the at least one processor to:

modify the historical timestamp of each data record to provide the online simulation data, wherein, the one or more instructions that cause the at least one processor to modify the historical timestamp of each data record to provide the online simulation data, cause the at least one processor to:

assign a timestamp to each data insertion action of a plurality of data insertion actions where the timestamp corresponds to a time of a simulated payment transaction of a plurality of simulated payment transactions;

determine a timeline for the plurality of data insertion actions and a plurality of data request actions based on the online simulation data, wherein the timeline represents a simulation of a real-time flow of data that is seen in production environment, wherein the plurality of data request actions comprises a plurality of requests for an inference from a stateful machine learning (ML) model, and wherein, the one or more instructions that cause the at least one processor to determine the timeline for the plurality of data insertion actions and the plurality of data request actions, cause the at least one processor to:

orchestrate the plurality of data insertion actions and the plurality of data request actions in the timeline; and

select the data sources of the plurality of data insertion actions and the plurality of data request actions;

perform a simulation of online activities involving the stateful ML model in an offline environment using the timeline for the plurality of data insertion actions and the plurality of data request actions, wherein the stateful ML model is configured to provide an output, in response to an input that comprises a simulated payment transaction, that includes a prediction of a category of a plurality of categories of transactions to which the simulated payment transaction is assigned, and wherein, the one or more instructions that cause the at least one processor to perform the simulation of online activities involving the stateful ML model in the offline environment using the timeline for the plurality of data insertion actions and the plurality of data request actions, cause the at least one processor to:

begin the simulation of online activities at a time equal to a start time of the timeline;

continually executing one or more instructions for each time step of a plurality of time steps based on the timeline; and

update a state of the stateful ML model after each data request action of the plurality of data request actions during the simulation of online activities; and

validate the stateful ML model based on the simulation of online activities.

14 . The computer program product of claim 13 , wherein the one or more instructions that cause the at least one processor to perform the simulation of online activities involving the stateful ML model, cause the at least one processor to:

perform a feature engineering procedure based on the online simulation data to provide a plurality of features for the simulation of online activities; and

validate the plurality of features for the simulation of online activities based on the historical transaction data.

15 . The computer program product of claim 13 , wherein the one or more instructions that cause the at least one processor to generate the online simulation data, cause the at least one processor to:

perform one or more data conversion operations on the historical transaction data.

16 . The computer program product of claim 13 , wherein the one or more instructions that cause the at least one processor to modify the historical timestamp of each data record, cause the at least one processor to:

replace the historical timestamp of each data record with a relative timestamp.

17 . The computer program product of claim 13 , wherein the one or more instructions that cause the at least one processor to perform the simulation of online activities involving the stateful ML model, cause the at least one processor to:

generate a simulation score of the stateful ML model based on the online simulation data; and

wherein the one or more instructions that cause the at least one processor to validate the stateful ML model, cause the at least one processor to:

compare the simulation score of the stateful ML model to a predetermined score.