Metrics framework for randomized experiments
In an example embodiment, input data for an experiment is automatically formatted and organized using a framework that presents a graphical user interface that is simple and flexible. The graphical user interface allows the user to make certain selections without allowing the user to make any changes that would cause the results of the experiment to be negatively affected. The framework also enforces temporal guarantees. Specifically, the framework enforces that an outcome being used to calculate a metric be measured after an exposure related to the outcome has occurred.
1. A method for improving a protocol for configuring a model, the method comprising:
generating, by a processor, a graphical user interface;
receiving, by the processor at the graphical user interface, a plurality of user inputs for configuring a plurality of parameters of a network operation evaluation protocol;
responsive to receiving the plurality of user inputs at the graphical user interface:
obtaining, by the processor, data from a database, the data including a plurality of events each corresponding to a different network operation authorization;
identifying, by the processor, an exposure event from the plurality of events, the exposure event having a corresponding token and timestamp, the timestamp of the exposure event indicating a first time at which the corresponding token was exposed to a control or treatment of the network operation evaluation protocol;
identifying, by the processor and from the plurality of events, one or more potential outcome events having the corresponding token;
dynamically determining, by the processor, a horizon time window based on a user inputting the plurality of user inputs to generate a set of the one or more potential outcome events that each correspond to a timestamp that is within the horizon time window having a defined end time and beginning with the timestamp of the exposure event having the corresponding token, wherein the processor iteratively generates a set of additional one or more potential outcome events within the horizon time window and having at least a second token;
labeling, by the processor, each potential outcome event of the set of potential outcome events as an actual outcome event;
assigning, by the processor, the exposure event having the corresponding token and one or more actual outcome events having the corresponding token to either a control group or a treatment group of the network operation evaluation protocol; and
executing, by the processor, one or more functions of the network operation evaluation protocol that map to the plurality of user inputs in a function mapping database, wherein the one or more functions are executed based on indications of successful or unsuccessful network operation authorizations of the one or more tokens corresponding to the one or more actual outcome events for the treatment group; and
responsive to the network operation evaluation protocol identifying an increased accuracy in evaluating an updated trained network operation authorization model over a previous network operation authorization model, deploying, by the processor, the updated trained network operation authorization model for a new event.
2. The method of claim 1 , wherein the graphical user interface graphically depicts a difference between results of the network operation protocol for the treatment group and results of the network operation protocol for the control group.
3. The method of claim 1 , wherein the one or more functions include applying a machine learned model trained by a machine learning algorithm to learn weights for features in the data.
4. The method of claim 1 , wherein the network operation evaluation protocol is defined in a configuration file.
5. The method of claim 4 , wherein the configuration file identifies one or more metrics to be measured or calculated during the network operation evaluation protocol.
6. The method of claim 5 , further comprising obtaining the one or more functions from a function library based on a mapping between the one or more metrics and the one or more functions.
7. The method of claim 1 , wherein the one or more functions include a map function and a reducer function, the map function identifying a particular value or property of an individual event and the reducer function aggregating the particular value or property over a plurality of events.
8. A system comprising:
a network;
one or more processors; and
a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the at least one processor to perform operations comprising:
generating a graphical user interface;
receiving, at the graphical user interface, a plurality of user inputs for configuring a plurality of parameters of a network operation evaluation protocol;
responsive to receiving the plurality of user inputs at the graphical user interface:
obtaining data from a database, the data including a plurality of events each corresponding to a different network operation authorization;
identifying an exposure event from the plurality of events, the exposure event having a corresponding token and timestamp, the timestamp of the exposure event indicating a first time at which the corresponding token was exposed to a control or treatment of the network operation evaluation protocol;
identifying, from the plurality of events, one or more potential outcome events having the corresponding token;
dynamically determining a horizon time window based on a user inputting the plurality of user inputs to generate a set of the one or more potential outcome events that each correspond to a timestamp that is within the horizon time window having a defined end time and beginning with the timestamp of the exposure event having the corresponding token, wherein the at least one processor iteratively generates a set of additional one or more potential outcome events within the horizon time window and having at least a second token;
labeling each potential outcome event of the set of potential outcome events as an actual outcome event;
assigning the exposure event having the corresponding token and one or more actual outcome events having the corresponding token to either a control group or a treatment group of the network operation evaluation protocol; and
executing one or more functions of the network operation evaluation protocol that map to the plurality of user inputs in a function mapping database, wherein the one or more functions are executed based on indications of successful or unsuccessful network operation authorizations of the one or more tokens corresponding to the one or more actual outcome events for the treatment group; and
responsive to the network operation evaluation protocol identifying an increased accuracy in evaluating an updated trained network operation authorization model over a previous network operation authorization model, deploying the updated trained network operation authorization model for a new event.
9. The system of claim 8 , wherein the graphical user interface graphically depicts a difference between results of the network operation evaluation protocol for the treatment group and results of the network operation evaluation protocol for the control group.
10. The system of claim 8 , wherein the one or more functions include applying a machine learned model trained by a machine learning algorithm to learn weights for features in the data.
11. The system of claim 8 , wherein the network operation evaluation protocol is defined in a configuration file.
12. The system of claim 11 , wherein the configuration file identifies one or more metrics to be measured or calculated during the network operation evaluation protocol.
13. The system of claim 12 , further comprising obtaining the one or more functions from a function library based on a mapping between the one or more metrics and the one or more functions.
14. The system of claim 12 , wherein the one or more functions include a map function and a reducer function, the map function identifying a particular value or property of an individual event and the reducer function aggregating the particular value or property over a plurality of events.
15. A non-transitory machine-readable medium comprising instructions which, when read by a machine, cause the machine to perform operations comprising:
generating a graphical user interface;
receiving, at the graphical user interface, a plurality of user inputs at the user interface for configuring a plurality of parameters of a network operation evaluation protocol;
responsive to receiving the plurality of user inputs at the graphical user interface:
obtaining data from a database, the data including a plurality of events each corresponding to a different network operation authorization;
identifying an exposure event from the plurality of events, the exposure event having a corresponding token and timestamp, the timestamp of the exposure event indicating a first time at which the corresponding token was exposed to a control or treatment of the network operation evaluation protocol;
identifying, from the plurality of events, one or more potential outcome events having the corresponding token;
dynamically determining a horizon time window based on a user inputting the plurality of user inputs to generate a set of the one or more potential outcome events that each correspond to that is within the horizon time window having a defined end time and beginning with the timestamp of the exposure event having the corresponding token, wherein the processor iteratively generates a set of additional one or more potential outcome events within the horizon time window and having at least a second token;
labeling each potential outcome event of the set of potential outcome events as an actual outcome event;
assigning the exposure event having the corresponding token and one or more actual outcome events having the corresponding token to either a control group or a treatment group of the network operation evaluation protocol; and
executing one or more functions of the network operation evaluation protocol that map to the plurality of user inputs in a function mapping database, wherein the one or more functions are executed based on indications of successful or unsuccessful network operation authorizations of the one or more tokens corresponding to the one or more actual outcome events for the treatment group; and
responsive to the network operation evaluation protocol identifying an increased accuracy in evaluating an updated trained network operation authorization model over a previous network operation authorization model, deploying the updated trained network operation authorization model for a new event.