Systems and methods for bias bounded sensitivity analysis of synthetic control models
Systems and methods for performing bias bounded sensitivity analysis of synthetic control models are provided. Bias bounds may be determined for a synthetic control model using data from the synthetic control model. If a difference between a synthetic control determined by the synthetic control model and an observed outcome are within the bias bounds, a causal effect determined using the synthetic control model may be untrustworthy. Graphical representations may be presented based, at least in part, on the observed outcome, the synthetic control, and the bias bounds.
1 . A system for determining a causal effect using a synthetic control model, the system comprising:
one or more processors; and
one or more computer-readable storage devices storing data instructions that, when executed by the one or more processors, cause the system to:
receive a selection of one or more donors, each donor in the one or more donors associated with timeseries data comprising data before an intervention and data after the intervention;
train the synthetic control model on the timeseries data associated with the one or more donors, the synthetic control model including one or more learned weights, each learned weight associated with a donor in the one or more donors;
compute a bias for a synthetic control from the synthetic control model, wherein computing the bias includes: selecting a weight from the one or more learned weights; for each donor in the one or more donors, computing a difference between an average of the timeseries data before the intervention and an average of the timeseries data after the intervention; selecting a difference from the computed differences; and computing the bias based on a product of a number of donors in the synthetic control model, the selected weight, and the selected difference; and
cause a visual output device of a computing device to present a graphical representation based, at least in part, on an observed outcome, the synthetic control, and the bias.
2 . The system of claim 1 , wherein the selected weight has a maximum absolute value from among the one or more learned weights.
3 . The system of claim 1 , wherein the selected difference has a maximum absolute value from among the computed differences.
4 . The system of claim 1 , wherein the graphical representation includes a difference between the observed outcome and the synthetic control.
5 . The system of claim 1 , wherein the graphical representation includes a cumulative difference between the observed outcome and the synthetic control.
6 . The system of claim 1 , wherein the graphical representation includes a table.
7 . The system of claim 1 , wherein the graphical representation includes a line chart.
8 . The system of claim 1 , wherein computing the bias is based on performing sensitivity analysis on the synthetic control model to determine trustworthiness of the causal effect.
9 . The system of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the system to:
compute a difference between the observed outcome and the synthetic control;
determine that the difference is within bounds of the bias; and
in response to determining that the difference is within the bounds of the bias, retrain the synthetic control model by selecting one or more further donors, and training the synthetic control model on further timeseries data associated with the one or more further donors.
10 . The system of claim 1 , wherein the donors comprise donor configurations of a media playback system, wherein the intervention involves adding a configuration with a new feature to the media playback system, and wherein the causal effect is of the new feature in relation to the media playback system.
11 . A method for determining a causal effect using a synthetic control model, the method comprising:
receiving a selection of a first one or more donors, each donor in the first one or more donors associated with timeseries data comprising data before an intervention and data after the intervention;
training the synthetic control model on the timeseries data associated with the first one or more donors, the synthetic control model including one or more learned weights, each learned weight associated with a donor in the first one or more donors;
computing bias bounds for a synthetic control from the synthetic control model, wherein computing the bias bounds includes: selecting a weight from the one or more learned weights; for each donor in the first one or more donors, computing a difference between an average of the timeseries data before the intervention and an average of the timeseries data after the intervention;
selecting a difference from the computed differences; and computing the bias bounds based on a product of a number of donors in the synthetic control model, the selected weight, and the selected difference; and
causing a visual output device of a computing device to present a graphical representation based, at least in part, on an observed outcome, the synthetic control, and the bias bounds.
12 . The method of claim 11 , wherein the selected weight has a maximum absolute value from among the one or more learned weights.
13 . The method of claim 11 , wherein the selected difference has a maximum absolute value from among the computed differences.
14 . The method of claim 11 , further comprising:
after causing the visual output device of the computing device to present the graphical representation, receiving a selection of a second one or more donors associated with timeseries data before the intervention and after the intervention and retraining the synthetic control model on the timeseries data associated with the second one or more donors.
15 . The method of claim 14 , wherein the second one or more donors includes at least one donor from the first one or more donors.
16 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:
receive a selection of one or more donors, each donor in the one or more donors associated with timeseries data comprising data before an intervention and data after the intervention;
train a synthetic control model on the timeseries data associated with the one or more donors, the synthetic control model including one or more learned weights, each learned weight associated with a donor in the one or more donors;
compute bias bounds for a synthetic control from the synthetic control model, wherein computing the bias bounds includes: selecting a weight from the one or more learned weights; for each donor in the one or more donors, computing a difference between an average of the timeseries data before the intervention and an average of the timeseries data after the intervention; selecting a difference from the computed differences; and computing the bias bounds based on a product of a number of donors in the synthetic control model, the selected weight, and the selected difference; and
cause a visual output device of a computing device to present a graphical representation based, at least in part, on an observed outcome, the synthetic control, and the bias bounds.
17 . The non-transitory computer-readable medium of claim 16 , wherein the selected weight has a maximum absolute value from among the one or more learned weights.
18 . The non-transitory computer-readable medium of claim 16 , wherein the selected difference has a maximum absolute value from among the computed differences.
19 . The non-transitory computer-readable medium of claim 16 , wherein the graphical representation includes a difference between the observed outcome and the synthetic control.
20 . The non-transitory computer-readable medium of claim 16 , wherein the graphical representation includes a cumulative difference between the observed outcome and the synthetic control.