Subgroup analysis in A/B testing
Described are techniques for A/B testing including a computer-implemented method of identifying, in an A/B testing database, a set of feature values with a statistically significant difference in A/B testing outcomes above a threshold. The method further includes partitioning records in the A/B testing database into a plurality of population strata according to the set of feature values. The method further includes performing A/B testing, and identifying heterogeneous outcomes of the A/B testing for respective strata of the plurality of population strata.
1. A computer-implemented method comprising:
distributing local versions of a population strata partitioning model to a plurality of known user devices, each known user device of the plurality of known devices comprising a secure hardware-based enclave;
locally training the distributed local versions of the population strata partitioning model within the secure hardware-based enclave of each known user device and using device-specific data, wherein the secure hardware-based enclave is a single, common, continuous security perimeter providing computational security;
returning model parameters from the distributed local versions of the population strata partitioning model to the population strata partitioning model in a data processing system;
aggregating, at the population strata partitioning model, the model parameters;
identifying, in an A/B testing database and using the population strata partitioning model, a set of feature values with a statistically significant difference in A/B testing outcomes above a threshold;
partitioning records in the A/B testing database into a plurality of population strata according to the identified set of feature values;
performing A/B testing on each of the plurality of population strata independently; and
identifying heterogeneous outcomes of the A/B testing for respective strata of the plurality of population strata to enable future A/B tests to be designed for specific population strata.
2. The method of claim 1 , further comprising:
identifying a minority stratum that repeatedly loses to at least one other stratum in the A/B testing;
designing a subsequent A/B test for the minority stratum using stratified randomization of the minority stratum and the at least one other stratum; and
encoding results of the subsequent A/B test in the A/B testing database.
3. The method of claim 1 , further comprising:
matching a new individual to a stratum of the plurality of population strata using the population strata partitioning model having aggregated parameters from the distributed local versions of the population strata partitioning model; and
adding the new individual to the stratum in the A/B testing database.
4. The method of claim 3 , wherein the new individual is matched to one and only one stratum of the plurality of population strata.
5. The method of claim 3 , wherein the new individual is matched to multiple strata of the plurality of population strata, wherein each of the multiple strata includes a metric associated with the match.
6. The method of claim 1 , further comprising:
encoding user behaviors and user conversions of respective users in the plurality of population strata for the A/B testing.
7. The method of claim 6 , further comprising:
automatically triggering a new A/B test in response to a detected divergence in user conversions, wherein the detected divergence comprises a difference above a second threshold between an expected conversion ratio and an observed conversion ratio, wherein the new A/B test is based on the plurality of population strata, the user behaviors, and the user conversions.
8. The method of claim 1 , wherein the method is performed by one or more computers according to A/B testing software that is downloaded to the one or more computers from a remote data processing system.
9. The method of claim 8 , wherein the method further comprises:
metering a usage of the A/B testing software; and
generating an invoice based on metering the usage.
10. A system comprising:
one or more computer readable storage media storing program instructions; and
one or more processors which, in response to executing the program instructions, are configured to perform a method comprising:
distributing local versions of a population strata partitioning model to a plurality of known user devices, each known user device of the plurality of known devices comprising a secure hardware-based enclave;
locally training the distributed local versions of the population strata partitioning model n within the secure hardware-based enclave of each known user device and using device-specific data, wherein the secure hardware-based enclave is a single, common, continuous security perimeter providing computational security;
returning model parameters from the distributed local versions of the population strata partitioning model to the population strata partitioning model in a data processing system;
aggregating, at the population strata partitioning model, the model parameters;
identifying, in an A/B testing database and using the population strata partitioning model, a set of feature values with a statistically significant difference in A/B testing outcomes above a threshold;
partitioning records in the A/B testing database into a plurality of population strata according to the identified set of feature values;
performing A/B testing on each of the plurality of population strata independently; and
identifying heterogeneous outcomes of the A/B testing for respective strata of the plurality of population strata to enable future A/B tests to be designed for specific population strata.
11. The system of claim 10 , the method further comprising:
identifying a minority stratum that repeatedly loses to at least one other stratum in the A/B testing;
designing a subsequent A/B test for the minority stratum using stratified randomization of the minority stratum and the at least one other stratum; and
encoding results of the subsequent A/B test in the A/B testing database.
12. The system of claim 10 , the method further comprising:
matching a new individual to a stratum of the plurality of population strata using the population strata partitioning model having aggregated parameters from the distributed local versions of the population strata partitioning model; and
adding the new individual to the stratum in the A/B testing database.
13. The system of claim 12 , wherein the new individual is matched to one and only one stratum of the plurality of population strata.
14. The system of claim 12 , wherein the new individual is matched to multiple strata of the plurality of population strata, wherein each of the multiple strata includes a metric associated with the match.
15. The system of claim 10 , the method further comprising:
encoding user behaviors and user conversions of respective users in the plurality of population strata for the A/B testing.
16. The system of claim 15 , the method further comprising:
automatically triggering a new A/B test in response to a detected divergence in user conversions, wherein the detected divergence comprises a difference above a second threshold between an expected conversion ratio and an observed conversion ratio, wherein the new A/B test is based on the plurality of population strata, the user behaviors, and the user conversions.
17. A computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions comprising instructions configured to cause one or more processors to perform a method comprising:
distributing local versions of a population strata partitioning model to a plurality of known user devices, each known user device of the plurality of known devices comprising a secure hardware-based enclave;
locally training the distributed local versions of the population strata partitioning model within the secure hardware-based enclave of each known user device and using device-specific data, wherein the secure hardware-based enclave is a single, common, continuous security perimeter providing computational security;
returning model parameters from the distributed local versions of the population strata partitioning model to the population strata partitioning model in a data processing system;
aggregating, at the population strata partitioning model, the model parameters;
identifying, in an A/B testing database and using the population strata partitioning model, a set of feature values with a statistically significant difference in A/B testing outcomes above a threshold;
partitioning records in the A/B testing database into a plurality of population strata according to the identified set of feature values;
performing A/B testing on each of the plurality of population strata independently; and
identifying heterogeneous outcomes of the A/B testing for respective strata of the plurality of population strata to enable future A/B tests to be designed for specific population strata.
18. The computer program product of claim 17 , the method further comprising:
identifying a minority stratum that repeatedly loses to at least one other stratum in the A/B testing;
designing a subsequent A/B test for the minority stratum using stratified randomization of the minority stratum and the at least one other stratum; and
encoding results of the subsequent A/B test in the A/B testing database.
19. The computer program product of claim 17 , the method further comprising:
matching a new individual to a stratum of the plurality of population strata using the population strata partitioning model having aggregated parameters from the distributed local versions of the population strata partitioning model; and
adding the new individual to the stratum in the A/B testing database, wherein the new individual is matched to one selected from a group consisting of: one and only one stratum of the plurality of population strata, and multiple strata of the plurality of population strata, wherein each of the multiple strata includes a metric associated with the match.
20. The computer program product of claim 17 , the method further comprising:
encoding user behaviors and user conversions of respective users in the plurality of population strata for the A/B testing; and
automatically triggering a new A/B test in response to a detected divergence in user conversions, wherein the detected divergence comprises a difference above a second threshold between an expected conversion ratio and an observed conversion ratio, wherein the new A/B test is based on the plurality of population strata, the user behaviors, and the user conversions.