System and method of handling complex experiments in a distributed system
A website building system (WBS) includes an experiment management system to provide experiment management services to at least two tenancies where a single tenancy conducts at least one experiment independently without interfering with experiments conducted by other tenancies, the experiment management system includes an experiment creator to receive an experiment specification with source material from an external source and to create an experiment definition based on inspection and analysis of the source material; an experiment integrator to integrate the experiment definition as an experiment into the experiment management system; an experiment configurer to configure a variant and population selection criteria for the experiment; and an experiment manager to manage experiment conflicts between concurrently running experiments for the at least two tenancies at runtime according to the variant and population selection.
1. A website building system (WBS), said WBS comprising:
at least one processor;
an experiment management system running on said at least one processor to provide experiment management services to at least two tenancies, wherein a single tenancy conducts at least one experiment independently without interfering with experiments conducted by other tenancies, said experiment management system comprising:
an experiment creator to receive an experiment specification with source material from an external source and to create an experiment definition based on inspection and analysis of said source material;
an experiment integrator to integrate said experiment definition as an experiment into said experiment management system;
an experiment configurer to configure a variant and population selection criteria for said experiment; and
an experiment manager to manage experiment conflicts between concurrently running experiments for said at least two tenancies at runtime according to said variant and population selection.
2. The WBS according to claim 1 wherein said at least two tenancies comprise of: a vendor of said WBS, at least one designer using said WBS to create websites, a vendor of a third party application and different user classes of said WBS.
3. The WBS according to claim 2 wherein said experiment management system provides separate access, business intelligence, conflict checking and related processing to said WBS for said at least two tenancies.
4. The WBS according to claim 1 and wherein said experiment manager comprises:
an experiment coordinator to coordinate the flow of multiple concurrently running experiments during runtime;
an experiment selector to perform experiment selection for a given tenancy according to a given target population criteria;
an active conflict resolver to detect and resolve conflicts and dependencies between features and components in concurrently running experiments; and
an experiment implementer to implement said experiment selection after said conflicts and dependencies have been resolved by said active conflict resolver.
5. The WBS according to claim 4 and wherein said active conflict resolver resolves conflicts between said experiment and said concurrently running experiments using experiment separation.
6. The WBS according to claim 4 and wherein said conflict resolver checks for said conflicts within at least one of: a definition and a specification of said experiment and a definition and specification of said concurrently running experiments.
7. The WBS according to claim 5 wherein said conflicts are at least one of: dependency, ambiguity, consistency, and collisions between said concurrently running experiments.
8. The WBS according to claim 4 wherein said active conflict resolver resolves said conflicts and said dependencies using an experiment priority calculated according to at least one of: a priority assigned by a developer, a priority assigned by said experiment manager, scope of said concurrently running experiments and running time of said concurrently running experiments.
9. The WBS according to claim 1 wherein said source material is at least one of: history of said experiment, changes to said experiment, changes to said target population criteria, designer parameters, and designer session parameters.
10. The WBS according to claim 1 and wherein said experiment is at least one of: an A/B test experiment, a multivariate experiment, a feature toggle experiment, a system upgrade and a marketing campaign.
11. A method for a website building system (WBS), said method comprising:
providing experiment management services to at least two tenancies, wherein a single tenancy conducts at least one experiment independently without interfering with experiments conducted by other tenancies, said providing experiment management comprising:
receiving an experiment specification with source material from an external source;
creating an experiment definition based on inspection and analysis of said source material;
integrating said experiment definition as an experiment into said providing experiment management services;
configuring a variant and population selection criteria for said experiment; and
managing experiment conflicts between concurrently running experiments for said at least two tenancies at runtime according to said variant and population selection.
12. The method according to claim 11 wherein said at least two tenancies comprise of: a vendor of said WBS, at least one designer using said WBS to create websites, a vendor of a third party application and different user classes of said WBS.
13. The method according to claim 12 wherein said providing experiment management services provides separate access, business intelligence, conflict checking and related processing to said WBS for said at least two tenancies.
14. The method according to claim 11 and wherein said managing experiment conflicts comprises:
coordinating the flow of multiple concurrently running experiments during runtime;
performing experiment selection for a given tenancy according to a given target population criteria;
detecting and resolving conflicts and dependencies between features and components in concurrently running experiments; and
implementing said performing experiment selection after said conflicts and dependencies have been resolved by said active conflict resolver.
15. The method according to claim 14 and wherein said active conflict resolver resolves conflicts between said experiment and said concurrently running experiments using experiment separation.
16. The method according to claim 14 and wherein said detecting and resolving conflicts checks for said conflicts within at least one of: a definition and a specification of said experiment and a definition and specification of said concurrently running experiments.
17. The method according to claim 15 wherein said conflicts are at least one of: dependency, ambiguity, consistency, and collisions between said concurrently running experiments.
18. The method according to claim 14 wherein said detecting and resolving conflicts resolves said conflicts and said dependencies using an experiment priority calculated according to at least one of: a priority assigned by a developer, a priority assigned by said managing experiment conflicts, scope of said concurrently running experiments and running time of said concurrently running experiments.
19. The method according to claim 11 wherein said source material is at least one of: history of said experiment, changes to said experiment, changes to said target population criteria, designer parameters, and designer session parameters.
20. The method according to claim 11 and wherein said experiment is at least one of: an A/B test experiment, a multivariate experiment, a feature toggle experiment, a system upgrade and a marketing campaign.