System and method of handling complex experiments in a distributed system

1. A system, said system comprising:

at least one processor; and

a website building system (WBS) running on said at least one processor to enable website designers to build and host websites for their end users, said WBS comprising:

an experiment manager running on said at least one processor, said experiment manager to manage multiple concurrent experiments at runtime of said WBS, said concurrent experiments to test at least features, components or system updates for said WBS, wherein said experiment manager at least selects a target population of website designers for an experiment, handles conflict resolution between said experiment and at least one other concurrent experiment, and collects experiment data; and

an experiment analyzer to analyze said experiment data during said runtime and to update said experiment manager accordingly.

2. The system according to claim 1 and wherein said experiment manager comprises:

an experiment coordinator to coordinate the flow of said multiple concurrent experiments during said runtime;

an experiment selector to perform experiment selection for a given website designer according to a given target population criteria;

an active conflict resolver to detect and resolve conflicts and dependencies between features and components in said experiment and in at least one other experiment running concurrently within said WBS; and

an experiment implementer to implement said experiment selection after said conflicts and dependencies have been resolved by said active conflict resolver.

3. The system according to claim 2 and also comprising:

an experiment terminator to terminate an experiment according to at least one of: said results from said experiment analyzer and a manual override;

an experiment overrider to override said experiment selection according to interaction from at least one developer of said WBS;

a data collector to collect said experiment data from said multiple concurrent experiments for further analysis by said experiment analyzer;

a baseline coordinator to update an experiment baseline according to analyzed results from said experiment analyzer; and

a variant modifier to recalculate and modify a variant selection during runtime.

4. The system according to claim 1 and wherein said experiment further includes modifications to source code, compiled code, data and WBS structures.

5. The system according to claim 1 and wherein said experiment is at least one of: an A/B test experiment, a multivariate experiment, a feature toggle experiment, a system upgrade and a marketing campaign.

6. The system according to claim 2 and wherein said active conflict resolver resolves conflicts between said experiment and said at least one other experiment using experiment separation.

7. The system according to claim 2 and wherein said experiment implementer implements at least one of: interpreted languages, compiled languages, data structures and bytecodes.

8. The system according to claim 3 and wherein said variant selection is according to at least one of: a regular user ID of a designer, an anonymous user ID, a site ID and a document ID.

9. The system according to claim 8 wherein said experiment analyzer correlates at least one of: said regular user ID of a designer, said anonymous user ID, said site ID and said document ID to create merged information reports.

10. The system according to claim 3 wherein said experiment data is at least one of: history of said experiment, changes to said experiment, changes to said target population criteria, designer parameters, and designer session parameters.

11. The system according to claim 2 wherein said active conflict resolver resolves said conflicts and said dependencies using an experiment priority calculated according to at least one of: a priority assigned by a developer, a priority assigned by said experiment manager, the scope of said concurrent experiments and the running time of said concurrent experiments.

12. A method; said method comprising:

for a website building system (WBS) enabling website designers to build and host websites for their end users, managing multiple concurrent experiments at runtime of said WBS, said concurrent experiments to test at least features, components or system updates for said WBS, wherein said managing comprises at least selecting a target population of website designers for an experiment, handling conflict resolution between said experiment and at least one other concurrent experiment, and collecting experiment data; and

analyzing said experiment data during said runtime and updating said managing accordingly.

13. The method according to claim 12 and wherein said managing further comprises:

coordinating the flow of said multiple concurrent experiments during said runtime;

performing experiment selection for a given website designer according to a given target population criteria;

detecting and resolving conflicts and dependencies between features and components in said experiment and at least one other experiment running concurrently within said WBS; and

implementing said experiment selection after said conflicts and dependencies have been resolved by said detecting and resolving conflicts.

14. The method according to claim 13 and also comprising:

terminating an experiment according to at least one of: said results from said analyzing and a manual override;

overriding said experiment selection according to interaction from at least one developer of said WBS;

collecting said experiment data from said multiple concurrent experiments for further analysis by said analyzing;

updating an experiment baseline according to analyzed results from said analyzing; and

recalculating and modifying a variant selection during runtime.

15. The method according to claim 12 and wherein said experiment further includes modifications to source code, compiled code, data and WBS structures.

16. The method according to claim 12 and wherein said experiment is at least one of: an A/B test experiment, a multivariate experiment, a feature toggle experiment, a system upgrade and a marketing campaign.

17. The method according to claim 13 and wherein said detecting and resolving conflicts and dependencies resolves conflicts between said experiment and said at least one other experiment using experiment separation.

18. The method according to claim 13 and wherein said implementing implements at least one of: interpreted languages, compiled languages, data structures and bytecodes.

19. The method according to claim 14 and wherein said variant selection is according to at least one of: a regular user ID of a designer, an anonymous user ID, a site ID and a document ID.

20. The method according to claim 19 wherein said analyzing correlates at least one of: said regular user ID of a designer, said anonymous user ID, said site ID and said document ID to create merged information reports.

21. The method according to claim 14 wherein said experiment data is at least one of: history of said experiment, changes to said experiment, changes to said target population criteria, designer parameters, and designer session parameters.

22. The method according to claim 13 wherein said detecting and resolving conflicts resolves said conflicts and said dependencies using an experiment priority calculated according to at least one of: a priority assigned by a developer, a priority assigned by said managing, the scope of said concurrent experiments and the running time of said concurrent experiments.