Comparing the performance of multiple application versions
Comparing the performance of multiple versions or branches/paths of an application (e.g., a web service or application) may be conducted within a suitable computing environment. Such an environment may be virtual in nature, cloud-based, or server-based, and is hosted with tools for simultaneously (or nearly simultaneously) executing multiple containers or other code collections with the same or similar operating conditions (e.g., network congestion, resource contention, memory management schemes). By arranging the performance test of different application versions in different sequences executed in parallel in separate containers, fair comparisons of the tested applications will be obtained. Testing sequences may be executed multiple times, and metrics are collected during each execution. Afterward, the results for each metric for each code version are aggregated and displayed to indicate their relative performance quantitatively and/or qualitatively.
1. A method of comparing performances of multiple versions of a set of program code, the method comprising:
initiating multiple simultaneous executions of the multiple versions of the set of program code by, for each of multiple iterations:
configuring multiple separate service containers to sequentially operate the multiple versions of the set of program code in different orders; and
simultaneously executing the multiple separate service containers;
during the multiple simultaneous executions, collecting a plurality of metrics reflecting performance of the multiple versions of the set of program code;
for each version of the set of program code, aggregating the plurality of metrics across the multiple simultaneous executions; and
automatically displaying results that identify, for each metric, quantitative results for each of the multiple versions of the set of program code and a qualitative indication regarding relative performances of the multiple versions of the set of program code.
2. The method of claim 1 , wherein the metrics reflect resource usage caused by execution of the multiple versions of the set of program code.
3. The method of claim 1 , wherein each service container comprises a separate lambda function within AWS Lambda™.
4. The method of claim 3 , wherein the multiple versions of the set of program code are executed by Google Lighthouse™ within the separate lambda functions.
5. The method of claim 1 , wherein:
the multiple versions of the set of program code include two versions of the set of program code; and
said initiating multiple simultaneous executions of the two versions of the set of program code comprises, for each of multiple iterations:
configuring a first service container to sequentially execute a first version of the set of program code without caching, the first version of the set of program code with caching, a second version of the set of program code without caching, and the second version of the set of program code with caching;
configuring a second service container to sequentially execute the second version of the set of program code without caching, the second version of the set of program code with caching, the first version of the set of program code without caching, and the first version of the set of program code with caching; and
simultaneously executing the first service container and the second service container.
6. The method of claim 5 , wherein:
the first service container is further configured to execute an additional first version of the set of program code prior to the first version of the set of program code without caching;
the second service container is further configured to execute an additional second version of the set of program code prior to the second version of the set of program code without caching; and
metrics collected during execution of the additional first and second versions of the set of program code are discarded.
7. The method of claim 1 , wherein the set of program code is a web-based application.
8. The method of claim 1 , wherein the plurality of metrics includes one or more of:
time to first page view;
time to interactive;
duration of time of main-thread work;
duration of time JavaScript® executed;
number of JavaScript® modules loaded;
number of JavaScript® bundles loaded;
number of API (Application Programming Interface) calls;
amount of data transferred via the API calls;
latency of the API calls; and
total amount of data transferred.
9. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method of comparing performances of multiple versions of a set of program code, the method comprising:
initiating multiple simultaneous executions of the multiple versions of the set of program code by, for each of multiple iterations:
configuring multiple separate service containers to sequentially operate the multiple versions of the set of program code in different orders; and
simultaneously executing the multiple separate service containers;
during the multiple simultaneous executions, collecting a plurality of metrics reflecting performance of the multiple versions of the set of program code;
for each version of the set of program code, aggregating the plurality of metrics across the multiple simultaneous executions; and
automatically displaying results that identify, for each metric, quantitative results for each of the multiple versions of the set of program code and a qualitative indication regarding relative performances of the multiple versions of the set of program code.
10. The non-transitory computer readable medium of claim 9 , wherein the metrics reflect resource usage caused by execution of the multiple versions of the set of program code.
11. The non-transitory computer readable medium of claim 9 , wherein:
the multiple versions of the set of program code include two versions of the set of program code; and
said initiating multiple simultaneous executions of the two versions of the set of program code comprises, for each of multiple iterations:
configuring a first service container to sequentially execute a first version of the set of program code without caching, the first version of the set of program code with caching, a second version of the set of program code without caching, and the second version of the set of program code with caching;
configuring a second service container to sequentially execute the second version of the set of program code without caching, the second version of the set of program code with caching, the first version of the set of program code without caching, and the first version of the set of program code with caching; and
simultaneously executing the first service container and the second service container.
12. The non-transitory computer readable medium of claim 11 , wherein:
the first service container is further configured to execute an additional first version of the set of program code prior to the first version of the set of program code without caching;
the second service container is further configured to execute an additional second version of the set of program code prior to the second version of the set of program code without caching; and
metrics collected during execution of the additional first and second versions of the set of program code are discarded.
13. A computer system for comparing performances of multiple versions of a set of program code, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the system to:
initiate multiple simultaneous executions of the multiple versions of the set of program code by, for each of multiple iterations:
configuring multiple separate service containers to sequentially operate the multiple versions of the set of program code in different orders; and
simultaneously executing the multiple separate service containers;
during the multiple simultaneous executions, collect a plurality of metrics reflecting performance of the multiple versions of the set of program code;
for each version of the set of program code, aggregate the plurality of metrics across the multiple simultaneous executions; and
automatically display results that identify, for each metric, quantitative results for each of the multiple versions of the set of program code and a qualitative indication regarding relative performances of the multiple versions of the set of program code.
14. The system of claim 13 , wherein the metrics reflect resource usage caused by execution of the multiple versions of the set of program code.
15. The system of claim 13 , wherein each service container comprises a separate lambda function within AWS Lambda™.
16. The system of claim 15 , wherein the multiple versions of the set of program code are executed by Google Lighthouse™ within the separate lambda functions.
17. The system of claim 13 , wherein:
the multiple versions of the set of program code include two versions of the set of program code; and
said initiating multiple simultaneous executions of the two versions of the set of program code comprises, for each of multiple iterations:
configuring a first service container to sequentially execute a first version of the set of program code without caching, the first version of the set of program code with caching, a second version of the set of program code without caching, and the second version of the set of program code with caching;
configuring a second service container to sequentially execute the second version of the set of program code without caching, the second version of the set of program code with caching, the first version of the set of program code without caching, and the first version of the set of program code with caching; and
simultaneously executing the first service container and the second service container.
18. The system of claim 17 , wherein:
the first service container is further configured to execute an additional first version of the set of program code prior to the first version of the set of program code without caching;
the second service container is further configured to execute an additional second version of the set of program code prior to the second version of the set of program code without caching; and
metrics collected during execution of the additional first and second versions of the set of program code are discarded.
19. The system of claim 13 , wherein the set of program code is a web-based application.
20. The system of claim 13 , wherein the plurality of metrics includes one or more of:
time to first page view;
time to interactive;
duration of time of main-thread work;
duration of time JavaScript® executed;
number of JavaScript® modules loaded;
number of JavaScript® bundles loaded;
number of API (Application Programming Interface) calls;
amount of data transferred via the API calls;
latency of the API calls; and
total amount of data transferred.
21. A method of comparing performances of multiple versions of a set of program code, the method comprising:
obtaining two versions of the set of program code, wherein each version differs from the other version;
for each of multiple iterations:
instantiating two containers;
configuring each container to execute the two versions of the set of program code in a different sequence;
simultaneously executing the containers; and
measuring usage of a plurality of resources during each execution of a version of the set of program code in each of the containers;
for each version of the set of program code, aggregating the resource usage measurements across all iterations and all container executions; and
outputting the aggregated resource usage measurements to include quantitative values for each resource for each version of the set of program code and qualitative indications of which of the multiple versions of the set of program code used each resource more efficiently;
wherein only a version of the set of program code that used resources more efficiently than the other version of the set of program code is released for execution in a production environment.
22. The method of claim 21 , further comprising:
modifying an online monitoring tool configured to monitor a single web page to cause the tool to perform said simultaneous execution of the containers for the multiple iterations.
23. The method of claim 21 , wherein within each container, each version of the set of program code is executed at least once with inactive caching and at least once with active caching.