Conducting performance snapshots during test and using feedback to control test based on customer experience parameters
The technology disclosed enables understanding the user experience of accessing a web page under high loads. A testing system generates a simulated load by retrieving and loading a single web object. A performance snapshot is taken of accessing an entire web page from the server under load. The performance snapshot may be performed by emulating a browser accessing a web page's URL, the web page comprising multiple objects that are independently retrieved and loaded. The simulated load is configured with a number of users per region of the world where the user load will originate, and a single object from the web page to retrieve. Performance data such as response time for the single object retrieved, number of hits per second, number of timeouts per sec, and errors per second may be recorded and reported. An optimal number of users may be determined to achieve a target user experience goal.
1. A method of understanding user experience under high loads and diverse regional traffic mix, the method including:
initiating a test using a hybrid test system including first simulated load generators and second web browser emulators that send a multitude of object requests related to a web page to a system under test from one or more user selected regions;
wherein the first simulated load generators:
generate object requests for at least one object on the web page, thereby simulating an object request load resulting from at least dozens of client-requestors; and
report at least object response time and error rates responsive to the object requests; and
wherein the second web browser emulators:
emulate a sequence of requests for multiple separately loadable object components of the web page from a plurality of the second web browser emulators; and
report at least initial response time, object load time, and returned errors for the separately loadable components;
receiving reports of test result statistics from the first simulated load generators and the second web browser emulators; and
reporting at least some of the test result statistics for display to a user.
2. The method of claim 1 , wherein the reporting includes reporting a plurality of graphs depicting at least a number of simulated users, response times from the second web browser emulators, hit rates from the first simulated load generators, and error rates from the first simulated load generators.
3. The method of claim 1 , further including reporting test result statistics from at least one of the second web browser emulators organized by requested object component and formatted to include a cascaded bar chart of initial response times and object load times for at least some of the separately loadable object components.
4. The method of claim 1 , further including reporting test result statistics averaged from a plurality of the second web browser emulators operating on hardware in one of the user selected regions, organized by requested object component, and formatted to include a cascaded bar chart of object response times for at least some of the separately loadable object components.
5. The method of claim 1 , further including reporting test result statistics from a plurality of the second web browser emulators operating on hardware in one of the user selected regions, organized by requested object component, and formatted to include a cascaded box-and-whisker chart of object response time distributions for at least some of the separately loadable object components.
6. The method of claim 1 , further including reporting in multiple cascaded bar charts timing derived from reports of a plurality of the second web browsers under different system loads, wherein each cascaded bar chart illustrates the response times for at least some of the separately loadable object components.
7. The method of claim 3 , further including:
receiving data indicating user selection of a bar in the cascaded bar chart; and
responsive to the user selection, transmitting data representing an approximate visual appearance to a user of the web page components at a time after start of loading the web page corresponding to the user selected bar.
8. The method of claim 3 , wherein the cascade bar chart further includes milestone indicators of events that change a visual appearance with a browser of the web page.
9. The method of claim 1 , further including:
receiving data that indicates a browser to emulate from among a plurality of browsers that have different operating characteristics; and
using a web browser emulator corresponding to the indicated browser to emulate.
10. The method of claim 2 , further including in the hit rates and the error rates results from both the first simulated load generators and the second web browser emulators.
11. The method of claim 2 , further including filtering the graphs by user selected region.
12. The method of claim 2 , further including:
using the hybrid test system from multiple user selected regions with both the first simulated load generators and the second web browser emulators operating on hardware in the multiple user selected regions; and
varying a mix of traffic from the multiple regions during the test.
13. A computer-implemented system that implements a method of understanding user experience under high loads and diverse regional traffic mix, the implementation including:
a processor and a memory coupled to the processor, the memory storing computer instructions that implement a method, including:
initiating a test using a hybrid test system including first simulated load generators and second web browser emulators that send a multitude of object requests related to a web page to a system under test from one or more user selected regions;
wherein the first simulated load generators:
generate object requests for at least one object on the web page, thereby simulating an object request load resulting from at least dozens of client-requestors; and
report at least object response time and error rates responsive to the object requests; and
wherein the second web browser emulators:
emulate a sequence of requests for multiple separately loadable object components of the web page from a plurality of the second web browser emulators; and
report at least initial response time, object load time, and returned errors for the separately loadable components;
receiving reports of test result statistics from the first simulated load generators and the second web browser emulators; and
reporting at least some of the test result statistics for display to a user.
14. The computer-implemented system of claim 13 , wherein the reporting further includes reporting in a plurality of graphs at least a number of simulated users, response times from the second web browser emulators, hit rates from the first simulated load generators, and error rates from the first simulated load generators.
15. The computer-implemented system of claim 13 , wherein the implementation further includes reporting test result statistics averaged from a plurality of the second web browser emulators operating on hardware in one of the user selected regions, organized by requested object component, and formatted to include a cascaded bar chart of object response times for at least some of the separately loadable object components.
16. The computer-implemented system of claim 13 , wherein the implementation further includes reporting test result statistics from a plurality of the second web browser emulators operating on hardware in one of the user selected regions, organized by requested object component, and formatted to include a cascaded box-and-whisker chart of object response time distributions for at least some of the separately loadable object components.
17. The computer-implemented system of claim 13 , wherein the implementation further includes reporting in multiple cascaded bar charts timing derived from reports of a plurality of the second web browsers under different system loads, wherein each cascaded bar chart illustrates the response times for at least some of the separately loadable object components.
18. The computer-implemented system of claim 13 , wherein the implementation further includes:
using the hybrid test system from multiple user selected regions with both the first simulated load generators and the second web browser emulators operating on hardware in the multiple user selected regions; and
varying a mix of traffic from the multiple regions during the test.
19. A non-transitory computer readable medium that stores computer instructions that, when run on hardware, implement understanding user experience under high loads and diverse regional traffic mix, the implementation including:
initiating a test using a hybrid test system including first simulated load generators and second web browser emulators that send a multitude of object requests related to a web page to a system under test from one or more user selected regions;
wherein the first simulated load generators:
generate object requests for at least one object on the web page, thereby simulating an object request load resulting from at least dozens of client-requestors; and
report at least object response time and error rates responsive to the object requests; and
wherein the second web browser emulators:
emulate a sequence of requests for multiple separately loadable object components of the web page from a plurality of the second web browser emulators; and
report at least initial response time, object load time, and returned errors for the separately loadable components;
receiving reports of test result statistics from the first simulated load generators and the second web browser emulators; and
reporting at least some of the test result statistics for display to a user.
20. The non-transitory computer readable medium of claim 19 , the implementation further including reporting in a plurality of graphs at least a number of simulated users, response times from the second web browser emulators, hit rates from the first simulated load generators, and error rates from the first simulated load generators.
21. The non-transitory computer readable medium of claim 19 , the implementation further including reporting test result statistics from at least one of the second web browser emulators organized by requested object component and formatted to include a cascaded bar chart of object response times for at least some of the separately loadable object components.
22. The non-transitory computer readable medium of claim 19 , the implementation further including:
receiving data indicating user selection of a bar in the cascaded bar chart; and
responsive to the user selection, transmitting data representing an approximate visual appearance to a user of the web page components at a time after start of loading the web page corresponding to the user selected bar.
23. The non-transitory computer readable medium of claim 19 , wherein the cascade bar chart further includes milestone indicators of events that change a visual appearance with a browser of the web page.
24. The non-transitory computer readable medium of claim 19 , the implementation further including:
receiving data that indicates a browser to emulate from among a plurality of browsers that have different operating characteristics; and
using a web browser emulator corresponding to the indicated browser to emulate.
25. The non-transitory computer readable medium of claim 21 , further including in the hit rates and the error rates results from both the first simulated load generators and the second web browser emulators.