Method and system for evaluating service performance perceived by user
Methods and systems for evaluating a service experience performance of a user may be provided. A method of evaluating a service experience performance may include capturing a screen based on a time unit until a landing image according to a scenario of a service is loaded, determining that loading of the landing image is completed based on the captured screen, deriving an amount of loading time until the loading of the landing image is completed, and indexing performance for the scenario of the service based on the amount of loading time and an evaluation reference time.
1. A method of evaluating a service experience performance of a computer device comprising at least one processor, the computer device configured to communicate with a server over a network, the method comprising:
by the at least one processor,
in response to an input of an instruction according to a scenario of a service, capturing a screen based on a time unit until a landing image according to the scenario of the service is loaded on the computer device;
determining that loading of the landing image is completed based on the captured screen;
in response to a result of the determining indicating that loading of the landing image has been completed, deriving an amount of loading time until the loading of the landing image is completed based on a difference between a loading completion time and a scenario start time, the loading completion time being a time point at which the loading of the landing image has been completed;
indexing performance for the scenario of the service based on the amount of loading time and an evaluation reference time over a plurality of times to obtain a plurality of indexed performances;
determining an average score of the plurality of indexed performances;
calculating a variability of the plurality of indexed performances over a plurality of times based on a standard deviation; and
determining a final score for the service by subtracting the standard deviation from the average score or calculating an average between the average score and the standard deviation.
2. The method of claim 1 , wherein
the scenario includes a scenario for a transition from a first screen of the service to a second screen of the service, and
the landing image includes an image corresponding to the second screen.
3. The method of claim 1 , wherein the capturing comprises capturing a displayed screen of an electronic device of a user included in a real user environment.
4. The method of claim 1 , wherein the determining includes determining that the loading of the landing image is completed when the captured screen corresponds to a desired landing image.
5. The method of claim 4 , wherein the deriving comprises determining a difference between a first time at which the screen corresponding to the desired landing image is captured and a second time at which the scenario of the service starts as the amount of loading time.
6. The method of claim 1 , wherein the determining comprises:
evaluating a change between consecutively captured screens; and
determining that the loading of the landing image is completed when the change between the consecutively captured screens is absent for a period of time or more.
7. The method of claim 6 , wherein the deriving comprises determining a difference between a first time at which a first screen among the consecutively captured screens without the change is captured and a second time at which the scenario of the service starts, as the amount of loading time.
8. The method of claim 1 , wherein the indexing comprises indexing the performance for the scenario of the service based on a value acquired by dividing the evaluation reference time by the amount of loading time.
9. The method of claim 1 , wherein the indexing comprises:
indexing the performance for the scenario of the service based on a value acquired by dividing the evaluation reference time by the amount of loading time; and
indexing the performance for the scenario of the service by assigning a highest score to the amount of loading time corresponding to the evaluation reference time or less.
10. The method of claim 1 , wherein the indexing comprises indexing the performance for the scenario of the service by assigning a grade according to a range of a value acquired by dividing the evaluation reference time by the amount of loading time.
11. The method of claim 1 , further comprising:
calculating, by the at least one processor, an average of indexed performances measured over a plurality of times.
12. A non-transitory computer-readable recording medium comprising instructions that, when executed by a processor, cause a computer device to perform the method of claim 1 .
13. A computer device comprising:
at least one processor configured to execute a computer-readable instruction such that the processor is configured to cause the computer device to,
in response to an input of an instruction according to a scenario of a service, capture a screen based on a time unit until a landing image according to the scenario of the service is loaded on the computer device,
determine that loading of the landing image is completed based on the captured screen,
in response to a result of the determining indicating that loading of the landing image has been completed, derive an amount of loading time until the loading of the landing image is completed based on a difference between a loading completion time and a scenario start time, the loading completion time being a time point at which the loading of the landing image has been completed,
index performance for the scenario of the service based on the amount of the loading time and an evaluation reference time over a plurality of times to obtain a plurality of indexed performances,
determine an average score of the plurality of indexed performances;
calculate a variability of the plurality of indexed performances over a plurality of times based on a standard deviation; and
determine a final score for the service by subtracting the standard deviation from the average score or calculating an average between the average score and the standard deviation.
14. The computer device of claim 13 , wherein the at least one processor is further configured to cause the computer device to capture a displayed screen of an electronic device of a user included in a real user environment.
15. The computer device of claim 13 , wherein the at least one processor is further configured to cause the computer device to determine that the loading of the landing image is completed when the captured screen corresponds to a desired landing image.
16. The computer device of claim 13 , wherein the at least one processor is further configured to cause the computer device to,
evaluate a change between consecutively captured screens, and
determine that the loading of the landing image is completed when the change between the consecutively captured screens is absent for a period of time or more.
17. The computer device of claim 13 , wherein the at least one processor is further configured to cause the computer device to index the performance for the scenario of the service based on a value acquired by dividing the evaluation reference time by the amount of the loading time.