Performance utilities for mobile applications
Techniques for evaluating software systems are provided, including measurement of performance ratings of features of an app-under-test are described. The evaluation may include analysis of a video recording of the user interface of the app-under-test, where the video analysis may include comparing the changes over time of co-located pixel in the video to produce a performance rating, for example a speed index. The results of multiple such tests can be compared by comparing the performance rating produced during execution of different tests.
1. A method for testing software, comprising:
recording of a display on a first mobile device as a first video during a first test including execution of at least one feature of a first version of a first app-under-test;
determining a first performance rating based on analysis of the first video including detection of visual changes over time in the first test from the first video;
recording of the display on a second mobile device as a second video during a second test including execution of the at least one feature of a second version of a second app-under-test;
determining a second performance rating based on analysis of the second video including detection of visual changes over time in the second test from the second video;
comparing the first and second tests based on the first performance rating and the second performance rating;
wherein the first performance rating and the second performance rating are visual speed ratings and the performance rating is visual speed index;
wherein the first and second performance ratings are based on an estimate of human visual perception of changes over time in the video being analyzed;
wherein list of tests with speed index ratings comprising many rows, each row represents one app test process including: a video identifier indicating video filename of the video captured during the test, the date and time of the test, the name of the test was run which including the type of mobile device and mobile device operating system used in that test, speed index rating, and a last visual change number resulting from analysis of the video; and
evaluating including the analysis of the video recording of a user interface of the first app-under-test and second app-under-test produce multiple different metrics from the video, the different metrics comprising:
evaluate area under a curve plotting the change of all pixels in the video of the user interface and weight certain pixels at certain times differently wherein the weight certain pixels at the certain times use knowledge of a particular user interface to weight a portion the video, such that the upper left-hand quarter of a screen over the remainder of the video,
a heat map indicated which pixels are most interesting to a user, the heat map derived from a click map that measures which areas of the user interface are most frequently clicked on, or derived from the portions of the user interface a user scrolls, or with eye tracking ability derived from the portions of a user interface that a user looks at most and wherein heat map inferred from the colors of pixels and render the pixels considered to have a higher user experience metric, and
a portion of a screen that provides buttons for user input is weighted higher than portions of a screen that provide design elements that provide no functionality.
2. The method of claim 1 , wherein further comprising: generating an indication of a fault based on the comparing.
3. The method of claim 1 , wherein the first mobile device includes a first operating system, the first operating system includes test instrumentation, and the first app-under-test is not modified for the first test; and further comprising:
sending an instruction to the test instrumentation on the first device to launch the first version of the app-under-test and to navigate to the at least one feature of the first app-under-test.
4. The method of claim 1 , wherein:
the first and second mobile devices are the same device, the first and second apps-under-test are the same, and the first and second versions of apps-under-test are not the same.
5. The method of claim 1 , wherein:
the first and second mobile devices are the same model of device and not the same device, and the first and second versions of the app-under-test are the same versions.
6. The method of claim 1 , wherein:
the first and second mobile devices are the same device, and first and second apps-under-test are not the same, and the at least one feature of a first version of a first app-under-test is the same as the at least one feature of a second version of a second app-under-test.
7. The method of claim 1 , further comprising:
initiating the first test by providing to the first mobile device a uniform resource locator (URL) indicating the first version of the app-under-test and further indicating the at least one feature of the first app-under-test; and
initiating the second test by providing to the second mobile device a URL indicating the second version of the app-under-test and further indicating the at least one feature of the second app-under-test.
8. A system for testing an app, comprising:
a test server including a test script;
a video analysis server;
a first mobile device including test software and a first version of an app-under-test;
a second mobile device includes test software and a second version of the app-under-test first test software; and
wherein the test script causes:
recording of a display on the first mobile device as a first video during a first test including execution of at least one feature of the first version of the app-under-test;
determining a first performance rating based on analysis of the first video including detection of visual changes over time in the first test from the first video by the video analysis server;
recording of the display on the first second mobile device as a second video during a second test including execution of the at least one feature of the second version of the app-under-test;
determining a second performance rating based on analysis of the second video including detection of visual changes over time in the second test from the second video by the video analysis server; and
comparing the first and second versions of the app-under-test based on the first performance rating and the second performance rating;
wherein the first performance rating and the second performance rating are visual speed ratings and the performance rating is visual speed index;
wherein the first and second performance ratings are based on an estimate of human visual perception of changes over time in the video being analyzed;
wherein list of tests with speed index ratings comprising many rows, each row represents one app test process including: a video identifier indicating video filename of the video captured during the test, the date and time of the test, the name of the test was run which including the type of mobile device and mobile device operating system used in that test, speed index rating, and a last visual change number resulting from analysis of the video; and
evaluating including the analysis of the video recording of a user interface of the first app-under-test and second app-under-test produce multiple different metrics from the video, the different metrics comprising:
evaluate area under a curve plotting the change of all pixels in the video of the user interface and weight certain pixels at certain times differently wherein the weight certain pixels at the certain times use knowledge of a particular user interface to weight a portion the video, such that the upper left-hand quarter of the screen over the remainder of the video,
a heat map indicated which pixels are most interesting to a user, the heat map derived from a click map that measures which areas of the user interface are most frequently clicked on, or derived from the portions of the user interface a user scrolls, or with eye tracking ability derived from the portions of a user interface that a user looks at most and wherein heat map inferred from the colors of pixels and render the pixels considered to have a higher user experience metric, and
a portion of a screen that provides buttons for user input is weighted higher than portions of a screen that provide design elements that provide no functionality.
9. The system of claim 8 , wherein the test script further causes: generating an indication of a fault based on the comparing.
10. The system of claim 8 , wherein:
the first mobile device includes a first operating system;
the first test software includes instrumentation of the first operating system;
and the first app-under-test is not modified to work with the test script.
11. The system of claim 8 , wherein:
the first and second mobile devices are the same device, and the first and second versions of the app-under-test are not the same.
12. The system of claim 8 , wherein:
the first and second mobile devices are the same model of device and not the same device, and the first and second versions of the app-under-test are the same versions.
13. The system of claim 8 , wherein:
the test script includes a URL indicating the first version of the app-under-test to be executed during the first test, and further indicating the at least one feature of the first app-under-test.
14. A computer readable memory containing instructions executable by a processor, the instructions causing:
recording of a display on a first mobile device as a first video during a first test including execution of at least one feature of a first version of an app-under-test;
determining a first performance rating based on analysis of the first video including detection of visual changes over time in the first test from the first video;
recording of the display on a second mobile device as a second video during a second test including execution of the at least one feature of a second version of the app-under-test;
determining a second performance rating based on analysis of the second video including detection of visual changes over time in the second test from the second video;
comparing the first and second tests based on the first performance rating and the second performance rating;
wherein the first performance rating and the second performance rating are visual speed ratings and the performance rating is visual speed index;
wherein the first and second performance ratings are based on an estimate of human visual perception of changes over time in the video being analyzed;
wherein list of tests with speed index ratings comprising many rows, each row represents one app test process including: a video identifier indicating video filename of the video captured during the test, the date and time of the test, the name of the test was run which including the type of mobile device and mobile device operating system used in that test, speed index rating, and a last visual change number resulting from analysis of the video; and
evaluating including the analysis of the video recording of a user interface of the first app-under-test and second app-under-test produce multiple different metrics from the video, the different metrics comprising:
evaluate area under a curve plotting the change of all pixels in the video of the user interface and weight certain pixels at certain times differently wherein the weight certain pixels at the certain times use knowledge of a particular user interface to weight a portion the video, such that the upper left-hand quarter of the screen over the remainder of the video,
a heat map indicated which pixels are most interesting to a user, the heat map derived from a click map that measures which areas of the user interface are most frequently clicked on, or derived from the portions of the user interface a user scrolls, or with eye tracking ability derived from the portions of a user interface that a user looks at most and wherein heat map inferred from the colors of pixels and render the pixels considered to have a higher user experience metric, and
a portion of a screen that provides buttons for user input is weighted higher than portions of a screen that provide design elements that provide no functionality.
15. The computer readable memory of claim 14 , wherein the instructions further cause:
generating an indication of a fault based on the comparing.