Performance utilities for mobile applications
Techniques for evaluating software systems are provided, including measurement of performance ratings of features of an app-under-test are described. The evaluation may include analysis of a video recording of the user interface of the app-under-test, where the video analysis may include comparing the changes over time of co-located pixel in the video to produce a performance rating, for example a speed index. The results of multiple such tests can be compared by comparing the performance rating produced during execution of different tests.
1. A method for testing software, wherein the method is computer-implemented and comprises:
executing a first feature of a first mobile device application;
detecting a first plurality of visual changes over time, of the first mobile device application, while the first feature of the first mobile device application is executed by a hardware processor;
weighting portions of the first plurality of visual changes over time according to one or both of eye tracking of a user and clicks of the user, while the first feature of the first mobile device application is executed;
executing a second feature of a second mobile device application;
detecting a second plurality of visual changes over time, of the second mobile device application, while the second feature of the second mobile device application is executed by the hardware processor;
weighting portions of the second plurality of visual changes over time according to one or both of eye tracking of the user and clicks of the user, while the second feature of the second mobile device application is executed; and
comparing the first mobile device application and the second mobile device application based on the weighted first plurality of visual changes and the weighted second plurality of visual changes and run the all comparison tests on the same device.
2. The method of claim 1 , wherein the method comprises generating an indication of a fault based on the comparing.
3. The method of claim 1 , wherein the first mobile device application is operable in a first operating system, and wherein the first operating system comprises test instrumentation, and wherein the first mobile device application is not modified.
4. The method of claim 1 , wherein the first mobile device application and the second mobile device application are operable in the same mobile device.
5. The method of claim 1 , wherein the first mobile device application and the second mobile device application are different versions of the same application.
6. The method of claim 1 , wherein the first feature is the same as the second feature.
7. The method of claim 1 , the method comprises:
initiating the first mobile device application by providing a uniform resource locator (URL) indicating a first version of an app-under-test; and
initiating the second mobile device application by providing a URL indicating a second version of the app-under-test.
8. The method of claim 1 , wherein the method comprises:
generating a first performance rating according to the first plurality of visual changes over time; and
generating a second performance rating according to the second plurality of visual changes over time.
9. The method of claim 8 , wherein the first performance rating and the second performance rating are visual speed ratings.
10. The method of claim 8 , wherein the first performance rating and the second performance rating are based on an estimate of human visual perception of changes over time in video being analyzed.
11. A system for testing an app, comprising:
a test computer server including a test script;
a video analysis server;
a first mobile device application;
a second mobile device application; and
wherein the test script is operable to:
executing a first feature of the first mobile device application;
detect a first plurality of visual changes over time, of the first mobile device application, while the first feature of the first mobile device application is executed by a hardware processor;
weight portions of the first plurality of visual changes over time according to one or both of eye tracking of a user and clicks of the user, while the first feature of the first mobile device application is executed;
executing a second feature of a second mobile device application;
detect a second plurality of visual changes over time, of the second mobile device application, while the second feature of the second mobile device application is executed by the hardware processor;
weight portions of the second plurality of visual changes over time according to one or both of eye tracking of the user and clicks of the user, while the second feature of the second mobile device application is executed; and
compare the first mobile device application and the second mobile device application based on the weighted first plurality of visual changes over time and the weighted second plurality of visual changes over time, and run the all compare tests on the same device.
12. The system of claim 11 , wherein the test script is operable to generate an indication of a fault based on the comparing.
13. The system of claim 11 , wherein the first mobile device application is operable in a first operating system, and wherein the first operating system comprises test instrumentation, and wherein the first mobile device application is not modified.
14. The system of claim 11 , wherein the first mobile device application and the second mobile device application are operable in the same mobile device.
15. The system of claim 11 , wherein the first mobile device application and the second mobile device application are different versions of the same application.
16. The system of claim 11 , wherein the first feature is the same as the second feature.
17. The system of claim 11 , the test script is operable to:
initiate the first mobile device application by providing a uniform resource locator (URL) indicating a first version of an app-under-test; and
initiate the second mobile device application by providing a URL indicating a second version of the app-under-test.
18. The system of claim 11 , wherein the test script is operable to:
generate a first performance rating according to the first plurality of visual changes over time; and
generate a second performance rating according to the second plurality of visual changes over time.
19. The system of claim 18 , wherein the first performance rating and the second performance rating are visual speed ratings.
20. The system of claim 18 , wherein the first performance rating and the second performance rating are based on an estimate of human visual perception of changes over time in video being analyzed.