Performance utilities for mobile applications
Techniques for evaluating software systems are provided, including measurement of performance ratings of features of an app-under-test are described. The evaluation may include analysis of a video recording of the user interface of the app-under-test, where the video analysis may include comparing the changes over time of co-located pixel in the video to produce a performance rating, for example a speed index. The results of multiple such tests can be compared by comparing the performance rating produced during execution of different tests.
1. A computer-implemented method for testing software, wherein the method comprises:
detecting a first plurality of visual changes over time, of a first mobile device application, while a first feature of the first mobile device application is executed by a hardware processor;
detecting a second plurality of visual changes over time, of a second mobile device application, while a second feature of the second mobile device application is executed by the hardware processor;
comparing the first mobile device application and the second mobile device application based on the first plurality of visual changes and the second plurality of visual changes; and
evaluating a first metric associated with the first mobile device application and a second metric associated with the second mobile device application, wherein:
the first mobile device application comprises a first user interface,
the second mobile device application comprises a second user interface,
the first metric is indicative of a change in pixels of the first user interface,
the second metric is indicative of a change in pixels of the second user interface,
certain pixels of the first user interface and/or the second user interface are weighted differently at certain times,
the different weighting is indicative of a user's interest in a portion the first user interface and/or the second user interface, and
the user's interest is indicated by a heat map derived from one or both of clicks by the user or eye tracking of the user.
2. The computer-implemented method of claim 1 , wherein the method comprises generating an indication of a fault based on the comparing.
3. The computer-implemented method of claim 1 , wherein the first mobile device application is operable in a first operating system, and wherein the first operating system comprises test instrumentation, and wherein the first mobile device application is not modified.
4. The computer-implemented method of claim 1 , wherein the first mobile device application and the second mobile device application are operable in the same mobile device.
5. The computer-implemented method of claim 1 , wherein the first mobile device application and the second mobile device application are different versions of the same application.
6. The computer-implemented method of claim 1 , wherein the first feature is the same as the second feature.
7. The computer-implemented method of claim 1 , the method comprises:
initiating the first mobile device application by providing a uniform resource locator (URL) indicating a first version of an app-under-test; and
initiating the second mobile device application by providing a URL indicating a second version of the app-under-test.
8. The computer-implemented method of claim 1 , wherein the method comprises:
generating a first performance rating according to the first plurality of visual changes over time; and
generating a second performance rating according to the second plurality of visual changes over time.
9. The computer-implemented method of claim 8 , wherein the first performance rating and the second performance rating are visual speed ratings.
10. The computer-implemented method of claim 8 , wherein the first performance rating and the second performance rating are based on an estimate of human visual perception of changes over time in video being analyzed.
11. A system for testing an app, comprising:
a test computer server including a test script;
a video analysis server;
a first a mobile device application;
a second tho mobile device application; and
wherein the test script is operable to:
detect a first plurality of visual changes over time, of the first tho mobile device application, while a first feature of the first mobile device application is executed by a hardware processor;
detect a second plurality of visual changes over time, of the second the mobile device application, while a second feature of the second mobile device application is executed by the hardware processor; and
compare the first mobile device application and the second the mobile device application based on the first plurality of visual changes over time and the second plurality of visual changes over time; and
evaluate a first metric associated with the first mobile device application and a second metric associated with the second mobile device application, wherein:
the first mobile device application comprises a first user interface,
the second mobile device application comprises a second user interface,
the first metric is indicative of a change in pixels of the first user interface,
the second metric is indicative of a change in pixels of the second user interface,
certain pixels of the first user interface and/or the second user interface are weighted differently at certain times,
the different weighting is indicative of a user's interest in a portion the first user interface and/or the second user interface, and
the user's interest is indicated by a heat map derived from one or both of clicks by the user or eye tracking of the user.
12. The system of claim 11 , wherein the test script is operable to generate an indication of a fault based on the comparing.
13. The system of claim 11 , wherein the first mobile device application is operable in a first operating system, and wherein the first operating system comprises test instrumentation, and wherein the first mobile device application is not modified.
14. The system of claim 11 , wherein the first mobile device application and the second mobile device application are operable in the same mobile device.
15. The system of claim 11 , wherein the first mobile device application and the second mobile device application are different versions of the same application.
16. The system of claim 11 , wherein the first feature is the same as the second feature.
17. The system of claim 11 , the test script is operable to:
initiate the first mobile device application by providing a uniform resource locator (URL) indicating a first version of an app-under-test; and
initiate the second mobile device application by providing a URL indicating a second version of the app-under-test.
18. The system of claim 11 , wherein the test script is operable to:
generate a first performance rating according to the first plurality of visual changes over time; and
generate a second performance rating according to the second plurality of visual changes over time.
19. The system of claim 18 , wherein the first performance rating and the second performance rating are visual speed ratings.
20. The system of claim 18 , wherein the first performance rating and the second performance rating are based on an estimate of human visual perception of changes over time in video being analyzed.