Automated testing of mobile devices using visual analysis
Described herein are techniques that may be used to automate testing of services on mobile devices using visual analysis. In some embodiments, a machine learning model is trained using interaction data received from a number of mobile devices by correlating user selections with visual elements (e.g., icons). During execution of a testing routine on a mobile device, screenshots are obtained of a screen of the mobile device and provided to the machine learning model. An action is generated based on the provided screenshot that simulates a user action (e.g., a user touch on the screen of the mobile device) at a location of an icon or other visual element associated with the testing routine. These steps are repeated until an end-state of the testing routine is detected.
1. A method comprising:
training a machine learning model based on a set of interaction data associated with a user goal, the interaction data including at least locations of touch selections and screenshot images, wherein the training includes removing outlier interactions from the set of interaction data, the outlier interactions including extra user selections or user selections for which coordinates are greater than a threshold distance from average coordinates for other user selections;
receiving a request to complete a testing routine on a mobile device;
until determining that the testing routine is complete, repeating a process that comprises:
obtaining a current snapshot of a screen of the mobile device;
determining a testing action for the mobile device based on the current snapshot, wherein determining the testing action for the mobile device based on the current snapshot comprises providing the current snapshot to the machine learning model; and
executing the testing action for the mobile device; and
responding to the request with whether the testing routine is a failure or success based upon the current snapshot.
2. The method of claim 1 , further comprising determining, based on the current snapshot, whether testing is complete, wherein the determining the testing action is performed upon determining that testing is not complete.
3. The method of claim 1 , wherein the testing action for the mobile device is determined based on a type of the mobile device or an operating system of the mobile device.
4. The method of claim 1 , wherein executing the testing action comprises simulating a human touch at a specific location on the screen of the mobile device.
5. The method of claim 4 , wherein the specific location on the screen of the mobile device corresponds to a location of an icon identified within the current snapshot.
6. The method of claim 5 , wherein the icon is an icon determined to be associated with the testing action.
7. The method of claim 1 , wherein the testing action is determined based upon actions taken by a number of users.
8. A computing device comprising:
a processor; and
a memory including instructions that, when executed with the processor, cause the computing device to, at least:
training a machine learning model based on a set of interaction data associated with a user goal, the interaction data including at least locations of touch selections and screenshot images, wherein the machine learning model is one of a plurality of machine learning models respectively associated with a plurality of user goals;
receive a request to complete a testing routine on a mobile device, wherein the testing routine is associated with the user goal;
until determining that the testing routine is complete, repeat a process that comprises:
obtaining a current snapshot of a screen of the mobile device;
selecting the machine learning model from the plurality of machine learning models based on the user goal;
determining a testing action for the mobile device based on the current snapshot, wherein determining the testing action for the mobile device based on the current snapshot comprises providing the current snapshot to the selected machine learning model; and
executing the testing action for the mobile device; and
respond to the request with whether the testing routine is a failure or success based upon the current snapshot.
9. The computing device of claim 8 , wherein the computing device comprises the mobile device and the instructions comprise a testing application.
10. The computing device of claim 8 , wherein the computing device is a server on which a testing platform is implemented.
11. The computing device of claim 9 , wherein the testing action for the mobile device is also determined based at least in part on a type of the mobile device or an operating system installed upon the mobile device.
12. The computing device of claim 8 , wherein the current snapshot of the screen of the mobile device comprises a number of icons displayed upon the screen of the mobile device.
13. The computing device of claim 12 , wherein the instructions further cause the computing device to determine a category and coordinates for each of the number of icons displayed upon the screen of the mobile device.
14. The computing device of claim 8 , wherein the testing routine is determined to be a failure upon determining that the current snapshot does not match an expected snapshot.
15. The computing device of claim 8 , wherein the testing routine is determined to be a success upon determining that the current snapshot is associated with an end of the testing routine.
16. The computing device of claim 8 , wherein the testing routine is associated with the user goal.
17. A non-transitory computer-readable media collectively storing computer-executable instructions that upon execution cause one or more computing devices to collectively perform acts comprising:
training a machine learning model based on a set of interaction data associated with a user goal, the interaction data including at least locations of touch selections and screenshot images, wherein the training includes removing outlier interactions from the set of interaction data, the outlier interactions including extra user selections or user selections for which coordinates are greater than a threshold distance from average coordinates for other user selections;
receiving a request to complete a testing routine on a mobile device;
until determining that the testing routine is complete, repeating a process that comprises:
obtaining a current snapshot of a screen of the mobile device;
determining a testing action for the mobile device based on the current snapshot, wherein determining the testing action for the mobile device based on the current snapshot comprises providing the current snapshot to the machine learning model; and
executing the testing action for the mobile device; and
responding to the request with whether the testing routine is a failure or success based upon the current snapshot.
18. The non-transitory computer-readable media of claim 17 , wherein the testing action for the mobile device is determined based on a testing flow associated with the testing routine.
19. The non-transitory computer-readable media of claim 18 , wherein the testing flow is an ordered series of steps generated by a user.
20. The non-transitory computer-readable media of claim 17 , wherein the process further comprises determining, based on the current snapshot, whether testing is complete, wherein the determining the testing action is performed upon determining that testing is not complete.