Automated videogame testing
An automated testing framework and associated tools for executable applications such as games focus on integration testing, wherein users create data-driven tests by using test modules and configuration data as building blocks. The tools facilitate cooperation between coders and non-technical Quality Assurance (QA) staff in creating automated tests, by simplifying the user interface for configuring tests. Components of the tools simulate user interactions with the application under test, for example, gamepad button presses. The tools are also capable skipping portions of gameplay or other interactive activity and directly jumping into a desired game mode during automated testing, and other functions.
1. A computer-implemented method for automatically testing reaction of an executable application to user inputs, the method comprising:
selecting, by at least one processor of a computer executing a first operating system in response to first user input, a script file encoding a user input sequence from multiple script files in a computer memory;
identifying, by the at least one processor, at least one test device configured for executing the executable application and communicating with the at least one processor;
determining, by the at least one processor, whether the at least one test device is operating the first operating system, an operating system compatible with the first operating system, or a second operating system that is not compatible with the first operating system;
if the at least one test device is operating the second operating system, instructing a server interposed between the at least one test device and the at least one processor to launch an agent application on the at least one test device;
if the at least one test device is operating one of the first operating system or an operating system compatible with the first operating system, causing the at least one test device to launch the agent application without involvement of any intermediate server;
establishing a connection between the at least one processor and the agent application operating on the at least one test device for outputting the sequence of commands;
generating, by the at least one processor based on the script file, a sequence of commands for processing by the executable application; and
outputting the sequence of commands for delivery to the at least one test device executing the executable application.
2. The method of claim 1 , wherein the selecting further comprises presenting a list of the multiple script files in a user interface and receiving a user selection from the list indicating the script file.
3. The method of claim 1 , wherein the selecting further comprises presenting a list of test devices satisfying one or more criteria in a user interface and receiving a user selection from the list indicating the at least one test device.
4. The method of claim 1 , further comprising, by the at least one processor, performing multiple instances of the method in parallel.
5. The method of claim 1 , further comprising receiving an indication from the at least one test device indicating a status of the executable application after executing the sequence of commands, and outputting the indication.
6. The method of claim 1 , wherein the executable application includes a test module comprising a communication layer that handles communication with the at least one processor, and a game layer that processes the sequence of commands.
7. The method of claim 6 , wherein the communication module of the executable application listens for incoming Transmission Control Protocol (TCP) connection requests generated by the at least one processor.
8. The method of claim 1 , wherein the generating further comprises transcoding the user input sequence from a scripting language to a command language.
9. The method of claim 8 , wherein the scripting language comprises Python and the command language comprises a JavaScript Object Notation Remote Procedure Call (JSON-RPC).
10. The method of claim 1 , wherein the outputting further comprises opening a Transmission Control Protocol (TCP) session with the test device.
11. An apparatus for automatically testing reaction of an executable application to user inputs, comprising at least one processor operatively coupled to a computer memory, the memory holding instructions that when executed by the processor, cause the apparatus to perform:
selecting a script file encoding a user input sequence from multiple script files in a computer memory;
identifying at least one test device configured for executing the executable application and communicating with the at least one processor;
determining whether the at least one test device is operating the first operating system, an operating system compatible with the first operating system, or a second operating system that is not compatible with the first operating system;
if the at least one test device is operating the second operating system, instructing a server interposed between the at least one test device and the at least one processor to launch an agent application on the at least one test device;
if the at least one test device is operating one of the first operating system or an operating system compatible with the first operating system, causing the at least one test device to launch the agent application without involvement of any intermediate server;
establishing a connection between the at least one processor and the agent application operating on the at least one test device for outputting the sequence of commands;
generating a sequence of commands for processing by the executable application; and
outputting the sequence of commands for delivery to the at least one test device executing the executable application.
12. The apparatus of claim 11 , wherein the memory holds further instructions for communicating with a communication layer of a test module of the executable application, wherein the communication layer handles communication and a game layer of the test module processes the sequence of commands.
13. The apparatus of claim 11 , wherein the memory holds further instructions for the generating at least in part by transcoding the user input sequence from a scripting language to a command language.
14. A system for automatically testing a video game by simulating various inputs provided by a user during game play, the system comprising:
an electronic testing device under the control of a first operating system, wherein the electronic testing device communicates with one or more electronic game play devices under control of one of the first operating system or an operating system compatible with the first operating system, without involvement of any intermediate server;
a second electronic game play device under the control of a second operating system not compatible with the first operating system; and
an electronic bridge device comprising a server interposed between the electronic testing device and the second electronic game play device configured to receive instructions from the electronic testing device, launch an automated test application on the second electronic game play device, wherein the electronic testing device is configured to launch the automated test application upon determining that the second electronic game play device is operating the second operating system that is not compatible with the first operating system.
15. The system of claim 14 , wherein the electronic testing device instructs the electronic bridge device to launch the automated test application on the second electronic game play device.
16. The system of claim 15 , wherein the electronic bridge device establishes a connection between the electronic testing device and the automated test application operating on the electronic game play device for outputting a sequence of commands from the electronic testing device to the second electronic game play device.
17. The system of claim 16 , wherein the electronic testing device generates the sequence of commands for delivery to the automated test application operating on the second electronic game play device.