System and method for performing automated API tests
A framework and a method for ad-hoc batch testing of APIs are provided, where batches of API calls are dynamically generated directly through the framework according inputs identifying the required tests and the sources of the test data, rather than through execution of prewritten test scripts that explicitly write out the test API calls in preset sequences. When performing the validation for an API test, a test payload is generated for the test, an endpoint is called using the test payload to obtain the response used for validation, where generating the test payload includes determining an API reference corresponding to the test, obtaining relevant data from the test data according to a reference key in the test, generating input assignment operations for one or more input parameters in the API reference according to the relevant data, and generating an API call based on the API reference.
1. A method for ad-hoc batch testing of APIs, the method comprising:
obtaining a plurality of API tests, and test data and validation data for the plurality of API tests;
categorizing, in accordance with a data model, data required for testing the plurality of API tests, into a plurality of groups including test scenarios, test cases, an inventory of APIs, the test data, and the validation data;
performing validation for each test of the plurality of API tests, wherein performing the validation includes:
generating a test payload for a respective test, wherein generating the test payload includes:
determining an API reference corresponding to the respective test;
obtaining relevant data from the test data according to a reference key in the respective test;
generating input assignment operations for one or more input parameters in the API reference according to the relevant data; and
generating an API call based on the API reference;
calling an endpoint corresponding to the respective test using the test payload;
receiving a response from the endpoint for the test payload; and
validating the response according to the validation data; and
generating and outputting one or more test reports based on the validated responses;
wherein performing the validation comprises:
generating a first test payload for a first test of the plurality of API tests;
calling a first endpoint corresponding to the first test using the first test payload;
receiving a first response from the first endpoint for the first test payload;
validating the first response according to the validation data;
selecting a second test according to a result of the validation of the first response, the result of the validation of the first response indicating whether the validation of the first response has passed or failed;
generating a second test payload for the second test of the plurality of API tests according to the first response;
calling a second endpoint corresponding to the second test using the second test payload;
receiving a second response from the second endpoint for the second test payload; and
validating the second response according to the validation data.
2. The method of claim 1 , further comprising obtaining an API inventory, wherein determining the API reference comprises identifying the API reference corresponding to the respective test according to the API inventory.
3. The method of claim 1 , wherein generating the input assignment operations comprises generating database queries to fetch data from a database corresponding to the respective test.
4. The method of claim 1 , further comprising, prior to calling the endpoint, obtaining an API inventory, and identifying the endpoint according to the API inventory.
5. The method of claim 1 , further comprising:
obtaining a plurality of API test scenarios, each API test scenario including a respective one or more API tests from amongst the plurality of API tests; and
performing the validation for each test corresponding to a first API test scenario before performing the validation for each test corresponding to a second API test scenario.
6. The method of claim 1 , further comprising:
obtaining validation levels for each test of the plurality of API tests; and
validating the response according to the validation level of the respective test.
7. The method of claim 6 , wherein the validation levels of the plurality of API tests include a first validation level, and validating the response comprises, in accordance with a determination that the respective test corresponds to the first validation level, verifying if a status code in the response matches an entry in the validation data.
8. The method of claim 7 , wherein the validation levels of the plurality of API tests include a second validation level, the method further comprising, in accordance with a determination that the respective test corresponds to the second validation level, verifying if one or more data fields in the response matches one or more values in the validation data.
9. The method of claim 8 , wherein the validation levels of the plurality of API tests include a third validation level, the method further comprising, in accordance with a determination that the test corresponds to the third validation level, verifying if a structure of the response matches a hierarchy in the validation data.
10. The method of claim 1 , wherein validating the response comprises fetching data from a database corresponding to the respective test and comparing the data to the response.
11. The method of claim 1 , wherein the validation is performed in parallel for each test of the plurality of API tests.
12. The method of claim 1 , wherein each test of the plurality of API tests validates a REST API, or a SOAP API.
13. The method of claim 1 , the method further comprising:
in accordance with a determination that the API reference uses an HTTP protocol, determining one or more URL parameters for the API reference while generating the test payload.
14. The method of claim 1 , wherein obtaining the plurality of API tests, and the test data and the validation data comprises:
receiving one or more input files corresponding to the plurality of API tests; and
extracting the plurality of API tests, the test data and the validation data from the one or more input files.
15. The method of claim 14 , wherein obtaining the plurality of API tests, and the test data and the validation data further comprises:
storing the plurality of API tests, the test data and the validation data into a database; and
retrieving the plurality of API tests, the test data and the validation data from the database.
16. The method of claim 1 , further comprising performing the validation for a first set of tests of the plurality of API tests in parallel.
17. An electronic device, comprising:
one or more processors; and
memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for:
obtaining a plurality of API tests, and test data and validation data for the plurality of API tests;
categorizing, in accordance with a data model, data required for testing the plurality of API tests, into a plurality of groups including test scenarios, test cases, an inventory of APIs, the test data, and the validation data;
performing validation for each test of the plurality of API tests, wherein performing the validation includes:
generating a test payload for a respective test, wherein generating the test payload includes:
determining an API reference corresponding to the respective test;
obtaining relevant data from the test data according to a reference key in the respective test;
generating input assignment operations for one or more input parameters in the API reference according to the relevant data; and
generating an API call based on the API reference;
calling an endpoint corresponding to the respective test using the test payload;
receiving a response from the endpoint for the test payload; and
validating the response according to the validation data; and
generating and outputting one or more test reports based on the validated responses;
wherein performing the validation comprises:
generating a first test payload for a first test of the plurality of API tests;
calling a first endpoint corresponding to the first test using the first test payload;
receiving a first response from the first endpoint for the first test payload;
validating the first response according to the validation data;
selecting a second test according to a result of the validation of the first response, the result of the validation of the first response indicating that the validation of the first response has passed or failed;
generating a second test payload for the second test of the plurality of API tests according to the first response;
calling a second endpoint corresponding to the second test using the second test payload;
receiving a second response from the second endpoint for the second test payload; and
validating the second response according to the validation data.
18. A non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for:
obtaining a plurality of API tests, and test data and validation data for the plurality of API tests;
categorizing, in accordance with a data model, data required for testing the plurality of API tests, into a plurality of groups including test scenarios, test cases, an inventory of APIs, the test data, and the validation data;
performing validation for each test of the plurality of API tests, wherein performing the validation includes:
generating a test payload for a respective test, wherein generating the test payload includes:
determining an API reference corresponding to the respective test;
obtaining relevant data from the test data according to a reference key in the respective test;
generating input assignment operations for one or more input parameters in the API reference according to the relevant data; and
generating an API call based on the API reference;
calling an endpoint corresponding to the respective test using the test payload;
receiving a response from the endpoint for the test payload; and
validating the response according to the validation data; and
generating and outputting one or more test reports based on the validated responses;
wherein performing the validation comprises:
generating a first test payload for a first test of the plurality of API tests;
calling a first endpoint corresponding to the first test using the first test payload;
receiving a first response from the first endpoint for the first test payload;
validating the first response according to the validation data;
selecting a second test according to a result of the validation of the first response, the result of the validation of the first response indicating that the validation of the first response has passed or failed;
generating a second test payload for the second test of the plurality of API tests according to the first response;
calling a second endpoint corresponding to the second test using the second test payload;
receiving a second response from the second endpoint for the second test payload; and
validating the second response according to the validation data.
19. A method for ad-hoc batch testing of APIs, the method comprising:
obtaining a plurality of API tests, and test data and validation data for the plurality of API tests;
performing validation for each test of the plurality of API tests, wherein performing the validation includes:
generating a test payload for the respective test, wherein generating the test payload includes:
determining an API reference corresponding to the respective test;
obtaining relevant data from the test data according to a reference key in the respective test, wherein the reference key indicates if one or more parameters in the API reference are hardcoded as constants or variables that need to be collected from the test data;
generating input assignment operations for one or more input parameters in the API reference according to the relevant data; and
generating an API call based on the API reference;
calling an endpoint corresponding to the respective test using the test payload;
receiving a response from the endpoint for the test payload; and
validating the response according to the validation data; and
generating and outputting one or more test reports based on the validated responses.
20. The method of claim 19 , further comprising obtaining an API inventory, wherein determining the API reference comprises identifying the API reference corresponding to the respective test according to the API inventory.
21. The method of claim 19 , further comprising:
obtaining validation levels for each test of the plurality of API tests, wherein the validation levels of the plurality of API tests include a third validation level; and
validating the response according to the validation level of the respective test.
22. The method of claim 21 , wherein the validation levels of the plurality of API tests include a first validation level, and validating the response comprises, in accordance with a determination that the respective test corresponds to the first validation level, verifying if a status code in the response matches an entry in the validation data.
23. The method of claim 22 , wherein the validation levels of the plurality of API tests include a second validation level, the method further comprising, in accordance with a determination that the respective test corresponds to the second validation level, verifying if one or more data fields in the response matches one or more values in the validation data.
24. The method of claim 23 , wherein the validation levels of the plurality of API tests include a third validation level, the method further comprising, in accordance with a determination that the test corresponds to the third validation level, verifying if a structure of the response matches a hierarchy in the validation data.