Granted Patent B1

US 12,273,255 · App. 18/479,712 · Granted Apr 8, 2025

Inventors: Abhijit Prakash Bhatnagar (Bellevue, WA); Yusof Ganji (Bellevue, WA); Mohsen Azimi (Seattle, WA); Jason Adonis Timmons (Seattle, WA); Jacob Shannan Carr (Seattle, WA); Tristan Niles Cecil (Los Altos, CA); Evan Corriere (Sunnyvale, CA); Sahil Sharma (San Francisco, CA); Xinrui Li (Seattle, WA); Huaqing Fang (New Jersey, NJ)

Assignee: Amazon Technologies, Inc.

H04L43/55H04L43/024H04L43/106H04L41/147

Abstract

Claims (100)

1. A system comprising:

one or more computer devices that implement an adaptive testing service (ATS), configured to:

execute a test case generation process, including to:

receive telemetry data of a production version of a network service in a production environment;

derive, from the telemetry data, a plurality of behaviors of the network service and frequency or recency metrics of individual ones of the behaviors;

generate a plurality of test cases executable to test individual ones of the behaviors, including weights for individual ones of the test cases determined based at least in part on the frequency or recency metrics; and

store the test cases and the weights in a test case repository; and

execute a code change testing process, including to:

receive a code change to a development version of the network service in a development environment;

execute the test cases in the test case repository to identify one or more changed behaviors of the development version of the network service as a result of the code change;

determine to accept or reject the code change based at least in part on the one or more changed behaviors and one or more weights of one or more test cases that identified the one or more changed behaviors; and

after the code change is applied into the development version, update at least one of the one or more weights to cause at least one of the one or more test cases to be deprecated in the test case repository.

2. The system of claim 1 , wherein:

the ATS is configured perform different types of tests on different types of network services in a plurality of environments;

the telemetry data is collected by one or more agents deployed in the production environment;

the agents are configured to perform test requests in the production environment according to instructions from a controller component of the ATS;

the ATS is configured to output test results obtained from the test requests via a user interface or an alarm; and

the different types of tests include two or more of:

a service performance test,

a network connectivity test,

a test of one or more HTTP behaviors,

a test of one or more reverse proxy behaviors,

an analysis of network packets, and

a service performance test.

3. The system of claim 1 , wherein ATS is configured to evolve the test cases in the test case repository, including to:

repeatedly update the test cases based on new telemetry data collected from the production version of the network service, including to (a) add a new test case for newly discovered behavior in the production version of the network service or (b) deprecate a test case or adjust a weight of the test case based on a change in the frequency or recency metrics of the test case; and

repeatedly update the test cases based on new code changes in the development version of the network service, including to (a) add a new test case for new behavior added to the development version of the network service or (b) deprecate a test case or adjust a weight of the test case based on usage of the test case.

4. The system of claim 1 , wherein the ATS is configured to:

derive, from telemetry data of multiple production versions of the network service executing in multiple production environments, a set of environment-neutral behaviors of the network service in the production environments;

add environment-specific parameters to individual ones of the environment-neutral behaviors to generate a set of environment-specific test cases for respective ones of the production versions; and

execute the environment-specific test cases in respective ones of the production environments to test the respective ones of the production versions.

5. The system of claim 4 , wherein the ATS is configured to execute the environment-specific test cases to monitor the production versions, wherein the monitoring verifies that:

(a) expected behaviors of an individual production version do not changed over time, and

(b) corresponding behaviors of two production versions do not deviate.

6. The system of claim 1 , wherein to generate a test case, the ATS is configured to:

generate the test case in a behavior specification language, wherein a precondition, a test action, and an expected outcome of the test case are expressed in a natural language; and

generate a descriptive name for the test case in the natural language.

7. A method comprising:

executing, by an adaptive testing service (ATS) implemented by one or more computer devices, a test case generation process, including:

receiving telemetry data of a production version of a network service in a production environment;

deriving, from the telemetry data, a plurality of behaviors of the network service and frequency or recency metrics of individual ones of the behaviors;

generating a plurality of test cases executable to test individual ones of the behaviors, including weights for individual ones of the test cases determined based at least in part on the frequency or recency metrics; and

storing the test cases and the weights in a test case repository; and

executing, by the ATS, a code change testing process, including:

receiving a code change to a development version of the network service in a development environment;

executing the test cases in the test case repository to identify one or more changed behaviors of the development version of the network service as a result of the code change;

determining to accept or reject the code change based at least in part on the one or more changed behaviors and one or more weights of one or more test cases that identified the one or more changed behaviors; and

updating the one or more weights of the one or more test cases based at least in part on one or more code changes to the development version.

8. The method of claim 7 , further comprising the ATS:

repeatedly update the test cases based on new telemetry data collected from the production version of the network service, including to (a) add a new test case for newly discovered behavior in the production version of the network service or (b) deprecate a test case or adjust a weight of the test case based on a change in the frequency or recency metrics of the test case; and

repeatedly update the test cases based on new code changes in the development version of the network service, including to (a) add a new test case for new behavior added to the development version of the network service, or (b) deprecate a test case or adjust a weight of the test case based on usage of the test case.

9. The method of claim 7 , further comprising the ATS:

deriving, from telemetry data of multiple production versions of the network service executing in multiple production environments, a set of environment-neutral behaviors of the network service in the production environments;

adding environment-specific parameters to individual ones of the environment-neutral behaviors to generate a set of environment-specific test cases for respective ones of the production versions; and

executing the environment-specific test cases in respective ones of the production environments to test the respective ones of the production versions.

10. The method of claim 7 , wherein the deriving of the behaviors comprises applying a clustering or classification technique to the telemetry data to identify behavior classes.

11. The method of claim 7 , wherein the telemetry data is collected by one or more agents deployed in the production environment.

12. The method of claim 7 , wherein the weights are used in the code change testing process to (a) select test cases to use to test code changes to the network service or (b) select test cases to deprecate.

13. The method of claim 7 , further comprising the ATS:

determining that the code change causes a changed behavior associated with test case;

generating user interface data indicating the code change, the changed behavior, and a weight of the test case; and

receiving user input indicating whether the changed behavior is expected or desired, wherein the code change is accepted or rejected based at least in part on the user input.

14. The method of claim 13 , wherein:

the changed behavior changes a previous behavior of the development version of the network service; and

the user interface data indicates a last time that the previous behavior was observed.

15. The method of claim 7 , wherein generating a test case comprises:

generating the test case in a behavior specification language, wherein a precondition, a test action, and an expected test result of the test case are expressed in a natural language; and

generating a descriptive name for the test case in the natural language.

16. The method of claim 7 , further comprising the ATS:

generating user interface data to implement a graphical user interface (GUI);

receiving, from the GUI, a user query to search the test case repository for one or more matching test cases;

causing the GUI to display a test case that matches the user query;

receiving, from the GUI, a user command to execute the test case on one or more environments;

executing the test case on the one or more environments according to the user command; and

causing the GUI to display test results obtained from the execution of the test case.

17. The method of claim 16 , wherein the test results include results data associated with multiple network protocols, including two or more of:

(a) IP protocol,

(b) TCP protocol,

(c) DNS protocol,

(d) HTTP protocol, and

(e) SSL/TLS protocol.

18. The method of claim 16 , wherein the test results include historical data collected during one or more previous executions of the test case.

19. The method of claim 7 , further comprising the ATS generating and executing different types of test cases, including two or more test cases that specify two or more of:

a service performance test,

a network connectivity test,

a test of one or more HTTP behaviors,

a test of one or more reverse proxy behaviors,

an analysis of network packets, and

a service performance test.

20. One or more non-transitory computer-readable storage media storing program instructions executable by one or more processors to implement an adaptive testing service (ATS) and cause the ATS to:

execute a test case generation process, including to:

receive telemetry data of a production version of a network service in a production environment;

derive, from the telemetry data, a plurality of behaviors of the network service and frequency or recency metrics of individual ones of the behaviors;

generate a plurality of test cases executable to test individual ones of the behaviors, including weights for individual ones of the test cases determined based at least in part on the frequency or recency metrics; and

store the test cases and the weights in a test case repository; and

execute a code change testing process, including to:

receive a code change to a development version of the network service in a development environment;

execute the test cases in the test case repository to identify one or more changed behaviors of the development version of the network service as a result of the code change;

determine to accept or reject the code change based at least in part on the one or more changed behaviors and one or more weights of one or more test cases that identified the one or more changed behaviors; and

update the one or more weights of the one or more test cases based at least in part on one or more code changes to the development version.

Assignments (1)

References Cited (20)