Dynamic communication architecture for testing computer security application features
A computer security system includes a test management system and associated communication architecture that enables creation of customized tests of computer security application features. A server stores a test script in a custom scripting language. The test script includes a set of control statements that may be organized in a decision tree to control facilitation of the test. Clients poll the server to independently obtain and execute the control statements. Execution of the control statements control which clients participate in a test, which feature will be tested in the test, and what telemetry data will be collected from the clients to evaluate the test. The server evaluates the telemetry data to determine an outcome of the test and determines whether to further distribute or roll back the tested feature based on the test outcome. The testing can be utilized to rapidly and robustly deploy features that will enhance computer security.
1. A method for facilitating testing of a feature of a computer security application, the method comprising:
obtaining, at a server, a script comprising a set of control statements;
receiving, from a plurality of clients, respective queries for the control statements;
providing the control statements to the clients in response to the respective queries, wherein each of the clients independently executes the control statements, and
wherein independently executing the control statements on the plurality of clients collectively causes the feature of the computer security application to be enabled on a subset of a plurality of clients that meet specified cohort criteria set forth in the control statements, and causes at least the subset of the plurality of clients to send telemetry data to the server as specified by the control statements, the telemetry data indicative of an effect of enabling the feature on the subset of the client applications;
in response to the execution of the control statements, receiving the telemetry data from at least the subset of the plurality of clients;
evaluating the telemetry data to determine a test outcome; and
executing an action based on the test outcome.
2. The method of claim 1 , wherein evaluating the telemetry data and executing the action comprises:
determining whether to roll back the feature or more widely distribute the feature; and
responsive to determining to more widely distribute the feature, causing the feature to be enabled on an additional set of clients outside the subset of clients.
3. The method of claim 1 , wherein each of the control statements includes a condition to be evaluated by a client and an action to be performed by the client based on evaluation of the condition, wherein the action for a first control statement comprises at least one of:
obtaining a second control statement for execution by the client, enabling or disabling the feature of the computer security application at the client, and sending telemetry data associated with the client to the server.
4. The method of claim 3 , wherein the control statements comprise a client-side control statement in which the condition is evaluated by the client based on locally-obtained data.
5. The method of claim 3 , wherein the control statements comprise a server-side control statement in which the condition is evaluated on behalf of the client by the server.
6. The method of claim 3 , wherein the set of control statements are organized in a decision tree.
7. The method of claim 1 , wherein enabling the feature comprises at least one of:
displaying a user message related to the computer security application; and
executing a set of computer instructions related to operation of the computer security application.
8. A non-transitory computer-readable storage medium storing instructions for facilitating testing of a feature of a computer security application, the instructions when executed by a processor causing the processor to perform steps including:
obtaining, at a server, a script comprising a set of control statements;
receiving, from a plurality of clients, respective queries for the control statements;
providing the control statements to the clients in response to the respective queries, wherein each of the clients independently executes the control statements, and
wherein independently executing the control statements on the plurality of clients collectively causes the feature of the computer security application to be enabled on a subset of a plurality of clients that meet specified cohort criteria set forth in the control statements, and causes at least the subset of the plurality of clients to send telemetry data to the server as specified by the control statements, the telemetry data indicative of an effect of enabling the feature on the subset of the client applications;
in response to the execution of the control statements, receiving the telemetry data from at least the subset of the plurality of clients;
evaluating the telemetry data to determine a test outcome; and
executing an action based on the test outcome.
9. The non-transitory computer-readable storage medium of claim 8 , wherein evaluating the telemetry data and executing the action comprises:
determining whether to roll back the feature or more widely distribute the feature; and
responsive to determining to more widely distribute the feature, causing the feature to be enabled on an additional set of clients outside the subset of clients.
10. The non-transitory computer-readable storage medium of claim 8 , wherein each of the control statements includes a condition to be evaluated by a client and an action to be performed by the client based on evaluation of the condition, wherein the action for a first control statement comprises at least one of:
obtaining a second control statement for execution by the client, enabling or disabling the feature of the computer security application at the client, and sending telemetry data associated with the client to the server.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the control statements comprise a client-side control statement in which the condition is evaluated by the client based on locally-obtained data.
12. The non-transitory computer-readable storage medium of claim 10 , wherein the control statements comprise a server-side control statement in which the condition is evaluated on behalf of the client by the server.
13. The non-transitory computer-readable storage medium of claim 10 , wherein the set of control statements are organized in a decision tree.
14. The non-transitory computer-readable storage medium of claim 8 , wherein enabling the feature comprises at least one of:
displaying a user message related to the computer security application; and
executing a set of computer instructions related to operation of the computer security application.
15. A computer system comprising:
a processor; and
a non-transitory computer-readable storage medium storing instructions for facilitating testing of a feature of a computer security application, the instructions when executed by the processor causing the processor to perform steps including:
obtaining, at a server, a script comprising a set of control statements;
receiving, from a plurality of clients, respective queries for the control statements;
providing the control statements to the clients in response to the respective queries, wherein each of the clients independently executes the control statements, and
wherein independently executing the control statements on the plurality of clients collectively causes the feature of the computer security application to be enabled on a subset of a plurality of clients that meet specified cohort criteria set forth in the control statements, and causes at least the subset of the plurality of clients to send telemetry data to the server as specified by the control statements, the telemetry data indicative of an effect of enabling the feature on the subset of the client applications;
in response to the execution of the control statements, receiving the telemetry data from at least the subset of the plurality of clients;
evaluating the telemetry data to determine a test outcome; and
executing an action based on the test outcome.
16. The computer system of claim 15 , wherein evaluating the telemetry data and executing the action comprises:
determining whether to roll back the feature or more widely distribute the feature; and
responsive to determining to more widely distribute the feature, causing the feature to be enabled on an additional set of clients outside the subset of clients.
17. The computer system of claim 15 , wherein each of the control statements includes a condition to be evaluated by a client and an action to be performed by the client based on evaluation of the condition, wherein the action for a first control statement comprises at least one of:
obtaining a second control statement for execution by the client, enabling or disabling the feature of the computer security application at the client, and sending telemetry data associated with the client to the server.
18. The computer system of claim 17 , wherein the control statements comprise a client-side control statement in which the condition is evaluated by the client based on locally-obtained data.
19. The computer system of claim 17 , wherein the control statements comprise a server-side control statement in which the condition is evaluated on behalf of the client by the server.
20. The computer system of claim 17 , wherein the set of control statements are organized in a decision tree.