Dynamic communication architecture for testing computer security application features
A computer security system includes a test management system and associated communication architecture that enables creation of customized tests of computer security application features. A server stores a test script in a custom scripting language. The test script includes a set of control statements that may be organized in a decision tree to control facilitation of the test. Clients poll the server to independently obtain and execute the control statements. Execution of the control statements control which clients participate in a test, which feature will be tested in the test, and what telemetry data will be collected from the clients to evaluate the test. The server evaluates the telemetry data to determine an outcome of the test and determines whether to further distribute or roll back the tested feature based on the test outcome. The testing can be utilized to rapidly and robustly deploy features that will enhance computer security.
1. A method for facilitating testing of a feature of a computer security application, the method comprising:
obtaining, at a server, a script comprising a set of control statements in a hierarchy, the hierarchy having a first path including a first subset of control statements and a second path including a second subset of control statement, wherein which path a client device traverses depends on evaluation of one or more conditions;
receiving, from a plurality of clients, respective queries for the control statements;
providing at least some of the control statements to the plurality of clients in response to the queries, wherein a first client of the plurality of clients follows the first path of the plurality of paths and a second client of the plurality of clients follows the second path of the plurality of paths;
receiving telemetry data from the first and the second clients, wherein the first and the second clients send telemetry data to the server as specified respectively by the first and second subsets of control statements, the telemetry data indicative of an effect of enabling the feature on client applications associated with the first and the second clients, respectively;
evaluating the telemetry data to determine a test outcome; and
executing an action based on the test outcome.
2. The method of claim 1 , wherein evaluating the telemetry data to determine a test outcome comprises:
determining whether to roll back the feature or more widely distribute the feature; and
responsive to determining to more widely distribute the feature, causing the feature to be enabled on an additional set of clients outside the subset of clients.
3. The method of claim 1 , wherein each of the control statements includes a condition to be evaluated by a client and an action to be performed by the client based on evaluation of the condition, wherein the action for a first control statement comprises at least one of:
obtaining a second control statement for execution by the client, enabling or disabling the feature of the computer security application at the client, and sending telemetry data associated with the client to the server.
4. The method of claim 3 , wherein the control statements comprise a client-side control statement in which the condition is evaluated by the client based on locally-obtained data.
5. The method of claim 3 , wherein the control statements comprise a server-side control statement in which the condition is evaluated on behalf of the client by the server.
6. The method of claim 1 , wherein enabling the feature comprises at least one of:
displaying a user message related to the computer security application; and
executing a set of computer instructions related to operation of the computer security application.
7. The method of claim 1 , wherein providing at least some of the control statements to the plurality of clients comprises:
providing a root control statement to the first client, wherein the root statement represents a starting point into the script that directs the first client to a sequence of subsequent control statements in the first subset of control statements.
8. The method of claim 7 , wherein providing at least some of the control statements to the plurality of clients further comprises, responsive to receiving a result of evaluating the root control statement from first client, providing the subsequent control statements in the first subset of control statements to the first client.
9. The method of claim 1 , wherein providing at least some of the control statements to the plurality of clients further comprises:
providing the set of control statements in the hierarchy to the first client, wherein the first path of the first set of control statements is identified from the set of control statements.
10. A non-transitory computer-readable storage medium storing instructions for facilitating testing of a feature of a computer security application, the instructions when executed by a processor causing the processor to perform steps including:
obtaining, at a server, a script comprising a set of control statements in a hierarchy, the hierarchy having a first path including a first subset of control statements and a second path including a second subset of control statement, wherein which path a client device traverses depends on evaluation of one or more conditions;
receiving, from a plurality of clients, respective queries for the control statements;
providing at least some of the control statements to the plurality of clients in response to the queries, wherein a first client of the plurality of clients follows the first path of the plurality of paths and a second client of the plurality of clients follows the second path of the plurality of paths;
receiving telemetry data from the first and the second clients, wherein the first and the second clients send telemetry data to the server as specified respectively by the first and second subsets of control statements, the telemetry data indicative of an effect of enabling the feature on client applications associated with the first and the second clients, respectively;
evaluating the telemetry data to determine a test outcome; and
executing an action based on the test outcome.
11. The non-transitory computer-readable storage medium of claim 10 , wherein evaluating the telemetry data to determine a test outcome comprises:
determining whether to roll back the feature or more widely distribute the feature; and
responsive to determining to more widely distribute the feature, causing the feature to be enabled on an additional set of clients outside the subset of clients.
12. The non-transitory computer-readable storage medium of claim 10 , wherein each of the control statements includes a condition to be evaluated by a client and an action to be performed by the client based on evaluation of the condition, wherein the action for a first control statement comprises at least one of:
obtaining a second control statement for execution by the client, enabling or disabling the feature of the computer security application at the client, and sending telemetry data associated with the client to the server.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the control statements comprise a client-side control statement in which the condition is evaluated by the client based on locally-obtained data.
14. The non-transitory computer-readable storage medium of claim 12 , wherein the control statements comprise a server-side control statement in which the condition is evaluated on behalf of the client by the server.
15. The non-transitory computer-readable storage medium of claim 10 , wherein enabling the feature comprises at least one of:
displaying a user message related to the computer security application; and
executing a set of computer instructions related to operation of the computer security application.
16. The non-transitory computer-readable storage medium of claim 10 , wherein providing at least some of the control statements to the plurality of clients comprises:
providing a root control statement to the first client, wherein the root statement represents a starting point into the script that directs the first client to a sequence of subsequent control statements in the first subset of control statements.
17. The non-transitory computer-readable storage medium of claim 16 , wherein providing at least some of the control statements to the plurality of clients further comprises, responsive to receiving a result of evaluating the root control statement from first client, providing the subsequent control statements in the first subset of control statements to the first client.
18. A computer system comprising:
a processor; and
a non-transitory computer-readable storage medium storing instructions for facilitating testing of a feature of a computer security application, the instructions when executed by the processor causing the processor to perform steps including:
obtaining, at a server, a script comprising a set of control statements in a hierarchy, the hierarchy having a first path including a first subset of control statements and a second path including a second subset of control statement, wherein which path a client device traverses depends on evaluation of one or more conditions;
receiving, from a plurality of clients, respective queries for the control statements;
providing at least some of the control statements to the plurality of clients in response to the queries, wherein a first client of the plurality of clients follows the first path of the plurality of paths and a second client of the plurality of clients follows the second path of the plurality of paths;
receiving telemetry data from the first and the second clients, wherein the first and the second clients send telemetry data to the server as specified respectively by the first and second subsets of control statements, the telemetry data indicative of an effect of enabling the feature on client applications associated with the first and the second clients, respectively;
evaluating the telemetry data to determine a test outcome; and
executing an action based on the test outcome.
19. The non-transitory computer-readable storage medium of claim 10 , wherein providing at least some of the control statements to the plurality of clients further comprises:
providing the set of control statements in the hierarchy to the first client, wherein the first path of the first set of control statements is identified from the set of control statements.
20. The computer system of claim 18 , wherein providing at least some of the control statements to the plurality of clients further comprises:
providing a root control statement to the first client, wherein the root statement represents a starting point into the script that directs the first client to a sequence of subsequent control statements in the first subset of control statements; and
responsive to receiving a result of evaluating the root control statement from first client, providing the subsequent control statements in the first subset of control statements to the first client.