Determining causal models for controlling environments
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining causal models for controlling environments. One of the methods includes repeatedly selecting control settings for the environment based on (i) a causal model that identifies causal relationships between possible settings for controllable elements in the environment and environment responses that reflect a performance of the control system in controlling the environment and (ii) current values of a set of internal parameters; and during the repeatedly selecting: monitoring environment responses to the selected control settings; determining, based on the environment responses, an indication that one or more properties of the environment have changed; and in response, modifying the current values of one or more of the internal parameters.
1. A method for controlling an environment, the method comprising:
repeatedly selecting control settings for the environment based on (i) a causal model that identifies causal relationships between possible settings for controllable elements in the environment and environment responses that reflect a performance of the control system in controlling the environment and (ii) current values of a set of internal parameters; and
during the repeatedly selecting:
monitoring environment responses to the selected control settings;
determining, based on the environment responses, an indication that one or more properties of the environment have changed; and
in response, modifying the current values of one or more of the internal parameters and controlling the environment based on the one or more modified internal parameters.
2. The method of claim 1 ,
wherein the internal parameters include a first set of parameters that define how frequently control settings are selected based on the causal model relative to being selected based on baseline probabilities for the possible settings for the controllable elements that are independent of the causal model, and
wherein modifying the current values of the one or more parameters comprises modifying the current values of the first set of parameters to decrease how frequently control settings are selected based on the current causal model relative to being selected based on baseline probabilities.
3. The method of claim 1 ,
wherein the internal parameters include a second set of parameters that defines which environment responses that have been previously received are included in generating the causal model, and
wherein modifying the current values comprises modifying the current values of the second set of parameters to decrease the number of system responses that are included in generating the causal model.
4. The method of claim 1 ,
wherein the internal parameters include a third set of parameters parameter that defines how procedural instances within the environment are clustered into a plurality of clusters for use in selecting control settings for the procedural instances, and
wherein modifying the current values of the one or more parameters comprises modifying the current values of the third set of parameters to decrease the number of clusters.
5. The method of claim 4 further comprising maintaining a separate set of ratio parameters per cluster and per controllable element so that different clusters use different ratios when selecting control settings for the same controllable element.
6. The method of claim 5 wherein the clustering parameters define hyperparameters of a clustering technique, wherein the hyperparameters include a number of procedural instances in each cluster and environmental characteristics used to cluster the procedural instances.
7. The method of claim 1 , wherein determining, based on the system responses, an indication that one or more properties of the environment have changed comprises:
determining that a difference between (i) system performance when control settings are selected based on the causal model and the current values of the internal parameters and (ii) system performance when control settings are selected based on baseline probabilities has decreased.
8. The method of claim 1 , wherein determining, based on the environment responses, an indication that one or more properties of the environment have changed comprises:
determining that the causal relationships for different possible settings for a controllable element of the environment in the causal model are not normally distributed.
9. The method of claim 8 , wherein the internal parameters include a second set of parameters that defines which environment responses that have been previously received are included in generating the causal model, and
wherein modifying the current values comprises modifying the current values of the second set of parameters to decrease the number of environment responses that are included in generating the causal model to a generate a causal model that has normally distributed causal relationships.
10. The method of claim 1 , wherein modifying the current values of one or more of the sets of internal parameters comprises modifying current values of a set of internal parameters that define a range of possible values from which a parameter that defines a number of environment responses that are included in generating the causal model is sampled.
11. The method of claim 10 , wherein modifying the current values comprises:
modifying the current values to adjust the range of possible values.
12. The method of claim 10 , wherein modifying the current values comprises:
modifying values that define probabilities for each of the possible values that are used to sample the parameter that defines a number of environment responses that are included in generating the causal model.
13. The method of claim 1 , further comprising, during the repeatedly selecting:
updating the causal model based on the monitored environment responses.
14. The method of claim 1 further comprising multiple causal models that each correspond to different segments of the environment that share certain characteristics.
15. The method of claim 14 wherein the causal relationships have confidence intervals that do not overlap.
16. The method of claim 1 further comprising updating the internal parameters use both a heuristic-based approach and by stochastic variation.
17. The method of claim 1 further comprising a separate set of data inclusion window parameters for each controllable element of the environment to allow use of different data inclusion windows for different controllable elements when updating the causal model.
18. A system comprising:
at least one computing device comprising one or more processors; and
at least one memory coupled to at least one of the one or more processors, wherein the at least one memory comprises instructions that configure the at least one computing device to:
repeatedly select control settings for the environment based on (i) a causal model that identifies causal relationships between possible settings for controllable elements in the environment and environment responses that reflect a performance of the control system in controlling the environment and (ii) current values of a set of internal parameters; and
during the repeatedly selection:
monitor environment responses to the selected control settings;
determine, based on the environment responses, an indication that one or more properties of the environment have changed; and
in response, modify the current values of one or more of the internal parameters and control the environment based on the one or more modified internal parameters.
19. One or more non-transitory computer-readable medium storing instructions that, when executed, configure at least one processor for:
repeatedly selecting control settings for the environment based on (i) a causal model that identifies causal relationships between possible settings for controllable elements in the environment and environment responses that reflect a performance of the control system in controlling the environment and (ii) current values of a set of internal parameters; and
during the repeatedly selecting:
monitoring environment responses to the selected control settings;
determining, based on the environment responses, an indication that one or more properties of the environment have changed; and
in response, modifying the current values of one or more of the internal parameters and controlling the environment based on the one or more modified internal parameters.