Method for creating predictive knowledge structures from experience in an artificial agent
Building a forecast for an autonomous agent at least comprises assigning a selected parameter of the autonomous agent to a scalar variable, adding a new policy to a set of policies where the new policy maps internal states of the autonomous agent to actions of the autonomous agent in which the mapping may optimize the scalar variable, and adding a new forecast to a set of forecasts where the forecast at least comprises a prediction regarding future values of the scalar variable following execution of the new policy, regardless whether the agent ever actually chooses to take actions in accordance with said new policy. A state of the autonomous agent may be evaluated following completion of each of the agent's actions by comparing the agent's state information with the predicted values of one or more forecasts. Whether to build an additional forecast may be determined based on the evaluation.
1. A method comprising the steps of:
building a forecast for an autonomous agent, said building at least comprising:
selecting a policy from a set of policies, said policy mapping states of said autonomous agent to actions of said autonomous agent; and
automatically choosing and adding a new forecast to a set of forecasts, said new forecast at least comprising a prediction regarding future states of said autonomous agent during execution and termination of said policy based on a closed-loop sequence of actions, where the policy is considered with conditions for termination of the policy;
evaluating a state of said autonomous agent following termination of said policy, said evaluation at least comprising comparing said state with said prediction;
creating a new policy that optimizes a function over observable signals and forecasts;
building a further new forecast, said further new forecast at least comprising a further prediction regarding future states of said autonomous agent during execution and termination of said new policy based on a closed-loop sequence of actions, where the new policy is considered with conditions for termination of the new policy;
evaluating a state of said autonomous agent following termination of said new policy, said evaluation at least comprising comparing said state with said further prediction; and
determining whether to build an additional forecast, said determining optionally at least in part based on said evaluation.
2. The method as recited in claim 1 , further comprising the steps of:
determining if said forecast is ineffective; and pruning said forecast from said set of forecasts upon said determination.
3. The method as recited in claim 1 , in which said building the forecast further comprises:
building a new policy, said building at least comprising:
selecting a state value of said autonomous agent; and
adding a new policy to the set of policies, said new policy mapping states of said autonomous agent to actions of said autonomous agent, said actions optimizing said state value.
4. The method as recited in claim 1 , in which said set of forecasts comprises a hierarchical structure.
5. The method as recited in claim 3 , in which said new policy further comprises starting and stopping criteria.
6. The method as recited in claim 1 , in which a state, set of states or state value predicted by any forecast in said set of forecasts is associated with at least one of said policies in said set of policies.
7. The method as recited in claim 3 , in which said selected state value comprises at least one of an observation signal, a forecast of interest, a function of a combination of observation signals, and a function of forecast values in said set of forecasts.
8. The method as recited in claim 1 , in which said step of determining whether to terminate the policy is further based on a threshold value.
9. The method as recited in claim 1 , in which said set of forecasts and said set of policies comprise a hierarchical structure.
10. A method comprising:
steps for building a forecast for an autonomous agent, said steps for building at least comprising selecting a policy from a set of policies, said policy mapping states of said autonomous agent to actions of said autonomous agent, and automatically choosing and adding a new forecast to a set of forecasts, said new forecast at least comprising a prediction regarding future states of said autonomous agent during execution and termination of said policy based on a closed-loop sequence of actions, where the policy is considered with conditions for termination of the policy;
steps for evaluating a state of said autonomous agent following termination of said policy;
steps for creating a new policy that optimizes a function over observable signals and forecasts;
steps for building a further new forecast, said further new forecast at least comprising a further prediction regarding future states of said autonomous agent during execution and termination of said new policy based on a closed-loop sequence of actions, where the new policy is considered with conditions for termination of the new policy;
steps for evaluating a state of said autonomous agent following termination of said new policy, said evaluation at least comprising comparing said state with said further prediction; and
steps for determining whether to build an additional forecast.
11. The method as recited in claim 10 , further comprising:
steps for determining if said forecast is ineffective; steps for pruning said forecast from said set of forecasts upon said determination.
12. A non-transitory computer-readable storage medium with an executable program stored thereon, wherein the program instructs one or more processors to perform the following steps:
building a forecast for an autonomous agent, said building at least comprising:
selecting a policy from a set of policies, said policy mapping states of said autonomous agent to actions of said autonomous agent; and
automatically choosing and adding a new forecast to a set of forecasts, said new forecast at least comprising a prediction regarding future states of said autonomous agent during execution and termination of said policy based on a closed-loop sequence of actions, where the policy is considered with conditions for termination of the policy;
evaluating a state of said autonomous agent following termination of said policy, said evaluation at least comprising comparing said state with said prediction;
creating a new policy that optimizes a function over observable signals and forecasts;
building a further new forecast, said further new forecast at least comprising a further prediction regarding future states of said autonomous agent during execution and termination of said new policy based on a closed-loop sequence of actions, where the new policy is considered with conditions for termination of the new policy;
evaluating a state of said autonomous agent following termination of said new policy, said evaluation at least comprising comparing said state with said further prediction; and
determining whether to build an additional forecast, said determining optionally based at least in part on said evaluation.
13. The program instructing the one or more processors as recited in claim 12 , further comprising the steps of: determining if said forecast is ineffective; and pruning said forecast from said set of forecasts upon said determination.
14. The program instructing the one or more processors as recited in claim 12 , in which said building the forecast further comprises:
building a new policy, said building at least comprising:
selecting a state value of said autonomous agent; and
adding a new policy to the set of policies, said new policy mapping states of said autonomous agent to actions of said autonomous agent, said actions optimizing said state value.
15. The program instructing the one or more processors as recited in claim 12 , in which said set of forecasts comprises a hierarchical structure.
16. The program instructing the one or more processors as recited in claim 14 , in which said new policy further comprises starting and stopping criteria.
17. The program instructing the one or more processors as recited in claim 12 , in which a state, set of states or state value predicted by any forecast in said set of forecasts is associated with at least one of said policies in said set of policies.
18. The program instructing the one or more processors as recited in claim 14 , in which said selected state value comprises at least one of an observation signal, a forecast of interest, a function of a combination of observation signals, and a function of forecast values in said set of forecasts.
19. The program instructing the one or more processors as recited in claim 14 , in which said step of determining whether to terminate the policy is further based on a threshold value.
20. The program instructing the one or more processors as recited in claim 12 , in which said set of forecasts and said set of policies comprise a hierarchical structure.