Method and system for modelling and control partially measurable systems
A controller for controlling a system that includes a policy to control the system is provided. The controller includes an interface connected to the system, the interface acquires an action state and a measurement state via sensors measuring the system, a memory to store computer-executable program modules including a model learning module and a policy learning module, a processor that performs steps of the program modules. The steps include offline-modeling to generate offline-learning states based on the action state and measurement state using the model learning program, providing the offline states to the policy learning program to generate policy parameters, and updating the policy of the system to operate the system based on the policy parameters.
1. A controller for controlling motion of a vehicle system, comprising:
an interface operatively coupled to the vehicle system, wherein the interface is configured to acquire a motion state of the vehicle system and a measurement state via one or more motion sensors measuring the vehicle system;
a memory configured to store computer-executable program modules including a first model learning module and a policy learning module;
a processor configured to perform steps of the computer-executable program modules, the steps include:
offline-modeling to generate offline states based on the motion state of the vehicle system and the measurement state using a model learning program, wherein the model learning program is configured to consider at least one of a squared exponential (SE) kernel and a multiplicative polynomial (MP) kernel or a semi-parametrical (SP) kernel to model the evolution of the vehicle system,
wherein the SE kernel and the MP kernel use the Gaussian Process Regression (GPR) with Gaussian Processes (GP) input for model learning, wherein the GP input includes the motion state and an input of the vehicle system measured by the one or more motion sensors,
wherein the first model learning module includes an offline estimator and a second model learning module, wherein the offline state estimator estimates and provides the offline states to the second model learning module,
wherein the policy learning module generates Monte Carlo (MC) based particles, wherein the MC based particles are obtained by sampling from a probability distribution using a Monte Carlo approach to estimate expected cumulative cost from particle trajectories propagated through a learned model, and includes a model of an online estimator configured to generate particle online estimates based on particle measurements and prior particle online estimates, wherein the particle online estimates correspond to an approximation of the particle measurements;
providing the offline states to the policy learning module to generate policy parameters used in the MC based particles; and
updating the policy of the vehicle system to operate the vehicle system based on the generated policy parameters.
2. The controller of claim 1 , wherein the processor is further configured to perform a step of learning behavior of the vehicle system based on a speed-integration model.
3. The controller of claim 1 , wherein the policy learning module includes a sensor model configured to generate particle measurements based on the particle states.
4. The controller of claim 1 , wherein the second model-learning module is further configured to generate and provide offline learned states to the policy learning module, wherein the policy learning module generates policy parameters.
5. The controller of claim 4 , wherein the computer-executable program modules further comprise a policy optimization program that is based on a Dropout method and an early stopping strategy configured to improve the policy parameters.
6. The controller of claim 1 , wherein the offline states are generated based on-acausal filters, Kalman smoother or central difference velocity approximators.
7. The controller of claim 1 , wherein the controller is connected to a motion controller of the vehicle system, wherein the controller is configured to provide the policy parameters to the motion controller of the vehicle system to update a policy of the motion controller.
8. The controller of claim 7 , wherein the motion controller is configured to control suspensions of the vehicle system.
9. The controller of claim 7 , wherein the motion controller is configured to control actuators of the vehicle system.
10. The controller of claim 7 , further comprising a model of an online state estimator, wherein the model of the online state estimator performs policy optimization based on the offline states and generates the policy parameters.
11. The controller of claim 1 , wherein the second model-learning module generates and provides offline learned states to the policy learning module, wherein the policy learning module includes a policy optimization program that performs policy optimization based on offline states from the first model learning module and generates the policy parameters.
12. The controller of claim 11 , wherein the policy learning module includes a system model that generates current particle states based on previous particle states and the motion state of the vehicle system.
13. The controller of claim 1 , wherein the vehicle system comprises a motion controller and a vehicle, wherein the processor is further configured to submit the generated policy parameters to the motion controller to update a policy unit of the motion controller in accordance with the generated policy parameters.