Inverse reinforcement learning with model predictive control
Described herein are systems and methods for inverse reinforcement learning to leverage the benefits of model-based optimization method and model-free learning method. Embodiments of a framework combining human behavior model with model predictive control are presented. The framework takes advantage of feature identification capability of a neural network to determine the reward function of model predictive control. Furthermore, embodiments of the present approach are implemented to solve the practical autonomous driving longitudinal control problem with simultaneous preference on safe execution and passenger comfort.
1. A computer-implement method for predicting parameters for a cost function by a neural network comprising:
given a set of recorded training data comprising states, control inputs, and observations:
responsive to a stop condition having not been reached, iterating a set of steps comprising:
predicting, using the neural network and a set of observations from the set of recorded training data, a set of parameters for the cost function to be used by a model predictive control (MPC) process to predict a future control input for a system to achieve a desired behavior;
computing a representation matrix according to dynamics of the system and the recorded training data;
computing a null space matrix of the representation;
updating an auxiliary parameter that, when multiplied by the null space matrix, produces an approximated set of parameters for the cost function, by minimizing a difference between the approximated set of parameters for the cost function and the predicted set of parameters for the cost function;
using the updated auxiliary parameter to calculate a loss of the neural network; and
updating parameters of the neural network using the loss by back propagation; and
responsive to a stop condition having been reached, outputting the neural network as a trained neural network.
2. The computer-implement method of claim 1 wherein the null space matrix has columns spanning the null space of the representation matrix.
3. The computer-implement method of claim 1 wherein the representation matrix is a combination of a system behavior matrix and a history matrix, the system behavior matrix is constructed using dynamics of the system, recorded states and recorded current control inputs, the history matrix is constructed from the recorded states and control inputs.
4. The computer-implement method of claim 3 wherein the dynamics of the system are first order partial derivatives of a system dynamic function with respect to states and control inputs respectively, the system dynamic function generates a next step of states given current states and current control inputs of the system.
5. The computer-implement method of claim 3 wherein the history matrix is a block-diagonal matrix augmented from current states and current control inputs.
6. The computer-implement method of claim 1 wherein the stop condition is that the loss is no greater than a loss threshold.
7. The computer-implement method of claim 1 wherein the difference between the approximated set of parameters for the cost function and the predicted set of parameters for the cost function is a least square difference.
8. A system for model predictive control (MPC) using a neural network comprising:
a neural network generating a cost function given an input of observations of environment, the neural network is pre-trained by:
receiving a set of recorded training data comprising states, control inputs, and observations;
initializing an auxiliary parameter and a loss for the neural network;
responsive to a stop condition having not been reached, iterating a set of steps comprising:
predicting, using the neural network and a set of observations from the set of recorded training data, a set of parameters for the cost function;
computing a representation matrix according to dynamics of the system and the recorded training data;
computing a null space matrix of the representation;
updating the auxiliary parameter that, when multiplied by the null space matrix, produces an approximated set of parameters for the cost function, by minimizing a difference between the approximated set of parameters for the cost function and the predicted set of parameters for the cost function;
using the updated auxiliary parameter to calculate a loss of the neural network; and
updating parameters of the neural network using the loss by back propagation; and
responsive to the stop condition having been reached, outputting the neural network as a trained neural network; and
a MPC module coupled to the neural network, the MPC module generates a future control input for the system, based on the input of observations and the cost function, to achieve a desired behavior.
9. The system of claim 8 wherein the null space matrix has columns spanning the null space of the representation matrix.
10. The system of claim 8 wherein the representation matrix is a combination of a system behavior matrix and a history matrix, the system behavior matrix is constructed using dynamics of the system, recorded states and recorded current control inputs, the history matrix is constructed from the recorded states and control inputs.
11. The system of claim 8 wherein the dynamics of the system are first order partial derivatives of a system dynamic function with respect to states and control inputs respectively, the system dynamic function generates a next step of states given current states and current control inputs of the system.
12. The system of claim 10 wherein the history matrix is a block-diagonal matrix augmented from current states and current control inputs.
13. The system of claim 8 wherein the difference between the approximated set of parameters for the cost function and the predicted set of parameters for the cost function is a least square difference.
14. A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes the steps for inverse reinforcement learning with model predictive control (MPC) to be performed comprising:
receiving a set of recorded training data comprising states, control inputs, and observations;
responsive to a stop condition having not been reached, iterating a set of steps comprising:
predicting, using a neural network and a set of observations from the set of recorded training data, a set of parameters for a cost function to be used by a model predictive control (MPC) process to predict a future control input for a system to achieve a desired behavior;
computing a representation matrix according to dynamics of the system and the recorded training data;
computing a null space matrix of the representation;
updating an auxiliary parameter that, when multiplied by the null space matrix, produces an approximated set of parameters for the cost function, by minimizing a difference between the approximated set of parameters for the cost function and the predicted set of parameters for the cost function;
using the updated auxiliary parameter to calculate a loss of the neural network; and
updating parameters of the neural network using the loss by back propagation; and
responsive to a stop condition having been reached, outputting the neural network as a trained neural network.
15. The non-transitory computer-readable medium or media of claim 14 wherein the null space matrix has columns spanning the null space of the representation matrix.
16. The non-transitory computer-readable medium or media of claim 14 wherein the representation matrix is a combination of a system behavior matrix and a history matrix, the system behavior matrix is constructed using dynamics of the system, recorded states and recorded current control inputs, the history matrix is constructed from the recorded states and control inputs.
17. The non-transitory computer-readable medium or media of claim 16 wherein the dynamics of the system are first order partial derivatives of a system dynamic function with respect to states and control inputs respectively, the system dynamic function generates a next step of states given current states and current control inputs of the system.
18. The non-transitory computer-readable medium or media of claim 16 wherein the history matrix is a block-diagonal matrix augmented from current states and current control inputs.
19. The non-transitory computer-readable medium or media of claim 14 wherein the stop condition is that the loss is no greater than a loss threshold.
20. The non-transitory computer-readable medium or media of claim 14 wherein the difference between the approximated set of parameters for the cost function and the predicted set of parameters for the cost function is a least square difference.