Optimal online adaptive controller
Various embodiments are disclosed for optimal online adaptive control. One such method includes a cost function determination by a critic network coupled to the system under control. The cost function is one produces a minimum value for a cost of the system under control when applied by an action network. The method also includes a control input determination by an action network. The control input determination uses the cost function to determine a control input to apply to the system under control. The control input is one that produces the minimum value for the cost of the system under control. The method also includes simultaneously tuning respective parameters of the critic network and the action network by applying respective tuning laws that do not involve the system dynamics function f(x) for the system under test.
1. A method of adaptively controlling a continuous-time system under control, the continuous-time system under control being described by a system dynamics function f(x), the method comprising:
in a critic network coupled to the continuous-time system under control, determining a cost function that produces a minimum value for a cost of the continuous-time system under control when applied by an action network;
in the action network also coupled to the continuous-time system under control, determining, using the cost function, a control input to apply to the continuous-time system under control that produces the minimum value for the cost of the continuous-time system under control; and
tuning respective parameters of the critic network and the action network together and continuously in time by applying respective tuning laws that do not involve the system dynamics function f(x).
2. The method of claim 1 , wherein the tuning laws use an integral reinforcement learning (IRL) form of a Bellman equation.
3. The method of claim 1 , wherein the only data obtained from the continuous-time system under test that is used during the tuning is measured input/output data from the continuous-time system under test.
4. The method of claim 1 , wherein the tuning law for the critic network is given by
W
^
.
1
=
-
a
1
Δ
ϕ
(
x
(
t
)
)
T
(
1
+
Δ
ϕ
(
x
(
t
)
)
T
Δ
ϕ
(
x
(
t
)
)
)
2
[
∫
t
-
T
t
(
Q
(
x
)
+
u
T
Ru
)
ⅆ
τ
+
Δ
ϕ
(
x
(
t
)
)
T
W
^
1
]
,
where {circumflex over ({dot over (w)} 1 corresponds to the tuned parameters, φ(t) is a set of basis functions associated with the continuous-time system under control, u T Ru is the energy used when applying the control input, and Q(x) is an amount of energy for a plurality of system states associated with the continuous-time system under control.
5. The method of claim 1 , wherein the tuning law for the critic network is given by
W
^
.
2
=
-
a
2
(
F
2
W
^
2
-
F
1
Δ
ϕ
(
x
(
t
)
)
T
W
^
1
)
-
1
4
a
2
D
_
1
(
x
)
W
^
2
Δ
ϕ
(
x
(
t
)
)
T
(
1
+
Δ
ϕ
(
x
(
t
)
)
T
Δ
ϕ
(
x
(
t
)
)
)
2
W
^
1
]
,
where {circumflex over ({dot over (w)} 2 corresponds to the tuned parameters, φ(t) is a set of basis functions associated with the continuous-time system under control, u T Ru is the energy used when applying the control input, and Q(x) is an amount of energy for a plurality of system states associated with the continuous-time system under control.
6. The method of claim 1 , wherein the tuning is performed when the continuous-time system under control is online.
7. The method of claim 1 , wherein the cost function is learned by the critic network in real time.
8. The method of claim 1 , wherein the continuous-time system under control is further described by a set of system dynamics and the tuning law for the critic network uses only a partially complete model of the system dynamics.
9. The method of claim 1 , wherein the cost of the continuous-time system under control corresponds to an amount of control energy used by the continuous-time system under control.
10. The method of claim 1 , wherein the cost of the continuous-time system under control corresponds to an amount of fuel used by the continuous-time system under control.
11. The method of claim 1 , wherein the continuous-time system under control is further described as a series of system states and the cost of the continuous-time system under control corresponds to an amount of energy used in the system states.
12. A controller for a continuous-time system under control, the continuous-time system under control being described by a system dynamics function f(x), the controller comprising:
an inner-loop feedback controller located in a feed forward path with respect to the continuous-time system under control;
an outer-loop feedback controller in the feed forward path; and
a policy iterator,
wherein the inner-loop feedback controller comprises the continuous-time system under control and an action network in a feedback loop coupled to the continuous-time system under control,
wherein the outer-loop feedback controller comprises a critic network in the feedback loop coupled to the action network, and
wherein the policy iterator is operable to tune respective parameters of the critic network and the action network together and continuously in time by applying respective tuning laws that do not involve the system dynamics function f(x).
13. The controller of claim 12 , wherein the only data used by the policy iterator from the continuous-time system under test is measured input/output data.
14. The controller of claim 12 , wherein the action network is operable to determine a control input to apply to the continuous-time system under control, wherein the action network learns the control input in real time.
15. The controller of claim 12 , wherein the critic network is operable to determine a minimum cost function to apply to the action network, wherein the critic network learns the minimum cost function in real time.
16. A non-transitory computer-readable medium embodying a control program executable in at least one computing device to control a continuous-time system under control, the continuous-time system under control being described by a system dynamics function f(x), the control program comprising:
critic network code that determines a cost function that produces a minimum value for a cost of the continuous-time system under control when applied by an action network;
action network code that determines, using the cost function, a control input to apply to the continuous-time system under control that produces the minimum value for the cost of the continuous-time system under control; and
policy iterator code that tunes respective parameters of the critic network and the action network together and continuously in time by applying respective tuning laws that do not involve the system dynamics function f(x).
17. The non-transitory computer-readable medium of claim 16 , wherein the tuning law for the critic network is given by
W
^
.
1
=
-
a
1
Δ
ϕ
(
x
(
t
)
)
T
(
1
+
Δ
ϕ
(
x
(
t
)
)
T
Δ
ϕ
(
x
(
t
)
)
)
2
[
∫
t
-
T
t
(
Q
(
x
)
+
u
T
Ru
)
ⅆ
τ
+
Δ
ϕ
(
x
(
t
)
)
T
W
^
1
]
,
where {circumflex over ({dot over (w)} 1 corresponds to the tuned parameters, φ(t) is a set of basis functions associated with the continuous-time system under control, u T Ru is the energy used when applying the control input, and Q(x) is an amount of energy for a plurality of system states associated with the continuous-time system under control.
18. The non-transitory computer-readable medium of claim 16 , wherein the tuning law for the critic network is given by
W
^
.
2
=
-
a
2
(
F
2
W
^
2
-
F
1
Δ
ϕ
(
x
(
t
)
)
T
W
^
1
)
-
1
4
a
2
D
_
1
(
x
)
W
^
2
Δ
ϕ
(
x
(
t
)
)
T
(
1
+
Δ
ϕ
(
x
(
t
)
)
T
Δ
ϕ
(
x
(
t
)
)
)
2
W
^
1
]
,
where {circumflex over ({dot over (w)} 2 corresponds to the tuned parameters, φ(t) is a set of basis functions associated with the continuous-time system under control, u T Ru is the energy used when applying the control input, and Q(x) is an amount of energy for a plurality of system states associated with the continuous-time system under control.
19. The non-transitory computer-readable medium of claim 16 , wherein the continuous-time system under control is further described by a set of system dynamics and the tuning law for the critic network code uses only a partially complete model of the system dynamics.
20. The non-transitory computer-readable medium of claim 16 , wherein the policy iterator code performs the tuning of the critic network and the action network together when the continuous-time system under control is online.