Optimal charging and discharging control for hybrid energy storage system based on reinforcement learning
Systems and methods are disclosed to manage a microgrid with a hybrid energy storage system (HESS) includes deriving a dynamic model of a bidirectional-power-converter (BPC)-interfaced HESS; applying a first neural network (NN) to estimate a system dynamic; and applying a second NN to calculate an optimal control input for the HESS through online learning based on the estimated system dynamics.
1. A method to manage a microgrid with a hybrid energy storage system (HESS), comprising:
deriving a dynamic model of a bidirectional-power-converter (BPC)-interfaced HESS;
applying a first neural network (NN) to estimate system dynamics of a nonlinear HESS model with AC and DC buses interconnected through a bidirectional DC/AC voltage source converter (VSC) and a PV array, and ultra-capacitor (UC) and lithium-ion batteries (LIB) connected on the DC bus, wherein the first NN approximates HESS model parameters using a perturbation term associated with the system dynamics and minimizes NN reconstruction errors for the nonlinear HESS model; and
applying a second NN to calculate an optimal control input for the HESS through online learning based on the estimated system dynamics.
2. The method of claim 1 , comprising testing the hybrid HESS with a non-linear system.
3. The method of claim 2 , by defining the tracking error as e(t)=U c (t)−U c * with U c * being a constant reference, comprising determining
e
.
(
t
)
=
f
(
e
(
t
)
)
+
g
(
t
)
u
(
t
)
+
D
(
t
)
=
W
e
*
τ
σ
1
(
e
)
+
W
u
*
τ
σ
1
(
u
)
+
W
d
*
τ
1
+
o
1
`
=
W
1
*
τ
σ
1
(
e
,
u
)
+
o
1
`
where ė(t)={dot over (U)} c (t)−{dot over (U)} c *={dot over (U)} c (t)−0={dot over (U)} c (t); f(e)=(R sc /L f −1/C uc R pc )U c and g(t)=R sc V t /L f are unknown nonlinear system dynamics; u(t)=α is the control input; D(t)=[(R pc +R sc )/C uc R p −R sc R f /L f ]I c is a perturbation term of a system with D(0)=0 and is bounded as ∥D(t)∥≤d max ;
α
=
1
1
-
D
,
∀S 1 is off and
α
=
1
D
,
∀S 2 is off; D is a duty ratio of PWM; R 1 , L f and C f are filter resistance, inductance and capacitance, respectively; R pc and R sc are internal parallel and series resistance of ultra-capacitor (UC), respectively, R pb and R sb are the internal parallel and series resistance of lithium-ion batteries (LIB), respectively; V c and U c are internal and external voltages of energy storage unit (ESU), respectively; V t is terminal bus voltage; W 1 *=[W e * W u * W d *] T ∈ N×1 are ideal unknown weights of NN identifier approximating system dynamics f(e), g(t) and D; σ 1 (e,u)=[σ 1 (e) σ 1 (u) 1]∈ N×1 is activation function where 1 represents a vector of ones, N is the number of hidden-layer neurons, {grave over (σ)} 1 is NN reconstruction error, W 1 * and {grave over (σ)} 1 are bounded as ∥W 1 *∥≤W 1M and ∥{grave over (σ)} 1 ∥≤{grave over (σ)} 1M , respectively, and D is a perturbation term associated with system dynamics and satisfies conditions to be estimated by the first NN.
4. The method of claim 2 , comprising tracking error dynamics estimation as {right arrow over (ė)}(t)=Ŵ 1 T σ 1 (e,u)+k 1 {tilde over (e)}(t) where Ŵ 1 =[Ŵ e Ŵ u Ŵ d ] T ∈ N×1 are the estimated NN identifier weights and k, is the selected parameter, where [Ŵ e Ŵ u Ŵ d ] is used to estimate weights [Ŵ e Ŵ u Ŵ d ] of NN identifier approximating system dynamics; σ 1 (e,u) is an activation function of tracking error e and control action u, N is the number of hidden-layer neurons.
5. The method of claim 2 , comprising defining {tilde over (e)}(t)=e(t)−ê(t) as an estimation error of tracking error, and determining dynamics of {tilde over (e)}(t) as
e
~
.
(
t
)
=
e
.
(
t
)
-
ê
.
(
t
)
=
W
1
*
T
σ
1
(
e
,
u
)
+
q
`
-
W
^
1
T
σ
1
(
e
,
u
)
-
k
1
e
~
(
t
)
=
W
~
1
T
σ
1
(
e
,
u
)
+
q
`
-
k
1
e
~
(
t
)
where the NN weight estimation error is defined as {tilde over (W)} 1 (t)=W 1 *−Ŵ 1 (t), and {tilde over ({dot over (W)})} 1 (t)=−{circumflex over ({dot over (W)})}; k 1 is selected parameter to maintain NN identifier stability; σ 1 (e,u) is activation function of tracking error e and control action u, N is number of hidden-layer neurons, {grave over (σ)} 1 is NN reconstruction error, W 1 * and {grave over (σ)} 1 are bounded as ∥W 1 *∥≤W 1M and ∥{grave over (σ)} 1 ∥≤{grave over (σ)} 1M , respectively.
6. The method of claim 2 , comprising forcing estimated NN identifier weight Ŵ 1 (t) to converge to a target weight W 1 * by updating Ŵ 1 as {circumflex over ({dot over (W)})} 1 =−k 2 Ŵ 1 T σ 1 (e,u)+σ 2 (e,u)e(t), where k 2 is a positive tuning parameter of NN identifier, σ 1 (e,u) is activation function of tracking error e and control action u, N is number of hidden-layer neurons, {grave over (σ)} 1 is NN reconstruction error, W 1 * and {grave over (σ)} 1 are bounded as ∥W 1 *∥≤W 1M and ∥{grave over (σ)} 1 ∥≤{grave over (σ)} 1M , respectively.
7. The method of claim 1 , comprising determining an approximated Hamiltonian of optimal control problem as:
H
^
(
e
,
u
^
,
V
^
)
=
Q
(
e
)
+
1
4
W
^
2
T
″
σ
2
W
^
u
R
-
1
W
^
u
T
″
σ
2
T
W
^
2
+
″
σ
2
T
W
^
2
[
W
^
1
T
(
t
)
σ
1
(
e
,
u
)
+
k
1
e
~
(
t
)
]
+
1
4
W
^
2
T
″
σ
2
″
σ
2
T
W
^
2
+
d
max
2
where Q(e)=e T Pe is a positive definite function of e with P being a symmetric positive definite matrix; Ŵ u T is estimated weight of dynamic; Ŵ 2 T ∈ 1×N are estimated weights of a cost function estimator; σ 1 and σ 2 are activation functions of tracking error e and control action u; k 1 is selected parameter to maintain NN identifier stability; d max is constant bound of disturbance; û is estimation of optimal control action and {circumflex over (V)}(e) is estimation of cost function; {tilde over (e)}(t) is estimation error of tracking error.
8. The method of claim 1 , comprising tuning a NN weight of cost function estimator as
W
2
^
.
=
k
3
2
Θ
(
e
,
u
^
)
″
∇
σ
2
W
^
u
R
-
1
W
^
u
T
J
1
∂
e
T
-
k
4
ω
H
^
(
1
+
ω
T
ω
)
2
where k 3 and k 4 are designed control coefficients, ω=−[“σ 2 Ŵ u R −1 Ŵ u T ” σ 2 T W 2 ]/2, and Θ(e,û) is an index operator given by
Θ
(
e
,
u
^
)
=
{
0
,
∀
J
.
1
=
J
1
∂
e
T
e
.
<
0
1
,
otherwise
where J 1 is an unbounded Lyapunov candidate and J 1∂e is its partial derivative with respect to error e; Ŵ u T is estimated weight of dynamic; Ŵ 2 T ∈ 1×N are estimated weights of a cost function estimator; σ 2 is an activation function of tracking error e and control action u; û is estimation of optimal control action; and R is a symmetric positive definite matrix.
9. The method of claim 8 , wherein J 1∂e comprises
∥ ė∥≤c 1 ∥e ∥≡( c 2 ∥J 1∂e ∥) 1/4
where c 1 and c 2 are constants and J 1∂e is its partial derivative with respect to error e.
10. The method of claim 8 , where ∥J 1∂e ∥ satisfies a general bound where
J
1
=
1
5
(
e
T
e
)
5
2
.
where J 1 is an unbounded Lyapunov candidate and J 1∂e is its partial derivative with respect to error e.
11. The method of claim 1 , comprising analyzing both grid-tied and islanded modes of the microgrids using measurement-based neural networks.
12. The method of claim 1 , wherein in a grid-tied mode, the main grid is considered as an infinite source which maintains bus voltage and reactive power at a point of common coupling (PCC) through a voltage source converter (VSC).
13. The method of claim 1 , wherein while in islanded mode, diesel generators (DGs) are deployed to maintain the bus voltage at common coupling (PCC).
14. The method of claim 1 , comprising controlling a photovoltaic (PV) array under a maximum power point tracking (MPPT) mode to maximize a renewable distribution energy resource (DER) utilization.
15. The method of claim 1 , comprising formulating an optimal control problem of hybrid energy storage system (HESS) using a reinforcement learning (RL) method to reduce the disturbances caused by charging & discharging (C&D) of one or more energy storage devices.
16. The method of claim 1 , comprising adapting different system dynamics based on input/output data.
17. The method of claim 1 , wherein the microgrid consists of AC and DC buses interconnected through a bidirectional DC/AC voltage source converter (VSC) and a PV array operates in maximum power point tracking (MPPT) mode, and the hybrid energy storage system (HESS) with both ultra-capacitor (UC) and lithium-ion batteries (LIB) are connected on a DC bus, and grid-tied and islanded operation mode switching of the microgrid is realized by operating the circuit breaker (CB).
18. A method to manage a microgrid with a hybrid energy storage system (HESS), comprising:
deriving a dynamic model of a bidirectional-power-converter (BPC)-interfaced HESS;
applying a first neural network (NN) to estimate a system dynamics; and
applying a second NN to calculate an optimal control input for the HESS through online learning based on the estimated system dynamics; and
representing a cost function V*(e) by a single-layer NN on set Ω as V*(e)=W 2 * T σ 2 (e,u)+{grave over (σ)} 2 where W 2 * T ∈ 1×N are ideal unknown weights of a cost function estimator, σ 2 (e,u) is an activation function of tracking error e and control action u, {grave over (σ)} 2 is an NN reconstruction error, W 2 * and {grave over (σ)} 2 are bounded as ∥W 2 *∥≤W 2M and ∥{grave over (σ)} 2 ∥≤{grave over (σ)} 2M with W 2M and {grave over (σ)} 2M as two constant bounds, respectively.
19. The method of claim 18 , comprising applying the cost function estimator as {circumflex over (V)}(e)=Ŵ 2 T σ 2 (e,u), where {circumflex over (V)}(e) is estimation of cost function V*(e), Ŵ 2 T ∈ 1×N are estimated weights of a cost function estimator and σ 2 (e,u) is an activation function of tracking error e and control action u.
20. A method to manage a microgrid with a hybrid energy storage system (HESS), comprising:
deriving a dynamic model of a bidirectional-power-converter (BPC)-interfaced HESS;
applying a first neural network (NN) to estimate a system dynamics; and
applying a second NN to calculate an optimal control input for the HESS through online learning based on the estimated system dynamics; and
deriving an estimated optimal control policy based on the two NNs as
u
^
=
-
1
2
R
-
1
W
^
u
T
″
σ
2
(
e
,
u
)
T
W
^
2
where “σ 2 (e,u)=∂σ 2 (e,u)/∂e is a partial derivative of σ 2 (e,u) with respect to tracking error e; R is a symmetric positive definite matrix; Ŵ u T is estimated weight of dynamic; Ŵ 2 T ∈ 1×N are estimated weights of a cost function estimator and σ 2 (e,u) is an activation function of tracking error e and control action u.