Federated reinforcement learning-based system and method for cooperative energy optimization

1 . A federated learning system for controlling a plurality of microgrids, comprising:

a plurality of household machine learning agents representing a first layer, including active consumer household agents configured to continuously learn model parameters and manage charging periods and discharging periods of household batteries;

a plurality of microgrid agents representing a second layer, each microgrid agent configured to maximize use of local energy based on a pricing policy for local transactions between households connected in the microgrid, including accessing power from other microgrids of the plurality of microgrids when there is insufficient local energy to cover local demand, and selling surplus energy to the other microgrids when power generation by the microgrid surpasses the local demand,

wherein each household machine learning agent is configured to sell energy to another household within its respective microgrid when there is an energy surplus at a household corresponding to the respective household machine learning agent and the energy price that is determined based on the pricing policy of the microgrid agent is at least a predetermined amount;

wherein the energy price that is determined based on the pricing policy of the microgrid has an objective of reducing carbon emission based on an objective function that maximize use of local energy within the microgrid;

a distributor agent representing a third layer configured to control load among the plurality of microgrids to minimize carbon emission by setting buy and sell prices for energy exchange within each of the plurality of microgrids based on a function that minimizes energy supply within the microgrids from non-renewable energy sources; and

a federated learning engine for obtaining and combining the model parameters from the plurality of household machine learning agents to update a global household machine learning agent; and

the objective function is represented as the following equation:

min

⁢

{

E

t

,

i

imp

3

(

r

t

sd

+

c

t

)

+

E

t

,

i

imp

2

⁢

r

t

sm

,

if

⁢

E

t

,

i

net

≥

0

,

E

t

,

i

exp

3

⁢

r

t

bd

+

E

t

,

i

exp

2

⁢

r

t

bm

,

if

⁢

E

t

,

i

net

<

0

,

wherein

E

t

,

i

imp

3

is a third layer energy import amount

wherein

r

t

sd

is a third layer energy sales price

wherein c t is a third layer emission amount

wherein

E

t

,

i

imp

2

is a second layer energy import amount

wherein

r

t

sm

is a second layer energy sales price

wherein

E

t

,

i

exp

3

is a third layer energy export amount

wherein

r

t

bd

is a third layer energy buying price

wherein

E

t

,

i

exp

2

is a second layer energy export amount

wherein

r

t

bm

is a second layer energy buying price

wherein

E

_

t

,

i

net

is a second layer net energy.

2 . The system of claim 1 , wherein each of the household machine learning agents is configured with an actor network and a critic network that undergo reinforcement learning.

3 . The system of claim 1 , wherein each of the household machine learning agents includes a trainable policy estimator.

4 . The system of claim 1 , wherein each of the household machine learning agents includes an advantage actor network and critic network, including an advantage function in which a value subtracted from a value of a Q function is used to re-calibrate rewards towards an average action,

wherein the Q function is an estimation of a reward for taking an action.

5 . The system of claim 1 , wherein a single agent is trained using neural network parameters that are shared by all of the plurality of household machine learning agents, in which each household machine learning agent uses observations of energy usage associated with the respective household to take energy buy or sell actions on behalf of the household.