Reinforcement learning for continued learning of optimal battery charging
Methods and systems of optimizing battery charging are disclosed. Battery state sensors are used to determine anode overpotential of a battery multiple times during multiple charge cycles. In a first phase, a reinforcement learning model (e.g., actor-critic model) is trained with rewards given throughout each charge cycle of the battery to optimize training. The reinforcement learning model can determine state-of-health characteristics of the battery over the charge cycles, and in a second phase, the reinforcement learning model is augmented accordingly. During this augmentation, the reinforcement learning model is trained with rewards given on a charge cycle-by-cycle basis, wherein rewards are given after looking at the charging optimization after the conclusion of each charge cycle. Commands are given to charge an on-field battery based on the augmented reinforcement learning model, associated state-of-health characteristics of the on-field battery are determined, and the reinforcement learning model is further augmented accordingly.
1 . A method of optimizing a charging of a vehicle battery based on both simulation data and field data, the method comprising:
receiving, at a simulator, simulation battery charge data associated with charging of a simulation vehicle battery;
determining, at the simulator, an anode overpotential of the simulation vehicle battery during each charge cycle of the simulation vehicle battery;
training a reinforcement learning model based on the anode overpotential, wherein the reinforcement learning model is trained with rewards given throughout each charge cycle of the simulation vehicle battery;
determining, via the reinforcement learning model, state-of-health characteristics of the simulation vehicle battery over multiple charge cycles;
augmenting the reinforcement learning model based on the state-of-health characteristics of the simulation vehicle battery over multiple charge cycles, wherein the reinforcement learning model is trained with rewards given on a charge cycle-by-cycle basis;
commanding a charging of an on-field vehicle battery based on an output of the augmented reinforcement learning model;
based on the charging of the on-field vehicle battery, determining state-of-health characteristics of the on-field vehicle battery;
further augmenting the augmented reinforcement learning model based on the state-of-health characteristics of the on-field vehicle battery, wherein the reinforcement learning model is further trained with rewards given based on multiple charge cycles; and
based on the further augmenting, outputting a trained reinforcement learning model configured to optimize charging of on-field vehicle batteries.
2 . The method of claim 1 , wherein the reinforcement learning model utilizes a twin delayed deep deterministic (TD3) policy gradient.
3 . The method of claim 1 , wherein the step of augmenting the reinforcement learning model based on the state-of-health characteristics of the simulation battery over multiple charge cycles further includes:
training the reinforcement learning model with rewards given only after each charge cycle has completed.
4 . The method of claim 1 , wherein the step of training the reinforcement learning model includes applying random noise.
5 . The method of claim 1 , wherein the step of commanding is sent via a remote server to a battery management system of an on-field vehicle.
6 . The method of claim 1 , wherein the step of augmenting the reinforcement learning model utilizes slowly varying Ornstein-Uhlenbeck noise.
7 . The method of claim 1 , further comprising:
transferring a reward system of the reinforcement learning model from (i) a first reward system in which the rewards are given throughout each charge cycle of the simulation vehicle battery to (ii) a second reward system in which the rewards are given on a charge cycle-by-cycle basis;
wherein the transferring uses an affine layer attached to a critic model.
8 . A system of optimizing a charging of a battery based on both simulation data and field data, the system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the one or more processors to:
receive, at a simulator, simulation battery charge data associated with charging of a simulation battery;
determine, at the simulator, an anode overpotential of the simulation battery during each charge cycle of the simulation battery;
train reinforcement learning model based on the anode overpotential, wherein the reinforcement learning model is trained with rewards given throughout each charge cycle of the simulation battery;
determine, via the reinforcement learning model, state-of-health characteristics of the simulation battery over multiple charge cycles;
augment the reinforcement learning model based on the state-of-health characteristics of the simulation battery over multiple charge cycles, wherein the reinforcement learning model is trained with rewards given on a charge cycle-by-cycle basis;
command a charging of an on-field battery based on an output of the augmented reinforcement learning model;
based on the charging of the on-field battery, determine state-of-health characteristics of the on-field battery;
further augment the augmented reinforcement learning model based on the state-of-health characteristics of the on-field battery, wherein the reinforcement learning model is further trained with rewards given based on multiple charge cycles; and
based on the further augmenting, output a trained reinforcement learning model configured to optimize charging of on-field batteries.
9 . The system of claim 8 , wherein the reinforcement learning utilizes a twin delayed deep deterministic (TD3) policy gradient.
10 . The system of claim 8 , wherein the augmenting of the reinforcement learning model based on the state-of-health characteristics of the simulation battery over multiple charge cycles further includes:
training the reinforcement learning model with rewards given only after each charge cycle has completed.
11 . The system of claim 8 , wherein the reinforcement learning model is trained by applying random noise.
12 . The system of claim 8 , wherein at least one of the at least one processors resides on a remote server, and wherein the command of the charging of an on-field battery is sent via the remote server to a battery management system associated with the on-field battery.
13 . The system of claim 8 , wherein slowly varying Ornstein-Uhlenbeck noise is utilized in perturbing a policy of the reinforcement learning model.
14 . The system of claim 8 , wherein the memory stores further instructions that, when executed by the one or more processors, cause the one or more processors to:
transfer a reward system of the reinforcement learning model from (i) a first reward system in which the rewards are given throughout each charge cycle of the simulation battery to (ii) a second reward system in which the rewards are given on a charge cycle-by-cycle basis;
wherein the transfer uses an affine layer attached to a critic model.
15 . The system of claim 8 , wherein the simulation battery is a vehicle battery, and wherein the on-field battery is an on-field vehicle battery.
16 . A non-transitory computer readable storage medium containing a plurality of program instructions, which when executed by a processor, cause the processor to perform the steps of:
receiving, at a simulator, simulation battery charge data associated with charging of a simulation vehicle battery;
determining, at the simulator, an anode overpotential of the simulation vehicle battery during each charge cycle of the simulation vehicle battery;
training reinforcement learning model based on the anode overpotential, wherein the reinforcement learning model is trained with rewards given throughout each charge cycle of the simulation vehicle battery;
determining, via the reinforcement learning model, state-of-health characteristics of the simulation vehicle battery over multiple charge cycles;
augmenting the reinforcement learning model based on the state-of-health characteristics of the simulation vehicle battery over multiple charge cycles, wherein the reinforcement learning model is trained with rewards given on a charge cycle-by-cycle basis;
commanding a charging of an on-field vehicle battery based on an output of the augmented reinforcement learning model;
based on the charging of the on-field vehicle battery, determining state-of-health characteristics of the on-field vehicle battery;
further augmenting the augmented reinforcement learning model based on the state-of-health characteristics of the on-field vehicle battery, wherein the reinforcement learning model is further trained with rewards given based on multiple charge cycles; and
based on the further augmenting, outputting a trained reinforcement learning model configured to optimize charging of on-field vehicle batteries.
17 . The non-transitory computer readable storage medium of claim 16 , wherein the reinforcement learning utilizes a twin delayed deep deterministic (TD3) policy gradient.
18 . The non-transitory computer readable storage medium of claim 16 , wherein the step of augmenting the reinforcement learning model based on the state-of-health characteristics of the simulation battery over multiple charge cycles further includes:
training the reinforcement learning model with rewards given only after each charge cycle has completed.
19 . The non-transitory computer readable storage medium of claim 16 , wherein the step of commanding is sent via a remote server to a battery management system of an on-field vehicle.
20 . The non-transitory computer readable storage medium of claim 16 , wherein the program instructions, when executed by the processor, cause the processor to further perform the step of:
transferring a reward system of the reinforcement learning model from (i) a first reward system in which the rewards are given throughout each charge cycle of the simulation vehicle battery to (ii) a second reward system in which the rewards are given on a charge cycle-by-cycle basis;
wherein the transferring uses an affine layer attached to a critic model.