Hybrid vehicle fuel efficiency using inverse reinforcement learning

1. A computer based method for controlling a powertrain of a hybrid electric vehicle (HEV) having an engine and a battery, comprising steps of:

collecting route information about a plurality of driving routes, wherein each driving route is comprised of a plurality of route segments connected by a plurality of intersections;

predicting a probability distribution over possible future route segments of multiple possible routes to be traversed by the HEV based on the collected route information;

computing a first value α 1 and a second value α 2 , wherein α 1 represents a proportion of an instantaneous power requirement (P req ) supplied by an engine of the HEV, and α 2 controls a recharging rate of a battery of the HEV, such that an expected energy expenditure over the probability distribution is reduced;

determining, based on α 1 and α 2 , how much engine power to use (P eng ) and how much battery power to use (P batt ); and

operating the powertrain according to P eng and P batt .

2. The method of claim 1 , further comprising repeating the steps of claim 1 after a time period has elapsed.

3. The method of claim 1 , wherein predicting the probability distribution over possible future paths of the HEV is performed based on past driver history.

4. The method of claim 3 , wherein past driver history the route information includes a set of action features (f) that represent aspects of each of the plurality of driving route intersections.

5. The method of claim 4 , wherein f includes one or more elements of the group containing:

an identity of an outgoing road segment;

a type of a road segment;

an angle of turn between an incoming road segment and an outgoing road segment;

an elevation change of a road segment; and

a cardinal direction of a road segment.

6. The method of claim 1 , wherein computing the first value α 1 and the second value α 2 such that the expected energy expenditure over the probability distribution is reduced comprises maximizing a value function V given by:

V

⁡

(

r

i

,

x

i

,

f

i

)

=

R

⁡

(

R

i

,

x

i

,

f

i

)

+

∑

r

i

⁢

P

⁡

(

r

i

+

1

|

r

i

)

⁢

V

⁡

(

r

i

+

1

,

x

i

+

1

,

f

i

+

1

)

wherein r represents a road segment, x represents a state-of-charge of the HEV's battery, f represents a fuel level of the HEV, (r i , x i , f i ) represents a current state of the HEV, (r i+1 , x i+1 , f i+1 ) represents a next state of the HEV, and R represents a reward function.

7. The method of claim 6 , wherein the reward function R of a state (r, x, f) is equal to a sum of residual fuel energy and battery energy at the state.

8. The method of claim 1 , wherein determining, based on α 1 and α 2 , how much engine power to use (P eng ) and how much battery power to use (P batt ) comprises:

determining whether a state-of-charge of the HEV's battery (x) is larger than a maximal relative charge allowed for the battery;

responsive to determining that x is larger than the maximal relative charge allowed for the battery:

determining that P eng =α 1 ·P req ; and

determining that P batt =(1−α 1 ) ·P req ; and

responsive to determining that x is not larger than the maximal relative charge allowed for the battery:

determining that P eng =P req +α 2 ·P charging ; and

determining that P batt =−α 2 ·P charging ;

wherein P charging is a maximum charging power capacity of the HEV's battery.

9. A non-transitory computer-readable storage medium storing executable computer program instructions for controlling a powertrain of a hybrid electric vehicle (HEV) having an engine and a battery, the instructions performing steps comprising:

collecting route information about a plurality of driving routes, wherein each driving route is comprised of a plurality of route segments connected by a plurality of intersections;

predicting a probability distribution over possible future route segments of multiple possible routes to traverse by the HEV based on the collected route information;

computing a first value α 1 and a second value α 2 , wherein α l represents a proportion of an instantaneous power requirement (P req ) supplied by an engine of the HEV, and a 2 controls a recharging rate of a battery of the HEV, such that an expected energy expenditure over the probability distribution is reduced;

determining, based on α l and α 2 , how much engine power to use (P eng ) and how much battery power to use (P batt ); and

operating the powertrain according to P eng and P batt .

10. The computer-readable storage medium of claim 9 , wherein the instructions perform steps further comprising repeating the steps of claim 9 after a time period has elapsed.

11. The computer-readable storage medium of claim 9 , wherein the route information includes a set of action features (f) that represent aspects of each of the plurality of intersections.

12. The computer-readable storage medium of claim 11 , wherein f includes one or more elements of the group containing:

an identity of an outgoing road segment;

a type of a road segment;

an angle of turn between an incoming road segment and an outgoing road segment;

an elevation change of a road segment; and

a cardinal direction of a road segment.

13. The computer-readable storage medium of claim 9 , wherein computing the first value α 1 and the second value α 2 such that the expected energy expenditure over the probability distribution is reduced comprises maximizing a value function V given by:

V

⁡

(

r

i

,

x

i

,

f

i

)

=

R

⁡

(

r

i

,

x

i

,

f

i

)

+

∑

r

i

⁢

P

⁡

(

r

i

+

1

|

r

i

)

⁢

V

⁡

(

r

i

+

1

,

x

i

+

1

,

f

i

+

1

)

wherein r represents a road segment, x represents a state-of-charge of the HEV's battery, f represents a fuel level of the HEV, (r i+ , x i+ , f i ) represents a current state of the HEV, (r i+1 ,x i+1 , f i+1 ) represents a next state of the HEV, and R represents a reward function.

14. The computer-readable storage medium of claim 13 , wherein the reward function R of a state (r, x, f) is equal to a sum of residual fuel energy and battery energy at the state.

15. The computer-readable storage medium of claim 9 , wherein determining, based on α 1 and α 2 , how much engine power to use (P eng ) and how much battery power to use (P batt ) comprises:

determining whether a state-of-charge of the HEV's battery (x) is larger than a maximal relative charge allowed for the battery;

responsive to determining that x is larger than the maximal relative charge allowed for the battery:

determining that P eng =α 1 ·P req ; and

determining that P batt =(1−α 1 )·P req ; and

responsive to determining that x is not larger than the maximal relative charge allowed for the battery:

determining that P eng =P req +α 2 ·P charging ; and

determining that P batt =−α 2 ·P charging ;

wherein P charging is a maximum charging power capacity of the HEV's battery.

16. A system for controlling a powertrain of a hybrid electric vehicle (HEV) having an engine and a battery, the system comprising:

at least one non-transitory computer-readable storage medium storing executable computer program instructions comprising instructions for:

collecting route information about a plurality of driving routes, wherein each driving route is comprised of a plurality of route segments connected by a plurality of intersections;

predicting a probability distribution over possible future route segments of the HEV based on the collected route information;

computing a first value α 1 and a second value α 2 , wherein α 1 represents a proportion of an instantaneous power requirement (P req ) supplied by an engine of the HEV, and α 2 controls a recharging rate of a battery of the HEV, such that an expected energy expenditure over the probability distribution is reduced;

determining, based on α 1 and α 2 , how much engine power to use (P eng ) and how much battery power to use (P batt ); and

operating the powertrain according to P eng and P batt ; and

a processor for executing the computer program instructions.

17. The method of claim 1 , wherein determining a driver model using an inverse reinforcement learning algorithm further comprises representing the route information as a Markov decision process having states s each state having a plurality of actions a.

18. The method of claim 17 , wherein each intersection is represented as a state s i and each route segment chosen at the state s i is represented as an action a i .

19. The computer-readable storage medium of claim 9 , wherein determining a driver model using an inverse reinforcement learning algorithm further comprises representing the route information as a Markov decision process having states s each state having a plurality of actions a.

20. The computer-readable storage medium of claim 19 , wherein each intersection is represented as a state s i and each route segment chosen at the state s i is represented as an action a i .

21. The method of claim 1 , further comprising:

determining a driver model by using an inverse reinforcement learning algorithm on the collected route information; and

predicting a probability distribution over possible future route segments of the HEV based on the determined driver model given a current position of the HEV.

22. The computer-readable storage medium of claim 9 , performing steps further comprising:

determining a driver model by using an inverse reinforcement learning algorithm on the collected route information; and

predicting a probability distribution over possible future route segments of the HEV based on the determined driver model given a current position of the HEV.