Apparatus for learning energy efficiency, a system having the same, and a method thereof
An energy efficiency improvement learning apparatus includes a processor configured to perform learning for vehicle control to minimize vehicle energy consumption required for vehicle driving under random conditions based on a reinforcement learning algorithm, and provide learned parameters of the reinforcement learning algorithm to a vehicle. The energy efficiency improvement learning apparatus also includes a storage medium configured to store data and algorithms, and the reinforcement learning algorithm executable by the processor.
1 . An energy efficiency improvement learning apparatus comprising:
a processor configured to:
perform real-time reinforcement learning for motor torque control of a vehicle to minimize vehicle energy consumption required for vehicle driving under random conditions based on a reinforcement learning algorithm, and
provide learned parameters of the real-time reinforcement learning algorithm to the vehicle,
wherein the processor is configured to:
end a reinforcement learning episode when a driving time during a predetermined learning period is greater than a predetermined time constraint condition, and
calculate a reward based on a vehicle speed when the driving time during the predetermined learning period is smaller than the predetermined time constraint condition; and
a storage medium configured to store data and algorithms, wherein the storage medium is configured to store the reinforcement learning algorithm executable by the processor.
2 . The energy efficiency improvement learning apparatus of claim 1 , wherein the processor is configured to:
randomly generate at least one of an entire path from a starting point to a destination, a road shape, a target driving time, an initial vehicle speed, an average target speed, an upper speed limit, a lower speed limit, or any combination thereof, and
start the reinforcement learning episode including a process of a state, an action, and a reward from an initial state to an end state.
3 . The energy efficiency improvement learning apparatus of claim 2 , wherein the action includes motor demand torque.
4 . The energy efficiency improvement learning apparatus of claim 2 , wherein the reward includes at least one of energy efficiency, a driving time, or a speed range.
5 . The energy efficiency improvement learning apparatus of claim 1 , wherein the processor is configured to:
perform a first action according to a first state,
determine a first reward according to the first action, and
transition to a second state when the reinforcement learning algorithm is executed.
6 . The energy efficiency improvement learning apparatus of claim 5 , wherein the storage medium is configured to store the first state, the first action, the first reward, and the second state as first data.
7 . The energy efficiency improvement learning apparatus of claim 6 , wherein the processor is configured to learn a policy function for determining what action to take by using at least one piece of data storing a state, an action, a reward, and a new state.
8 . The energy efficiency improvement learning apparatus of claim 1 , wherein the processor is configured to determine a reward based on a predetermined speed constraint condition, a predetermined time constraint condition, and energy required for driving.
9 . The energy efficiency improvement learning apparatus of claim 1 , wherein the processor is configured to, after starting the reinforcement learning episode under the random conditions, compare the driving time during the predetermined learning period with the predetermined time constraint condition.
10 . The energy efficiency improvement learning apparatus of claim 1 , wherein the processor is configured to determine the reward based on the vehicle speed as a predetermined negative reward if the vehicle speed is lower than a predetermined lower limit or higher than a predetermined upper limit.
11 . The energy efficiency improvement learning apparatus of claim 1 , wherein the processor is configured to:
when the vehicle speed is lower than a predetermined lower limit for ending learning or is higher than a predetermined upper limit for ending learning:
determine the reward based on the vehicle speed as a predetermined maximum negative reward, and
end the reinforcement learning episode.
12 . The energy efficiency improvement learning apparatus of claim 11 , wherein the processor is configured to, after ending the reinforcement learning episode, determine a final reward by calculating a reward based on a time and summing the reward based on the vehicle speed and the reward based on the time.
13 . The energy efficiency improvement learning apparatus of claim 1 , wherein the processor is configured to, when the vehicle speed is a value between a predetermined lower limit speed for ending learning and a predetermined upper limit speed for ending learning, determine a weight reward using a predetermined weight and required motor power.
14 . The energy efficiency improvement learning apparatus of claim 13 , wherein the processor is configured to:
determine whether a driving distance during the predetermined learning period is smaller than a predetermined target driving distance, and
when the driving distance during the predetermined learning period is smaller than the predetermined target driving distance, determine a final reward by summing the reward based on the vehicle speed and the weight reward.
15 . The energy efficiency improvement learning apparatus of claim 13 , wherein the processor is configured to:
determine whether a driving distance during the predetermined learning period is smaller than a predetermined target driving distance, and
when the driving distance during the predetermined learning period is equal to or greater than the predetermined target driving distance:
end the reinforcement learning episode,
calculate a time-based reward, and
sum a vehicle speed-based reward, the time-based reward, and the weight reward to determine a final reward.
16 . The energy efficiency improvement learning apparatus of claim 15 , wherein the processor is configured to calculate the time-based reward by using a maximum time constraint reward value, a time factor, and a distance factor.
17 . The energy efficiency improvement learning apparatus of claim 13 , wherein the processor is configured to:
linearly assign a weight to a ratio of a target time to an actual driving time, and
exponentially assign a weight to the ratio of a target distance to the actual driving distance.
18 . An energy efficiency improvement system, comprising:
an energy efficiency improvement learning apparatus configured to:
perform real-time reinforcement learning for motor torque control of a vehicle to minimize vehicle energy consumption required for vehicle driving under random conditions based on a reinforcement learning algorithm, and
provide learned parameters of the reinforcement learning algorithm to the vehicle, wherein the energy efficiency improvement learning apparatus is configured to:
end a reinforcement learning episode when a driving time during a predetermined learning period is greater than a predetermined time constraint condition, and
calculate a reward based on a vehicle speed when the driving time during the predetermined learning period is smaller than the predetermined time constraint condition; and
a vehicle control apparatus configured to determine a command for motor torque control of the vehicle by inputting vehicle state information to the reinforcement learning algorithm to which learned parameters received from the energy efficiency improvement learning apparatus are applied.
19 . An electrical energy efficiency improvement method for a vehicle, comprising:
performing, by a processor, real-time reinforcement learning for motor torque control of a vehicle to minimize vehicle energy consumption required for vehicle driving under random conditions based on a reinforcement learning algorithm, wherein performing real-time reinforcement learning includes i) ending a reinforcement learning episode when a driving time during a predetermined learning period is greater than or equal to a predetermined time constraint condition, and ii) calculating a reward based on a vehicle speed when the driving time during the predetermined learning period is smaller than the predetermined time constraint condition; and
providing, by the processor, learned parameters of the reinforcement learning algorithm to the vehicle.