Method and system for adaptive cycle-level traffic signal control
Methods, systems, and processor-readable media for adaptive cycle-level traffic signal control are described. An adaptive cycle-level traffic signal controller and control method that operate within a continuous action space. A reinforcement learning algorithm called Proximal Policy Optimization (PPO), which is a type of actor-critic model for reinforcement learning, may be used to generate signal cycle phase durations selected from a continuous range of values. The controller thus does not treat the action space as discrete, but instead produces continuous values as output. The generated phase durations may define a full traffic signal cycle. The inputs to the controller may indicate current and past states of the traffic environment. The average duration of delay of vehicles in the traffic environment may be used to calculate the reward for the reinforcement learning model that drives the behavior of the controller.
1. A method for training a reinforcement learning model to generate traffic signal cycle data, the method comprising:
processing a training data sample indicating an initial state of a traffic environment affected by a traffic signal by:
using the reinforcement learning model to generate the traffic signal cycle data by applying a policy to the training data sample and one or more past training data samples, the traffic signal cycle data comprising one or more phase durations of one or more respective phases of a traffic signal cycle, each phase duration being a value selected from a continuous range of values;
determining an updated state of the traffic environment following application of the generated traffic signal cycle data to the traffic signal;
generating a reward by applying a reward function to the initial state of the traffic environment and the updated state of the traffic environment; and
adjusting the policy based on the reward; and
repeating one or more times the step of processing, wherein the training data sample is updated for the updated state of the traffic environment.
2. The method of claim 1 , wherein:
the traffic environment is a simulated traffic environment; and
the traffic signal is a simulated traffic signal.
3. The method of claim 1 , wherein the one or more phase durations comprises phase durations for each phase of at least one cycle of the traffic signal.
4. The method of claim 1 , wherein the one or more phase durations consists of a phase duration for one phase of a cycle of the traffic signal.
5. The method of claim 1 , wherein:
the reinforcement learning model is an actor-critic model;
the policy is an actor policy; and
the reward function is a critic reward function.
6. The method of claim 5 , wherein the actor-critic model is a proximal policy optimization (PPO) model.
7. The method of claim 1 , wherein each training data sample comprises traffic data, including location data and velocity data for each of a plurality of vehicles in the traffic environment.
8. The method of claim 1 , wherein each training data sample comprises traffic data, including traffic density data and traffic velocity data for each of a plurality of regions of the traffic environment.
9. The method of claim 1 , wherein:
determining an updated state of the traffic environment comprises determining a length of each of one or more queues of stationary vehicles in the traffic environment, the length indicating a number of stationary cars in the queue; and
the one or more past training data samples comprises:
one or more past training data samples corresponding to one or more queue peak times, each queue peak time being a time at which the length of one of the queues was at a local maximum; and
one or more past training data samples corresponding to one or more one or more past training data samples corresponding to one or more queue trough times, each queue trough time being a time at which a length of one of the queues was at a local minimum.
10. The method of claim 1 , wherein the one or more past training data samples correspond to one or more phase transition times, each phase transition time being a time at which the traffic signal transitions between two phases of the traffic signal cycle.
11. The method of claim 1 , wherein the reward function is applied to the initial state of the traffic environment and the updated state of the traffic environment to calculate the reward based on an estimated number of stationary vehicles in the traffic environment over a previous traffic signal cycle.
12. The method of claim 11 , wherein the one or more past training data samples correspond to one or more phase transition times, each phase transition time being a time at which the traffic signal transitions between two phases of the traffic signal cycle.
13. The method of claim 1 , wherein each training data sample comprises traffic signal phase data indicating:
the current phase of the traffic signal cycle; and
an elapsed time during the current phase.
14. The method of claim 1 , wherein:
the one or more phase durations comprises phase durations for each phase of at least one cycle of the traffic signal;
the reinforcement learning model is a proximal policy optimization (PPO) actor-critic model;
the policy is an actor policy;
the reward function is a critic reward function;
each training data sample comprises:
traffic signal phase data indicating:
the current phase of the traffic signal cycle; and
an elapsed time during the current phase; and
traffic data, including traffic density data and traffic velocity data for each of a plurality of regions of the traffic environment;
the reward function is applied to the initial state of the traffic environment and the updated state of the traffic environment to calculate the reward based on an estimated number of stationary vehicles in the traffic environment over a previous traffic signal cycle; and
the one or more past training data samples correspond to one or more phase transition times, each phase transition time being a time at which the traffic signal transitions between two phases of the traffic signal cycle.
15. A system for training a reinforcement learning model to generate traffic signal cycle data, comprising:
a processor device; and
a memory storing:
the reinforcement learning model; and
machine-executable instructions thereon which, when executed by the processing device, cause the system to:
process a training data sample indicating an initial state of a traffic environment affected by a traffic signal by:
using the reinforcement learning model to generate the traffic signal cycle data by applying a policy to the training data sample and one or more past training data samples, the traffic signal cycle data comprising one or more phase durations of one or more respective phases of a traffic signal cycle, each phase duration being a value selected from a continuous range of values;
determining an updated state of the traffic environment following application of the generated traffic signal cycle data to the traffic signal;
generating a reward by applying a reward function to the initial state of the traffic environment and the updated state of the traffic environment; and
adjusting the policy based on the reward; and
repeat one or more times the step of processing, wherein the training data sample is updated for the updated state of the traffic environment.
16. The system of claim 15 , wherein:
the reinforcement learning model is a proximal policy optimization (PPO) actor-critic model;
the policy is an actor policy; and
the reward function is a critic reward function.
17. The system of claim 15 , wherein:
the reward function is applied to the initial state of the traffic environment and the updated state of the traffic environment to calculate the reward based on an estimated number of stationary vehicles in the traffic environment over a previous traffic signal cycle; and
the one or more past training data samples correspond to one or more phase transition times, each phase transition time being a time at which the traffic signal transitions between two phases of the traffic signal cycle.
18. A system for generating traffic signal cycle data, comprising:
a processor device; and
a memory storing:
a trained reinforcement learning model trained in accordance with the method of claim 1 ; and
machine-executable instructions which, when executed by the processing device, cause the system to:
receive, from a traffic monitoring system, traffic environment state data indicating a state of a real traffic environment, wherein the traffic environment used to train the reinforcement learning model is the real traffic environment or a simulated version thereof;
use the reinforcement learning model to generate traffic signal cycle data by applying the policy to at least the traffic environment state data; and
send, to a traffic control system, the traffic signal cycle data.
19. A non-transitory processor-readable medium having stored thereon a trained reinforcement learning model trained in accordance with the method of claim 1 .
20. A non-transitory processor-readable medium having machine-executable instructions stored thereon which, when executed by a processor device, cause the processor device to perform the method of claim 1 .