Apparatus and method for controlling traffic signals of traffic lights in sub-area by using reinforcement learning model
Provided are methods and apparatuses for controlling traffic signals of traffic lights in a sub-area by using a neural network model. The method according to an embodiment of the present disclosure may configure state information of a sub-area by using downstream information obtained in a current cycle time for each of a plurality of intersections included in the sub-area. In addition, the method may input the state information to a trained reinforcement learning model, and obtain action information of the sub-area including green times and offsets, by using an output from the trained reinforcement learning model. Furthermore, the method may generate coordinated signal values for applying the action to traffic lights in the sub-area in a subsequent cycle time.
1. A method of controlling traffic signals of a plurality of traffic lights in a sub-area by using a neural network model, the method comprising:
configuring state information of the sub-area by using downstream information for a current cycle time, wherein the downstream information is configured for each of a plurality of intersections included in the sub-area;
obtaining action information including green times and offsets for the sub-area by inputting the state information to a trained neural network model;
determining whether an offset is set to within a preset absolute value range; and
generating a coordinated signal value for applying the action information to the plurality of traffic lights in the sub-area during a transition process configured with a plurality of subsequent cycle times, in response to determining that the offset is set to a value out of the preset absolute value range,
wherein the neural network model is trained using a reinforcement learning algorithm which is based on the action information, the state information, and reward information, and
wherein the reward information for each intersection is defined as an arithmetic mean of stop rate obtained at downstream of each of a plurality of links of intersection, each of the stop rates is a value obtained by diving a processed queue length by a processed traffic volume of the downstream of one of the plurality of links of the intersection.
2. The method of claim 1 , wherein
the downstream information includes the processed traffic volume and the processed queue length,
the processed traffic volume is calculated based on a traffic volume defined as a number of vehicles passing a particular point per hour and a maximum traffic volume not related to a geometric structure of a road, and
the processed queue length is calculated based on a number of waiting vehicles and a length of a downstream.
3. The method of claim 1 , further comprising
configuring state information of the sub-area by using downstream information obtained in a subsequent cycle time after the transition process, in response to determining that the offset is set to a value out of the preset absolute value range.
4. The method of claim 1 , wherein the coordinated signal value is generated to gradually apply the action information to the plurality of traffic lights in the sub-area during the transition process configured with the plurality of subsequent cycle times, in response to determining that the offset is set to a value out of the preset absolute value range.
5. The method of claim 1 , wherein a number of subsequent cycle times in the transition process is determined based on an extent to which the offset exceeds the preset absolute value range.
6. The method of claim 1 , further comprising
generating a coordinated signal value for applying the action information to the plurality of traffic lights in the sub-area in an earliest cycle time among the plurality of subsequent cycle times, in response to determining that the offset is set to a value within the preset absolute value range.
7. The method of claim 1 , wherein
the reinforcement learning algorithm comprises a deep deterministic policy gradient (DDPG) algorithm, and
the neural network model is trained to output the action information, wherein the green times and the offsets minimize a total number of stops of vehicles passing through the sub-area.
8. The method of claim 1 , wherein
a green time is set to a value between a minimum green time and a maximum green time, and
the offset is set to a value within an absolute value range of a fixed cycle time.
9. The method of claim 1 , wherein
the reward information is defined by using a stop rate representing a ratio of vehicles that have experienced a stop among vehicles passing through during a specific cycle time.
10. The method of claim 1 , wherein
the neural network model is an actor network, and is trained based on reinforcement learning including a critic network,
the actor network is trained by using the state information as input data and the action information as output data, to determine an optimal policy for controlling the traffic signals of the plurality of traffic lights in the sub-area, and
the critic network is trained by using, as input data, action information output from the actor network and using, as output data, the reward information for evaluating adequacy of the action information output from the actor network.
11. An apparatus for controlling traffic signals of a plurality of traffic lights in a sub-area by using a neural network model, the apparatus comprising:
a memory storing at least one program; and
at least one processor configured to drive the neural network model by executing the at least one program,
wherein the at least one processor is further configured to
configure state information of the sub-area by using downstream information for each of a plurality of intersections included in the sub-area, the downstream information being obtained in a current cycle time,
input the state information into trained neural network model and obtain an action of the sub-area by using an output of the trained neural network model, the action including green times and offsets,
determine whether an offset is set to within a preset absolute value range; and
generate a coordinated signal value for applying action information to the plurality of traffic lights in the sub-area during a transition process configured with a plurality of subsequent cycle times, in response to determining that the offset is set to a value out of the preset absolute value range,
wherein the neural network model is trained using a reinforcement learning algorithm which is based on the action information, the state information, and reward information, and
wherein the reward information for each intersection is defined as arithmetic mean of stop rates obtained at downstream of each of a plurality of links of the intersection, each of the stop rates is a value obtained by dividing a processed queue length by a processed traffic volume of the downstream of one of the plurality links of the intersection.
12. A non-transitory computer-readable recording medium that stores a program that, when executed by a computer, configures the computer to:
configure state information of a sub-area by using downstream information for a current cycle time, wherein the downstream information is configured for each of a plurality of intersections included in the sub-area,
obtain action information including green times and offsets for the sub-area by inputting the state information to a trained neural network model,
determine whether an offset is within a preset absolute value range; and
generate a coordinated signal value for applying the action information to a plurality of traffic lights in the sub-area during a transition process configured with a plurality of subsequent cycle times, in response to determining that the offset is set to a value out of the preset absolute value range,
wherein the neural network model is trained using a reinforcement learning algorithm which is based on the action information, the state information, and reward information, and
wherein the reward information for each intersection is defined as arithmetic mean of stop rates obtained at downstream of each of a plurality of links of the intersection, each of the stop rates is a value obtained by dividing a processed queue length by a processed traffic volume of the downstream of one of the plurality links of the intersection.