Method and system for controlling heavy-haul train based on reinforcement learning

1. A method for controlling a heavy-haul train based on reinforcement learning, comprising:

obtaining operation state information of a heavy-haul train at a current time point, wherein the heavy-haul train comprises a plurality of vehicles, the vehicles comprise a traction locomotive and freight wagons, and the operation state information comprises a speed of the heavy-haul train and a position of the heavy-haul train; and

obtaining a heavy-haul train action of a next time point according to the operation state information of the heavy-haul train at the current time point and a heavy-haul train virtual controller, and sending the heavy-haul train action of the next time point to a heavy-haul train control unit to control operation of the heavy-haul train;

wherein the heavy-haul train virtual controller stores line information of the heavy-haul train;

a training process of the heavy-haul train virtual controller comprises: constructing a training architecture of the heavy-haul train virtual controller, wherein the training architecture of the heavy-haul train virtual controller comprises the expert strategy network, the reinforcement learning network, a memory, and a simulation environment for heavy-haul train operation, and the memory stores the operation state data of the heavy-haul train; inputting same-cycle positions in the operation state data of the heavy-haul train to the expert strategy network and the actor network of the reinforcement learning network, and inputting same-cycle speeds in the operation state data of the heavy-haul train to the expert strategy network and the actor network of the reinforcement learning network, to obtain an expert action and a reinforcement learning action; inputting the reinforcement learning action as well as the same-cycle speeds and the same-cycle positions in the operation state data of the heavy-haul train to the critic networks of the reinforcement learning network, to obtain a critic loss; calculating an instruction loss according to the expert action and the reinforcement learning action; and updating actor network parameters of the reinforcement learning network according to a comprehensive loss, until a required heavy-haul train virtual controller is obtained, wherein the comprehensive loss is a sum of the critic loss and the instruction loss; and

the reinforcement learning network comprises one actor network and two critic networks; the reinforcement learning network is constructed according to a soft actor-critic (SAC) reinforcement learning algorithm; the operation state data of the heavy-haul train is a temporal sequence of positions, speeds, and rewards of the heavy-haul train in M cycles.

2. The method for controlling a heavy-haul train based on reinforcement learning according to claim 1 , wherein a training process of the expert strategy network comprises:

pre-training a recurrent neural network by using historical data of actual operation of the heavy-haul train, to obtain the expert strategy network, wherein the historical data of actual operation of the heavy-haul train is a temporal sequence of positions, speeds, and control forces of the heavy-haul train on a line in N cycles.

3. The method for controlling a heavy-haul train based on reinforcement learning according to claim 1 , wherein the training process of the heavy-haul train virtual controller further comprises:

updating critic network parameters of the reinforcement learning network according to a reward of the operation state data of the heavy-haul train.

4. The method for controlling a heavy-haul train based on reinforcement learning according to claim 1 , wherein the training process of the heavy-haul train virtual controller further comprises:

inputting the reinforcement learning action to the simulation environment for heavy-haul train operation, to obtain updated operation state data of the heavy-haul train, and storing the updated operation state data of the heavy-haul train into the memory.

5. The method for controlling a heavy-haul train based on reinforcement learning according to claim 1 , wherein a process of constructing the simulation environment for heavy-haul train operation comprises:

determining a kinematic model of the heavy-haul train;

determining, in a reinforcement learning environment, a reward function of an operation process of the heavy-haul train, wherein the reward function comprises a speed reward function, an explore range reward function, and a maximum coupler force reward function; and

forming the simulation environment for heavy-haul train operation according to the kinematic model of the heavy-haul train and the reward function.

6. The method for controlling a heavy-haul train based on reinforcement learning according to claim 5 , wherein the determining a kinematic model of the heavy-haul train comprises:

modeling the operation process of the heavy-haul train by using a multi-particle longitudinal dynamic equation of the heavy-haul train, to obtain the kinematic model of the heavy-haul train.

7. A system for controlling a heavy-haul train based on reinforcement learning, comprising:

an information obtaining module, configured to obtain operation state information of a heavy-haul train at a current time point, wherein the heavy-haul train comprises a plurality of vehicles, the vehicles comprise a traction locomotive and freight wagons, and the operation state information comprises a speed of the heavy-haul train and a position of the heavy-haul train; and

an action determining module, configured to obtain a heavy-haul train action of a next time point according to the operation state information of the heavy-haul train at the current time point and a heavy-haul train virtual controller, and send the heavy-haul train action of the next time point to a heavy-haul train control unit to control operation of the heavy-haul train;

wherein the heavy-haul train virtual controller stores line information of the heavy-haul train;

a training process of the heavy-haul train virtual controller comprises: constructing a training architecture of the heavy-haul train virtual controller, wherein the training architecture of the heavy-haul train virtual controller comprises the expert strategy network, the reinforcement learning network, a memory, and a simulation environment for heavy-haul train operation, and the memory stores the operation state data of the heavy-haul train; inputting same-cycle positions in the operation state data of the heavy-haul train to the expert strategy network and the actor network of the reinforcement learning network, and inputting same-cycle speeds in the operation state data of the heavy-haul train to the expert strategy network and the actor network of the reinforcement learning network, to obtain an expert action and a reinforcement learning action; inputting the reinforcement learning action as well as the same-cycle speeds and the same-cycle positions in the operation state data of the heavy-haul train to the critic networks of the reinforcement learning network, to obtain a critic loss; calculating an instruction loss according to the expert action and the reinforcement learning action; and updating actor network parameters of the reinforcement learning network according to a comprehensive loss, until a required heavy-haul train virtual controller is obtained, wherein the comprehensive loss is a sum of the critic loss and the instruction loss; and

the reinforcement learning network comprises one actor network and two critic networks; the reinforcement learning network is constructed according to a soft actor-critic (SAC) reinforcement learning algorithm; the operation state data of the heavy-haul train is a temporal sequence of positions, speeds, and rewards of the heavy-haul train in M cycles.

8. The system for controlling a heavy-haul train based on reinforcement learning according to claim 7 , wherein a training process of the expert strategy network in the action determining module comprises:

pre-training a recurrent neural network by using historical data of actual operation of the heavy-haul train, to obtain the expert strategy network, wherein the historical data of actual operation of the heavy-haul train is a temporal sequence of positions, speeds, and control forces of the heavy-haul train on a line in N cycles.