Autonomous lane change method and apparatus, and storage medium
An autonomous lane change method includes: calculating a local neighbor feature and a global statistical feature of an autonomous vehicle at a current moment based on travel information of the autonomous vehicle at the current moment and motion information of obstacles in lanes within a sensing range of the autonomous vehicle; obtaining a target action indication based on the local neighbor feature, the global statistical feature, and a current control policy; and executing the target action according to the target action indication. On the basis of the local neighbor feature, the global statistical feature is further introduced into the current control policy to obtain the target action indication. The target action obtained by combining local and global road obstacle information is a globally optimal decision action.
1. An autonomous lane change method for an autonomous lane change apparatus that comprises a processor, the method comprising:
calculating, by the processor, a local neighbor feature and a global statistical feature of an autonomous vehicle at a current moment based on travel information of the autonomous vehicle at the current moment and further based on motion information of obstacles in lanes within a sensing range of the autonomous vehicle, wherein the local neighbor feature represents motion status information of a neighboring obstacle of the autonomous vehicle relative to the autonomous vehicle, and the global statistical feature represents denseness of the obstacles in the lanes within the sensing range;
obtaining, by the processor, a target action indication based on the local neighbor feature, the global statistical feature, and a current control policy, wherein the target action indication indicates the autonomous vehicle to execute a target action, and the target action comprises at least one of a lane change type or a keeping straight type;
executing, by the processor, the target action according to the target action indication to cause the autonomous vehicle to execute the target action;
obtaining, by the processor, feedback information in response to executing the target action; and
updating, by the processor, the current control policy based on the feedback information to obtain an updated control policy,
wherein the feedback information is used to update the current control policy, wherein the feedback information comprises travel information generated after the autonomous vehicle executes the target action, future travel information of the autonomous vehicle, and future motion information of the obstacles in the lanes within the sensing range of the autonomous vehicle, and wherein when the target action is the lane change type, the feedback information further comprises a ratio of a period of time for executing the target action to a historical average period of time and a denseness change between obstacles in a lane in which the autonomous vehicle is located before the lane change type and obstacles in a lane in which the autonomous vehicle is located after the lane change type, wherein the historical average period of time is an average period of time for which the autonomous vehicle executes a similar action within a preset historical period of time.
2. The method according to claim 1 , wherein the updating the current control policy based on the feedback information to obtain the updated control policy comprises:
calculating, by the processor based on the feedback information, a subsequent local neighbor feature and a subsequent global statistical feature of the autonomous vehicle, and a reward corresponding to the target action;
determining, by the processor, four-tuple information at the current moment, wherein the four-tuple information at the current moment corresponds to a vehicle condition at the current moment, and comprises: a feature at the current moment, the target action, the reward corresponding to the target action, and a subsequent feature, wherein the feature at the current moment comprises the local neighbor feature and the global statistical feature of the autonomous vehicle at the current moment, and the subsequent feature comprises the subsequent local neighbor feature and the subsequent global statistical feature of the autonomous vehicle; and
updating, by the processor, the current control policy based on the four-tuple information at the current moment to obtain the updated control policy.
3. The method according to claim 2 , wherein based on the target action being the keeping straight type, the updating the current control policy based on the four-tuple information at the current moment to obtain the updated control policy comprises:
generating, by the processor based on the four-tuple information at the current moment, a target value corresponding to the four-tuple information;
iteratively updating, by the processor, by using a gradient descent method, a parameter Q in a first preset function that comprises the target value; and
replacing, by the processor, a parameter q in the current control policy with an iteratively updated parameter q, to obtain the updated control policy.
4. The method according to claim 2 , wherein based on the target action being the lane change type, the updating the current control policy based on the four-tuple information at the current moment to obtain the updated control policy comprises:
obtaining, by the processor, extended four-tuple information at the current moment, wherein the extended four-tuple information at the current moment corresponds to an extended vehicle condition at the current moment, and the extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment according to a symmetry rule and a monotone rule, wherein the symmetry rule indicates that locations of obstacles in all left lanes and obstacles in all right lanes of the lane in which the autonomous vehicle is located are symmetrically exchanged by using the lane in which the autonomous vehicle is located as an axis, and the monotone rule indicates that a distance increases between front and back neighboring obstacles of the autonomous vehicle in a target lane of the lane change type executed by the autonomous vehicle, and/or indicates that a change in a distance between front and back neighboring obstacles of the autonomous vehicle in a non-target lane is less than a preset distance range; and
updating, by the processor, the current control policy based on the four-tuple information at the current moment and the extended four-tuple information at the current moment to obtain the updated control policy.
5. The method according to claim 4 , wherein the updating the current control policy based on the four-tuple information at the current moment and the extended four-tuple information at the current moment to obtain the updated control policy comprises:
generating, by the processor based on i th four-tuple information in the four-tuple information at the current moment and the extended four-tuple information at the current moment, a target value corresponding to the i th four-tuple information, wherein i is a positive integer not greater than n, and n is a total quantity of four-tuple information comprised in the four-tuple information at the current moment and the extended four-tuple information comprised in the four-tuple information at the current moment;
iteratively updating, by the processor, by using a gradient descent method, a parameter q in a second preset function that comprises the target value corresponding to the i th four-tuple information; and
replacing, by the processor, a parameter q in the current control policy with an iteratively updated parameter q, to obtain the updated control policy.
6. The method according to claim 2 , wherein based on the target action being the keeping straight type, the updating the current control policy based on the four-tuple information at the current moment to obtain the updated control policy comprises:
updating, by the processor, the current control policy based on the four-tuple information at the current moment, four-tuple information at a historical moment, and extended four-tuple information at the historical moment, to obtain the updated control policy at, wherein
the four-tuple information at the historical moment corresponds to a vehicle condition at the historical moment, and comprises: a feature at the historical moment, a target action at the historical moment, a reward corresponding to the target action at the historical moment, and a subsequent feature of the historical moment, wherein the feature at the historical moment comprises a local neighbor feature and a global statistical feature of the autonomous vehicle at the historical moment, and the subsequent feature of the historical moment comprises a subsequent local neighbor feature and a subsequent global statistical feature of the autonomous vehicle at the historical moment; and the extended four-tuple information at the historical moment corresponds to an extended vehicle condition at the historical moment, and the extended vehicle condition at the historical moment is obtained by processing the vehicle condition at the historical moment according to a symmetry rule and a monotone rule.
7. The method according to claim 6 , wherein the updating the current control policy based on the four-tuple information at the current moment, four-tuple information at the historical moment, and extended four-tuple information at the historical moment, to obtain the updated control policy comprises:
generating, by the processor based on j th four-tuple information in the four-tuple information at the current moment, the four-tuple information at the historical moment, and the extended four-tuple information at the historical moment, a target value corresponding to the j th four-tuple information, wherein j is a positive integer not greater than m, and m is a total quantity of four-tuple information comprised in the four-tuple information at the current moment, the four-tuple information at the historical moment, and the extended four-tuple information at the historical moment;
iteratively updating, by the processor, by using a gradient descent method, a parameter q in a third preset function that comprises the target value corresponding to the j th four-tuple information; and
replacing, by the processor, a parameter q in the current control policy with an iteratively updated parameter q, to obtain the updated control policy.
8. The method according to claim 2 , wherein based on the target action being the lane change type, the updating the current control policy based on the four-tuple information at the current moment to obtain the updated control policy comprises:
obtaining, by the processor, extended four-tuple information at the current moment, wherein the extended four-tuple information at the current moment corresponds to an extended vehicle condition at the current moment, and the extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment according to a symmetry rule and a monotone rule; and
updating, by the processor, the current control policy based on the four-tuple information at the current moment, the extended four-tuple information at the current moment, four-tuple information at a historical moment, and extended four-tuple information at the historical moment to obtain the updated control policy, wherein the four-tuple information at the historical moment corresponds to a vehicle condition at the historical moment, the extended four-tuple information at the historical moment corresponds to an extended vehicle condition at the historical moment, and the extended vehicle condition at the historical moment is obtained by processing the vehicle condition at the historical moment according to a symmetry rule and a monotone rule.
9. An autonomous lane change apparatus, comprising:
a processor; and
a memory electronically coupled to the processor,
wherein the memory is configured to store computer-readable program instructions and
wherein the processor is configured to execute the computer-readable program instructions stored in the memory to perform an autonomous lane change method comprising:
calculating, by the processor, a local neighbor feature and a global statistical feature of an autonomous vehicle at a current moment based on travel information of the autonomous vehicle at the current moment and motion information of obstacles in lanes within a sensing range of the autonomous vehicle, wherein the local neighbor feature represents motion status information of a neighboring obstacle of the autonomous vehicle in relation to the autonomous vehicle, and the global statistical feature represents denseness of the obstacles in the lanes within the sensing range;
obtaining, by the processor, a target action indication based on the local neighbor feature, the global statistical feature, and a current control policy, wherein the target action indication indicates the autonomous vehicle to execute a target action, and the target action comprises at least one of a lane change type or a keeping straight type;
executing, by the processor, the target action according to the target action indication to cause the autonomous vehicle to execute the target action;
obtaining, by the processor, feedback information by executing the target action; and
updating, by the processor, the current control policy based on the feedback information to obtain an updated control policy,
wherein the feedback information is used to update the current control policy, wherein the feedback information comprises travel information generated after the autonomous vehicle executes the target action, future travel information of the autonomous vehicle, and future motion information of the obstacles in the lanes within the sensing range of the autonomous vehicle, and wherein when the target action is the lane change type, the feedback information further comprises a ratio of a period of time for executing the target action to a historical average period of time and a denseness change between obstacles in a lane in which the autonomous vehicle is located before the lane change type and obstacles in a lane in which the autonomous vehicle is located after the lane change type, wherein the historical average period of time is an average period of time for which the autonomous vehicle executes a similar action within a preset historical period of time.
10. The apparatus according to claim 9 , wherein the updating the current control policy based on the feedback information to obtain the updated control policy comprises:
calculating, based on the feedback information, a subsequent local neighbor feature and a subsequent global statistical feature of the autonomous vehicle, and a reward corresponding to the target action;
determining four-tuple information at the current moment, wherein the four-tuple information at the current moment corresponds to a vehicle condition at the current moment, and comprises: a feature at the current moment, the target action, the reward corresponding to the target action, and a subsequent feature, wherein the feature at the current moment comprises the local neighbor feature and the global statistical feature of the autonomous vehicle at the current moment, and the subsequent feature comprises the subsequent local neighbor feature and the subsequent global statistical feature of the autonomous vehicle; and
updating the current control policy based on the four-tuple information at the current moment to obtain the updated control policy.
11. The apparatus according to claim 10 , wherein based on the target action being the keeping straight type, the updating the current control policy based on the four-tuple information at the current moment to obtain the updated control policy comprises:
generating, based on the four-tuple information at the current moment, a target value corresponding to the four-tuple information;
iteratively updating, by using a gradient descent method, a parameter q in a first preset function that comprises the target value; and
replacing a parameter q in the current control policy with an iteratively updated parameter q, to obtain the updated control policy.
12. The apparatus according to claim 10 , wherein based on the target action being the lane change type, the updating the current control policy based on the four-tuple information at the current moment to obtain the updated control policy comprises:
obtaining extended four-tuple information at the current moment, wherein the extended four-tuple information at the current moment corresponds to an extended vehicle condition at the current moment, and the extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment according to a symmetry rule and a monotone rule, wherein the symmetry rule indicates that locations of obstacles in all left lanes and obstacles in all right lanes of the lane in which the autonomous vehicle is located are symmetrically exchanged by using the lane in which the autonomous vehicle is located as an axis, and the monotone rule indicates that a distance increases between front and back neighboring obstacles of the autonomous vehicle in a target lane of the lane change type executed by the autonomous vehicle, and/or indicates that a change in a distance between front and back neighboring obstacles of the autonomous vehicle in a non-target lane is less than a preset distance range; and
updating the current control policy based on the four-tuple information at the current moment and the extended four-tuple information at the current moment to obtain the updated control policy.
13. The apparatus according to claim 12 , wherein the updating the current control policy based on the four-tuple information at the current moment and the extended four-tuple information at the current moment to obtain the updated control policy comprises:
generating, based on i th four-tuple information in the four-tuple information at the current moment and the extended four-tuple information at the current moment, a target value corresponding to the i th four-tuple information, wherein i is a positive integer not greater than n, and n is a total quantity of four-tuple information comprised in the four-tuple information at the current moment and the extended four-tuple information comprised in the four-tuple information at the current moment;
iteratively updating, by using a gradient descent method, a parameter q in a second preset function that comprises the target value corresponding to the i th four-tuple information; and
replacing a parameter q in the current control policy with an iteratively updated parameter q, to obtain the updated control policy.
14. The apparatus according to claim 10 , wherein based on the target action being keeping straight type, the updating the current control policy based on the four-tuple information at the current moment to obtain the updated control policy comprises:
updating the current control policy based on the four-tuple information at the current moment, four-tuple information at a historical moment, and extended four-tuple information at the historical moment, to obtain the updated control policy, wherein
the four-tuple information at the historical moment corresponds to a vehicle condition at the historical moment, and comprises: a feature at the historical moment, a target action at the historical moment, a reward corresponding to the target action at the historical moment, and a subsequent feature at the historical moment, wherein the feature at the historical moment comprises a local neighbor feature and a global statistical feature of the autonomous vehicle at the historical moment, and the subsequent feature of the historical moment comprises a subsequent local neighbor feature and a subsequent global statistical feature of the autonomous vehicle at the historical moment; and the extended four-tuple information at the historical moment corresponds to an extended vehicle condition at the historical moment, and the extended vehicle condition at the historical moment is obtained by processing the vehicle condition at the historical moment according to a symmetry rule and a monotone rule.
15. The apparatus according to claim 14 , wherein the updating the current control policy based on the four-tuple information at the current moment, four-tuple information at the historical moment, and extended four-tuple information at the historical moment, to obtain the updated control policy comprises:
generating, based on j th four-tuple information in the four-tuple information at the current moment, the four-tuple information at the historical moment, and the extended four-tuple information at the historical moment, a target value corresponding to the j th four-tuple information, wherein j is a positive integer not greater than m, and m is a total quantity of four-tuple information comprised in the four-tuple information at the current moment, the four-tuple information at the historical moment, and the extended four-tuple information at the historical moment;
iteratively updating, by using a gradient descent method, a parameter q in a third preset function that comprises the target value corresponding to the j th four-tuple information; and
replacing a parameter q in the current control policy with an iteratively updated parameter q, to obtain the updated control policy.
16. The apparatus according to claim 10 , wherein based on the target action being lane change, the updating the current control policy based on the four-tuple information at the current moment to obtain the updated control policy comprises:
obtaining extended four-tuple information at the current moment, wherein the extended four-tuple information at the current moment corresponds to an extended vehicle condition at the current moment, and the extended vehicle condition at the current moment is obtained by processing the vehicle condition at the current moment according to a symmetry rule and a monotone rule; and
updating the current control policy based on the four-tuple information at the current moment, the extended four-tuple information at the current moment, four-tuple information at a historical moment, and extended four-tuple information at the historical moment to obtain the updated control policy, wherein the four-tuple information at the historical moment corresponds to a vehicle condition at the historical moment, the extended four-tuple information at the historical moment corresponds to an extended vehicle condition at the historical moment, and the extended vehicle condition at the historical moment is obtained by processing the vehicle condition at the historical moment according to a symmetry rule and a monotone rule.
17. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-readable program instructions, that, when executed by a computer, cause the computer to perform an autonomous lane change method, the autonomous lane change method comprising:
calculating a local neighbor feature and a global statistical feature of an autonomous vehicle at a current moment based on travel information of the autonomous vehicle at the current moment and motion information of obstacles in lanes within a sensing range of the autonomous vehicle, wherein the local neighbor feature represents motion status information of a neighboring obstacle of the autonomous vehicle relative to the autonomous vehicle, and the global statistical feature represents denseness of the obstacles in the lanes within the sensing range;
obtaining a target action indication based on the local neighbor feature, the global statistical feature, and a current control policy, wherein the target action indication indicates the autonomous vehicle to execute a target action, and the target action comprises at least one of a lane change type or a keeping straight type;
executing the target action according to the target action indication to cause the autonomous vehicle to execute the target action;
obtaining, by the processor, feedback information in response to executing the target action; and
updating, by the processor, the current control policy based on the feedback information to obtain an updated control policy,
wherein the feedback information is used to update the current control policy, wherein the feedback information comprises travel information generated after the autonomous vehicle executes the target action, future travel information of the autonomous vehicle, and future motion information of the obstacles in the lanes within the sensing range of the autonomous vehicle, and wherein when the target action is the lane change type, the feedback information further comprises a ratio of a period of time for executing the target action to a historical average period of time and a denseness change between obstacles in a lane in which the autonomous vehicle is located before the lane change type and obstacles in a lane in which the autonomous vehicle is located after the lane change type, wherein the historical average period of time is an average period of time for which the autonomous vehicle executes a similar action within a preset historical period of time.