Automatic driving decision making method, system and device and computer storage medium
An automatic driving method includes following steps: S 101 : acquiring real-time traffic environment information in a travel process of an autonomous vehicle at a current moment; S 102 : mapping the real-time traffic environment information based on a preset mapping relationship to obtain mapped traffic environment information; S 103 : adjusting a target deep reinforcement learning model based on a pre-stored existing deep reinforcement learning model and the mapped traffic environment information; and S 104 : judging whether automatic driving is finished, and in response to the automatic driving is not finished, returning to perform the step of acquiring the real-time traffic environment information in the travel process of the autonomous vehicle at the current moment. An automatic driving system, an automatic driving device and a computer medium storing the automatic driving method are further provided.
1. An automatic driving method, comprising:
acquiring, by vehicle-mounted sensor apparatuses, real-time traffic environment information in a travel process of an autonomous vehicle at a current moment; wherein a next driving action of the autonomous vehicle is predicted according to the real-time traffic environment information in an automatic driving process;
mapping, by an automatic driving computing device, the real-time traffic environment information based on a preset mapping relationship to obtain mapped traffic environment information;
adjusting, by the automatic driving computing device, a target deep reinforcement learning model based on a pre-stored existing deep reinforcement learning model and the mapped traffic environment information; and
judging, by the automatic driving computing device, whether automatic driving is finished, and in response to the automatic driving is not finished, returning to perform the step of acquiring the real-time traffic environment information in the travel process of the autonomous vehicle at the current moment,
wherein the mapping relationship comprises a mapping relationship between the real-time traffic environment information and existing traffic environment information of the existing deep reinforcement learning model; the mapped traffic environment information is processed on the basis of parameters of existing policy network of the existing deep reinforcement learning model to obtain the next driving action of the autonomous vehicle;
wherein before mapping the real-time traffic environment information based on the preset mapping relationship to obtain the mapped traffic environment information, the method further comprises: acquiring target traffic environment information; reading the existing traffic environment information; calculating a distance value between the target traffic environment information and the existing traffic environment information in a reproducing kernel Hilbert space; and determining the mapping relationship by minimizing the distance value.
2. The method as claimed in claim 1 , wherein the step of adjusting the target deep reinforcement learning model based on the pre-stored existing deep reinforcement learning model and the mapped traffic environment information comprises:
processing the mapped traffic environment information based on parameters of an existing policy network of the existing deep reinforcement learning model to obtain a vehicle action;
calculating a value of a value function of the vehicle action based on an evaluation network of the target deep reinforcement learning mode;
acquiring a return value of the vehicle action; and
updating parameters of the evaluation network based on the return value and the value of the value function.
3. The method as claimed in claim 2 , wherein the step of updating parameters of the evaluation network based on the return value and the value of the value function comprises:
calculating a loss value by a calculation formula of loss function based on the return value and the value of the value function; and
updating the parameters of the evaluation network by minimizing the loss value,
wherein, the loss function computational formula comprises:
L
=
1
N
∑
t
(
r
t
+
γ
Q
ω
′
(
s
t
+
1
,
a
t
+
1
)
-
Q
ω
(
s
t
,
a
t
)
)
2
;
wherein, L represents the loss value; N represents a number of collected samples; r t represents a return value at time t; γ represents a discount factor, and 0<γ<1; Q′ ω (s t+1 , a t+1 ) represents a value of the value function calculated by a target network in the evaluation network at time t+1; s t+1 represents traffic environment information at time t+1; a t+1 represents a vehicle action at time t+1; Q ω (s t , a t ) represents a value of the value function calculated by a prediction network in the evaluation network at time t; s t represents traffic environment information at time t; and a t represents a vehicle action at time t.
4. The method as claimed in claim 2 , wherein after updating the parameters of the evaluation network based on the return value and the value of the value function, the method further comprises:
updating parameters of a policy network of the target deep reinforcement learning model.
5. The method as claimed in claim 1 , wherein the step of calculating the distance value between the target traffic environment information and the existing traffic environment information in the reproducing kernel Hilbert space comprises:
calculating the distance value between the target traffic environment information and the existing traffic environment information by a calculation formula of distance value in the reproducing kernel Hilbert space,
wherein, the calculation formula of distance value comprises:
MMD
H
(
D
S
,
D
T
)
=
1
n
∑
i
=
1
n
A
T
s
Si
-
1
m
∑
j
=
1
m
A
T
s
Tj
H
;
wherein, MMD H (D S , D T ) represents the distance value; D S represents the existing traffic environment information; D T represents the target traffic environment information; n represents the number of samples in the existing traffic environment information; m represents the number of samples in the target traffic environment information; A represents the mapping relationship; T represents transposition; s S represents traffic environment information in the existing traffic environment information; s T represents traffic environment information in the target traffic environment information; and H represents the reproducing kernel Hilbert space.
6. The method as claimed in claim 5 , wherein, the step of determining the mapping relationship by minimizing the distance value comprises:
determining the mapping relationship by minimizing the distance value based on a regular linear regression method or a support vector machine method or a principal component analysis method.
7. An automatic driving device, comprising:
a memory configured to store computer programs; and
a processor configured to perform the computer programs to perform the steps of:
acquiring, by vehicle-mounted sensor apparatuses, real-time traffic environment information in a travel process of an autonomous vehicle at a current moment; wherein a next driving action of the autonomous vehicle is predicted according to the real-time traffic environment information in an automatic driving process;
mapping, by an automatic driving computing device, the real-time traffic environment information based on a preset mapping relationship to obtain mapped traffic environment information;
adjusting, by the automatic driving computing device, a target deep reinforcement learning model based on a pre-stored existing deep reinforcement learning model and the mapped traffic environment information; and
judging, by the automatic driving computing device, whether automatic driving is finished, and under a condition that the automatic driving is not finished, returning to perform the step of acquiring the real-time traffic environment information in the travel process of the autonomous vehicle at the current moment,
wherein the mapping relationship comprises a mapping relationship between the real-time traffic environment information and existing traffic environment information of the existing deep reinforcement learning model; the mapped traffic environment information is processed on the basis of parameters of existing policy network of the existing deep reinforcement learning model to obtain the next driving action of the autonomous vehicle;
the processor is configured to: acquire target traffic environment information; read the existing traffic environment information; calculate a distance value between the target traffic environment information and the existing traffic environment information in a reproducing kernel Hilbert space; and determine the mapping relationship by minimizing the distance value.
8. A non-transitory computer-readable storage medium storing computer program that, when executed by a processor, cause the processor to:
acquire, by vehicle-mounted sensor apparatuses, real-time traffic environment information in a travel process of an autonomous vehicle at a current moment; wherein a next driving action of the autonomous vehicle is predicted according to the real-time traffic environment information in an automatic driving process;
map, by an automatic driving computing device, the real-time traffic environment information based on a preset mapping relationship to obtain mapped traffic environment information;
adjust, by the automatic driving computing device, a target deep reinforcement learning model based on a pre-stored existing deep reinforcement learning model and the mapped traffic environment information; and
judge, by the automatic driving computing device, whether automatic driving is finished, and under a condition that the automatic driving is not finished, returning to perform the step of acquiring the real-time traffic environment information in the travel process of the autonomous vehicle at the current moment,
wherein the mapping relationship comprises a mapping relationship between the real-time traffic environment information and existing traffic environment information of the existing deep reinforcement learning model; the mapped traffic environment information is processed on the basis of parameters of existing policy network of the existing deep reinforcement learning model to obtain the next driving action of the autonomous vehicle;
the processor is configured to: acquire target traffic environment information; read the existing traffic environment information; calculate a distance value between the target traffic environment information and the existing traffic environment information in a reproducing kernel Hilbert space; and determine the mapping relationship by minimizing the distance value.
9. The method as claimed in claim 2 , wherein, the step of acquiring a return value of the vehicle action comprises:
acquiring the return value according to the vehicle action in combination with a set standard.
10. The automatic driving device as claimed in claim 7 , the processor is configured to:
process the mapped traffic environment information based on parameters of an existing policy network of the existing deep reinforcement learning model to obtain a vehicle action;
calculate a value of a value function of the vehicle action based on an evaluation network of the target deep reinforcement learning model;
acquire a return value of the vehicle action; and
update parameters of the evaluation network based on the return value and the value of the value function.
11. The automatic driving device as claimed in claim 10 , the processor is configured to:
calculate a loss value by a calculation formula of loss function based on the return value and the value of the value function; and
update the parameters of the evaluation network by minimizing the loss value,
wherein the calculation formula of loss function comprises:
L
=
1
N
∑
t
(
r
t
+
γ
Q
ω
′
(
s
t
+
1
,
a
t
+
1
)
-
Q
ω
(
s
t
,
a
t
)
)
2
;
wherein L represents the loss value; N represents a number of collected samples; r t represents a return value at time t; γ represents a discount factor, and 0<γ<1; Q′ ω (s t+1 , a t+1 ) represents a value of the value function calculated by a target network in the evaluation network at time t+1; s t+1 represents traffic environment information at time t+1; a t+1 represents a vehicle action at time t+1; Q ω (s t , a t ) represents a value of the value function calculated by a prediction network in the evaluation network at time t; s t represents traffic environment information at time t; and a t represents a vehicle action at time t.
12. The automatic driving device as claimed in claim 10 , the processor is configured to:
update parameters of a policy network of the target deep reinforcement learning model.
13. The automatic driving device as claimed in claim 7 , the processor is configured to:
calculate the distance value between the target traffic environment information and the existing traffic environment information by a calculation formula of distance value in the reproducing kernel Hilbert space,
wherein, the calculation formula of distance value comprises:
MMD
H
(
D
S
,
D
T
)
=
1
n
∑
i
=
1
n
A
T
s
Si
-
1
m
∑
j
=
1
m
A
T
s
Tj
H
;
wherein, MMD H (D S , D T ) represents the distance value; D S represents the existing traffic environment information; D T represents the target traffic environment information; n represents the number of samples in the existing traffic environment information; m represents the number of samples in the target traffic environment information; A represents the mapping relationship; T represents transposition; s S represents traffic environment information in the existing traffic environment information; s T represents traffic environment information in the target traffic environment information; and H represents the reproducing kernel Hilbert space.
14. The automatic driving device as claimed in claim 13 , the processor is configured to:
determine the mapping relationship by minimizing the distance value based on a regular linear regression method or a support vector machine method or a principal component analysis method.
15. The non-transitory computer-readable storage medium as claimed in claim 8 , the processor is configured to:
process the mapped traffic environment information based on parameters of an existing policy network of the existing deep reinforcement learning model to obtain a vehicle action;
calculate a value of a value function of the vehicle action based on an evaluation network of the target deep reinforcement learning model;
acquire a return value of the vehicle action; and
update parameters of the evaluation network based on the return value and the value of the value function.
16. The non-transitory computer-readable storage medium as claimed in claim 15 , the processor is configured to:
calculate a loss value by a calculation formula of loss function based on the return value and the value of the value function; and
update the parameters of the evaluation network by minimizing the loss value,
wherein, the loss function computational formula comprises:
L
=
1
N
∑
t
(
r
t
+
γ
Q
ω
′
(
s
t
+
1
,
a
t
+
1
)
-
Q
ω
(
s
t
,
a
t
)
)
2
;
wherein, L represents the loss value; N represents a number of collected samples; r t represents a return value at time t; γ represents a discount factor, and 0<γ<1; Q′ ω (s t+1 , a t+1 ) represents a value of the value function calculated by a target network in the evaluation network at time t+1; s t+1 represents traffic environment information at time t+1; a t+1 represents a vehicle action at time t+1; Q ω (s t , a t ) represents a value of the value function calculated by a prediction network in the evaluation network at time t; s t represents traffic environment information at time t; and a t represents a vehicle action at time t.
17. The non-transitory computer-readable storage medium as claimed in claim 15 , the processor is configured to:
update parameters of a policy network of the target deep reinforcement learning model.