Self-driving method, training method, and related apparatus
A self-driving method and a related apparatus, the method including determining, by a self-driving apparatus, a task feature vector of a self-driving task according to M groups of historical paths of the self-driving task, where the task feature vector is a vector representing features of the self-driving task, and where M is an integer greater than 0, determining, by the self-driving apparatus, according to the task feature vector and a status vector, a target driving operation that needs to be performed, where the status vector indicates a driving status of the self-driving apparatus, and performing, by the self-driving apparatus, the target driving operation.
1 . A self-driving method, comprising:
obtaining a self-driving task by a self-driving apparatus;
determining, by the self-driving apparatus, during performance of the self-driving task, a task feature vector of the self-driving task according to M groups of historical paths of the self-driving task, wherein the task feature vector is a vector representing features of the self-driving task, wherein M is an integer greater than 0, wherein each historical path of the M groups of historical paths comprises at least one tuple corresponding to a time step, wherein each tuple comprises a driving status of the self-driving apparatus at a corresponding time step after the self-driving apparatus performs a driving operation at a previous time step, a driving operation performed by the self-driving apparatus at the corresponding time step, and a reward value obtained by the self-driving apparatus by performing the driving operation at the corresponding time step, wherein the determining the task feature vector comprises obtaining, by the self-driving apparatus, the M groups of historical paths, wherein the determining the task feature vector further comprises obtaining M groups of intermediate feature vectors by performing, by the self-driving apparatus, feature extraction on the M groups of historical paths to generate task features using a task abstraction network comprising one or more hidden layers and a final layer that is a fully connected layer, and wherein the task features are extracted at the one or more hidden layers to generate M groups of intermediate feature vectors, that are mapped, by the fully connected layer, to the task feature vector, which is an H-dimensional feature vector;
updating, by the self-driving apparatus, before the obtaining the M groups of historical paths and according to N losses corresponding to N groups of historical paths, a group of task abstraction network parameters corresponding to the self-driving task using at least gradient values of a first loss function relative to parameters of the task abstraction network;
determining, by the self-driving apparatus, according to the task feature vector and a status vector, a target driving operation that needs to be performed, wherein the status vector indicates a driving status of the self-driving apparatus, wherein the status vector is a vector generated by the self-driving apparatus based on data sensed by a sensor system of the self-driving apparatus and represents a driving status of the self-driving apparatus at a time step, wherein the status vector comprises at least a location, a speed, a steering angle, and an acceleration of the self-driving apparatus, and further comprises a status of an ambient environment of the self-driving apparatus, wherein the status vector is a G-dimensional vector, wherein the target driving operation that needs to be performed is determined by providing the task feature vector and the status vector as inputs to a shared policy network, and wherein the target driving operation that needs to be performed is further determined according to a relationship between a target threshold and a value obtained by dividing G by H; and
performing, by the self-driving apparatus, the target driving operation by generating a control instruction associated with the target driving operation and indicating one or more operational parameters for one or more controls of a vehicle controlled by the self-driving apparatus, and further by sending the control instruction to a control system, wherein sending the control instruction to the control system causes the control system to control the one or more controls according to the control instruction and further causes the self-driving apparatus to perform the self-driving task based on the target driving operation;
wherein a calculation formula of the first loss function is:
L
(
θ
TA
)
=
E
τ
[
∑
t
=
0
H
(
log
π
θ
policy
(
a
t
❘
s
t
,
h
(
θ
TA
)
)
(
∑
t
′
=
t
H
r
t
′
-
b
(
s
t
)
)
)
]
;
wherein θ TA represents a parameter of the task abstraction network, wherein φ θ policy (a t |s t ,h(θ TA )) represents a probability that an input of the shared policy network is a status vector s t and a feature vector h(θ TA ) and an output is a driving operation a t , wherein H is a maximum quantity of points, wherein t represents a time point, wherein b(s t ) represents a reference function used to calculate a return obtained by the self-driving apparatus at the time point t, wherein r t ′ represents a real return obtained by the self-driving apparatus at the time point t, and wherein E τ [ ] represents calculating an expectation in a bracket.
2 . The method according to claim 1 , wherein the M groups of intermediate feature vectors correspond in a one-to-one manner to the M groups of historical paths.
3 . The method according to claim 2 , wherein the shared policy network is a neural network implementing reinforcement learning.
4 . The method according to claim 3 , wherein the obtaining the M groups of historical paths comprises obtaining at least one group of historical paths of the M groups of historical paths by performing:
obtaining a first driving operation by processing, by the self-driving apparatus, a reference feature vector and a first status vector using the shared policy network, wherein the reference feature vector is one of a zero vector or a feature vector that is of the self-driving task and that is determined according to N groups of historical paths of the self-driving task, wherein the first status vector indicates a driving status of the self-driving apparatus at a first time point, and wherein N is an integer greater than 0;
performing, by the self-driving apparatus, the first driving operation at the first time point;
obtaining, by the self-driving apparatus, a driving status of the self-driving apparatus at a second time point and a return for performing the first driving operation, wherein the second time point is a next time point of the first time point; and
determining, by the self-driving apparatus, a group of historical paths according to information about T time points for completing or executing the self-driving task, wherein the T time points comprise the first time point, wherein information about any one time point of the T time points comprises a driving status of the self-driving apparatus at the any one time point, a driving operation performed by the self-driving apparatus at the any one time point, and a return obtained by the self-driving apparatus at the any one time point, and wherein T is an integer not less than 1.
5 . The method according to claim 4 , further comprising performing, before the processing the reference feature vector:
obtaining, by the self-driving apparatus, the N groups of historical paths, wherein the N losses correspond in a one-to-one manner to the N groups of historical paths;
obtaining N intermediate feature vectors by performing, by the self-driving apparatus, feature extraction on the N groups of historical paths using the task abstraction network, wherein the N intermediate feature vectors correspond in a one-to-one manner to the N groups of historical paths, and wherein the task abstraction network processes the N groups of historical paths using the group of task abstraction network parameters; and
determining, by the self-driving apparatus, the reference feature vector according to the N intermediate feature vectors.
6 . The method according to claim 5 , wherein the updating the group of task abstraction network parameters comprises:
updating, by the self-driving apparatus using the gradient values of the first loss function relative to parameters of the task abstraction network, a group of relative parameters of the task abstraction network with respect to the self-driving task, wherein the group of task abstraction network parameters comprises differences between the parameters of the task abstraction network and the group of relative parameters, and wherein the first loss function provides a sum of the N losses.
7 . The method according to claim 5 , further comprising:
updating, by the self-driving apparatus, parameters of the task abstraction network using gradient values of a second loss function relative to the parameters of the task abstraction network, wherein the second loss function provides at least one of a sum of F losses of F tasks or a mean of the F losses, wherein the F losses correspond in a one-to-one manner to the F tasks, wherein a loss of any one task of the F tasks is obtained through calculation according to a latest batch of historical paths of the any one task, and wherein Fis an integer greater than 1; and
updating, by the self-driving apparatus, parameters of the shared policy network using gradient values of the second loss function relative to the parameters of the shared policy network.
8 . The method according to claim 7 , further comprising:
updating, by the self-driving apparatus, a target learning rate using gradient values of the second loss function relative to the target learning rate, wherein the target learning rate is a learning rate for updating the group of task abstraction network parameters of the task abstraction network with respect to the self-driving task.
9 . The method according to claim 8 , wherein a calculation formula of the second loss function is:
L (θ)= E t [min{ r t (θ) A t ,clip( r t (θ),1−ε,1+ε, r t (θ)) A t }−βKL[π θ old ,π θ ]];
wherein θ={θ TA , θ policy , α}, wherein θ TA represents a parameter of the task abstraction network, wherein θ policy represents a parameter of the shared policy network, and α represents the target learning rate, wherein
r
t
(
θ
)
=
π
θ
(
a
t
❘
s
t
)
π
θ
old
(
a
t
❘
s
t
)
,
wherein π θ (a t |s t ) represents a probability that, after θ is updated, an input of the shared policy network is a status vector s t and an output is a driving operation a t , wherein φ θ old (a t |s t ) represents a probability that, before θ is updated, an input of the shared policy network is the status vector s t and an output is the driving operation a t , wherein
A
t
=
∑
t
′
=
t
H
r
t
′
-
b
(
s
t
)
,
wherein H is a maximum quantity of points, wherein b(s t ) represents a reference function used to calculate a return obtained by the self-driving apparatus at a time point t, wherein r t ′ represents a real return obtained by the self-driving apparatus at the time point t, wherein clip (r t (θ), 1−ε, 1+ε, r t (θ)) represents that a minimum value of r t (θ) is 1−ε, a maximum value of r t (θ) is 1+ε, and ε is a constant term greater than 0, wherein KL[π_(θ_old),π_θ]] represents relative entropy of two probability distributions obtained before and after θ is updated, and β is a hyperparameter, wherein and E t [ ] represents calculating an expectation in a bracket.
10 . A self-driving apparatus, comprising:
a processor;
a control system; and
a non-transitory computer readable medium connected to the processor and storing program for execution by the processor, the program including instructions to:
obtain a self-driving task by the self-driving apparatus;
determine, during performance of the self-driving task by the self-driving apparatus, a task feature vector of the self-driving task based on M groups of historical paths of the self-driving task, wherein the task feature vector is a vector representing features of the self-driving task, wherein M is an integer greater than 0, wherein each historical path of the M groups of historical paths comprises at least one tuple corresponding to a time step, wherein each tuple comprises a driving status of the self-driving apparatus at a corresponding time step after the self-driving apparatus performs a driving operation at a previous time step, a driving operation performed by the self-driving apparatus at the corresponding time step, and a reward value obtained by the self-driving apparatus by performing the driving operation at the corresponding time step, wherein the determining the task feature vector comprises obtaining, by the self-driving apparatus, the M groups of historical paths, wherein the determining the task feature vector further comprises obtaining M groups of intermediate feature vectors by performing, by the self-driving apparatus, feature extraction on the M groups of historical paths to generate task features using a task abstraction network comprising one or more hidden layers and a final layer that is a fully connected layer, and wherein the task features are extracted at the one or more hidden layers to generate M groups of intermediate feature vectors, that are mapped, by the fully connected layer, to the task feature vector, which is an H-dimensional feature vector;
update, before the obtaining the M groups of historical paths and according to N losses corresponding to N groups of historical paths, a group of task abstraction network parameters corresponding to the self-driving task using at least gradient values of a first loss function relative to parameters of the task abstraction network;
determine, according to the task feature vector and a status vector, a target driving operation that needs to be performed, wherein the status vector indicates a driving status of the self-driving apparatus, wherein the status vector is a vector generated by the self-driving apparatus based on data sensed by a sensor system of the self-driving apparatus and represents a driving status of the self-driving apparatus at a time step, wherein the status vector comprises at least a location, a speed, a steering angle, and an acceleration of the self-driving apparatus, and further comprises a status of an ambient environment of the self-driving apparatus, wherein the status vector is a G-dimensional vector, wherein the target driving operation that needs to be performed is determined by providing the task feature vector and the status vector as inputs to a shared policy network, and wherein the target driving operation that needs to be performed is determined according to a relationship between a target threshold and a value obtained by dividing G by H; and
cause the control system to control the self-driving apparatus to perform the target driving operation by generating a control instruction associated with the target driving operation and indicating one or more operational parameters for one or more controls of a vehicle controlled by the self-driving apparatus, and further by sending the control instruction to the control system, wherein sending the control instruction to the control system causes the control system to control the one or more controls according to the control instruction and further causes the self-driving apparatus to perform the self-driving task based on the target driving operation;
wherein a calculation formula of the first loss function is:
L
(
θ
TA
)
=
E
τ
[
∑
t
=
0
H
(
log
π
θ
policy
(
a
t
❘
s
t
,
h
(
θ
TA
)
)
(
∑
t
′
=
t
H
r
t
′
-
b
(
s
t
)
)
)
]
;
wherein θ TA represents a parameter of the task abstraction network, wherein π θ policy (a t |s t ,h(θ TA ) represents a probability that an input of the shared policy network is a status vector s t and a feature vector h(θ TA ) and an output is a driving operation a t , wherein H is a maximum quantity of points, wherein t represents a time point, wherein b(s t ) represents a reference function used to calculate a return obtained by the self-driving apparatus at the time point t, wherein r t ′ represents a real return obtained by the self-driving apparatus at the time point t, and wherein E τ [ ] represents calculating an expectation in a bracket.
11 . The self-driving apparatus according to claim 10 ,
wherein the M groups of intermediate feature vectors correspond in a one-to-one manner to the M groups of historical paths.
12 . The self-driving apparatus according to claim 11 ,
wherein the shared policy network is a neural network configured to implement reinforcement learning.
13 . The self-driving apparatus according to claim 12 , wherein the instructions to obtain the M groups of historical paths include instructions to obtain at least one group of historical paths of the M groups of historical paths, and wherein the instructions to obtain at least one group of historical paths of the M groups of historical paths include instructions to:
obtain a first driving operation by processing a reference feature vector and a first status vector using the shared policy network, wherein the reference feature vector is at least one of a zero vector or a feature vector that is of the self-driving task and that is determined according to N groups of historical paths of the self-driving task, wherein the first status vector represents a driving status of the self-driving apparatus at a first time point, and wherein N is an integer greater than 0;
cause the control system to control the self-driving apparatus to perform the first driving operation at the first time point;
obtain a driving status of the self-driving apparatus at a second time point and a return for performing the first driving operation, wherein the second time point is a next time point of the first time point; and
determine a group of historical paths according to information about T time points for completing or executing the self-driving task, wherein the T time points comprise the first time point, wherein information about any one time point of the T time points comprises a driving status of the self-driving apparatus at the any one time point, a driving operation performed by the self-driving apparatus at the any one time point, and a return obtained by the self-driving apparatus at the any one time point, and wherein T is an integer not less than 1.
14 . The self-driving apparatus according to claim 13 , the program further includes instructions to:
obtain the N groups of historical paths,
wherein the N losses correspond in a one-to-one manner to the N groups of historical paths;
obtain N intermediate feature vectors by performing feature extraction on the N groups of historical paths using the task abstraction network, wherein the N intermediate feature vectors correspond in a one-to-one manner to the N groups of historical paths, and wherein the task abstraction network processes the N groups of historical paths using the group of task abstraction network parameters; and
determine the reference feature vector according to the N intermediate feature vectors.
15 . The self-driving apparatus according to claim 14 , wherein the instructions to update the group of task abstraction network parameters include instructions to:
update a group of relative parameters of the task abstraction network with respect to the self-driving task using the gradient values of the first loss function relative to parameters of the task abstraction network, wherein the group of task abstraction network parameters are differences between the parameters of the task abstraction network and the group of relative parameters, and wherein the first loss function provides a sum of the N losses.
16 . The self-driving apparatus according to claim 14 , wherein the program further includes instructions to:
update parameters of the task abstraction network by using gradient values of a second loss function relative to the parameters of the task abstraction network, wherein the second loss function is configured to provide at least one of a sum of F losses of F self-driving tasks or a mean of the F losses, wherein the F losses correspond in a one-to-one manner to the F self-driving tasks, wherein a loss of any one task of the F self-driving tasks is obtained through calculation based on a latest batch of historical paths of the any one task, and wherein F is an integer greater than 1; and
update parameters of the shared policy network using gradient values of the second loss function relative to the parameters of the shared policy network.
17 . The self-driving apparatus according to claim 16 , wherein the program further includes instructions to
update a target learning rate using gradient values of the second loss function relative to the target learning rate, wherein the target learning rate is a learning rate for updating the group of task abstraction network parameters of the task abstraction network with respect to the self-driving task.
18 . The self-driving apparatus according to claim 17 , wherein a calculation formula of the second loss function is:
L (θ)= E t [min{ r t (θ) A t ,clip( r t (θ),1−ε,1+ε, r t (θ)) A t }−βKL[π θ old ,π θ ]];
wherein θ={θ TA , θ policy , α}, wherein θ TA represents a parameter of the task abstraction network, θ policy represents a parameter of the shared policy network, and α represents the target learning rate, wherein
r
t
(
θ
)
=
π
θ
(
a
t
❘
s
t
)
π
θ
old
(
a
t
❘
s
t
)
,
wherein π θ (a t |s t ) represents a probability that, after θ is updated, an input of the shared policy network is a status vector s t and an output is a driving operation a t , wherein π θ old (a t |s t ) represents a probability that, before e is updated, an input of the shared policy network is the status vector s t and an output is the driving operation a t ;
A
t
=
∑
t
′
=
t
H
r
t
′
-
b
(
s
t
)
,
wherein H is a maximum quantity of points, b(s t ) represents a reference function used to calculate a return obtained by the self-driving apparatus at a time point t, wherein r t ′ represents a real return obtained by the self-driving apparatus at the time step t, wherein clip (r t (θ), 1−ε, 1+ε, r t (θ)) represents that a minimum value of r t (θ) is 1−ε, a maximum value of r t (θ) is 1+ε, and ε is a constant term greater than 0; KL[π_(θ_old),π_θ] represents relative entropy of two probability distributions obtained before and after θ is updated, wherein β is a hyperparameter, and wherein E t [ ] represents calculating an expectation in a bracket.