Interpretable imitation learning via prototypical option discovery for decision making
A method for learning prototypical options for interpretable imitation learning is presented. The method includes initializing options by bottleneck state discovery, each of the options presented by an instance of trajectories generated by experts, applying segmentation embedding learning to extract features to represent current states in segmentations by dividing the trajectories into a set of segmentations, learning prototypical options for each segment of the set of segmentations to mimic expert policies by minimizing loss of a policy and projecting prototypes to the current states, training option policy with imitation learning techniques to learn a conditional policy, generating interpretable policies by comparing the current states in the segmentations to one or more prototypical option embeddings, and taking an action based on the interpretable policies generated.
1. A method for learning prototypical options for interpretable imitation learning, the method comprising:
initializing options by bottleneck state discovery, each of the options presented by an instance of trajectories generated by experts;
applying segmentation embedding learning to extract features to represent current states in segmentations by dividing the trajectories into a set of segmentations;
learning prototypical options for each segment of the set of segmentations to mimic expert policies by minimizing loss of a policy and projecting prototypes to the current states;
learning prototypical option embedding using an objective function:
ℒ
option
=
-
λ
1
*
ℒ
I
L
loss
+
λ
2
*
∑
i
=
1
K
min
m
=
1
M
=
f
ϕ
(
s
v
m
′
:
v
m
)
-
e
i
2
2
)
+
λ
3
*
∑
i
=
1
K
.
∑
j
=
i
+
1
K
max
(
0
,
d
min
-
e
i
-
e
j
)
where L IL loss is an imitation learning loss, f φ is, the second term is a segment representation function for a segment s ν′ m ,ν m from segment ν m to segment ν m′ , e i and e j are embedded prototypes, K is a number of prototypes, M is a number of segments, d min is a threshold value, and λ 1 , λ 2 , and λ 3 , are weighting parameters;
training option policy with imitation learning techniques to learn a conditional policy;
generating interpretable policies by comparing the current states in the segmentations to one or more prototypical option embeddings;
generating dosage options for a patient based on the interpretable policies;
displaying the dosage options on a user interface for a user; and
taking an action based on the dosage options.
2. The method of claim 1 , wherein option initialization includes identifying states from the current states that connect different densely connected regions in a state space.
3. The method of claim 2 , wherein a soft attention mechanism is employed to obtain important states with particular attention weights.
4. The method of claim 3 , wherein the important states are found with density-based spatial clustering of applications with noise (DBSCAN).
5. The method of claim 1 , wherein the bottleneck state discovery divides the trajectories generated by the experts into disjoint segments of variable length by a density-based clustering method.
6. The method of claim 1 , wherein each of the options includes an intra-option policy, a termination condition, an initiation state set, and an option prototype.
7. The method of claim 6 , wherein the option prototype is defined by a sub-trajectory generated by the experts.
8. The method of claim 1 , wherein each of the one or more prototypical option embeddings is assigned with a respective closest segment embedding in a training set.
9. The method of claim 1 , wherein the loss is a least square loss.
10. The method of claim 1 , wherein a diversity regularization term is employed to penalize one or more of the prototypical options that are close to each other.
11. A non-transitory computer-readable storage medium comprising a computer-readable program for learning prototypical options for interpretable imitation learning, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
initializing options by bottleneck state discovery, each of the options presented by an instance of trajectories generated by experts;
applying segmentation embedding learning to extract features to represent current states in segmentations by dividing the trajectories into a set of segmentations;
learning prototypical options for each segment of the set of segmentations to mimic expert policies by minimizing loss of a policy and projecting prototypes to the current states; learning prototypical option embedding using an objective function:
ℒ
option
=
-
λ
1
*
ℒ
I
L
loss
+
λ
2
*
∑
i
=
1
K
min
m
=
1
M
=
f
ϕ
(
s
v
m
′
:
v
m
)
-
e
i
2
2
)
+
λ
3
*
∑
i
=
1
K
.
∑
j
=
i
+
1
K
max
(
0
,
d
min
-
e
i
-
e
j
)
where L IL loss is an imitation learning loss, f φ is, the second term is a segment representation function for a segment s ν′ m ,ν m from segment ν m to segment ν m′ , e i and e j are embedded prototypes, K is a number of prototypes, M is a number of segments, d min is a threshold value, and λ 1 , λ 2 , and λ 3 are weighting parameters:
training option policy with imitation learning techniques to learn a conditional policy;
generating interpretable policies by comparing the current states in the segmentations to one or more prototypical option embeddings;
generating dosage options for a patient based on the interpretable policies;
displaying the dosage options on a user interface for a user; and
taking an action based on the interpretable policies generated dosage options.
12. The non-transitory computer-readable storage medium of claim 11 , wherein option initialization includes identifying states from the current states that connect different densely connected regions in a state space.
13. The non-transitory computer-readable storage medium of claim 12 , wherein a soft attention mechanism is employed to obtain important states with particular attention weights.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the important states are found with density-based spatial clustering of applications with noise (DBSCAN).
15. The non-transitory computer-readable storage medium of claim 11 , wherein the bottleneck state discovery divides the trajectories generated by the experts into disjoint segments of variable length by a density-based clustering method.
16. The non-transitory computer-readable storage medium of claim 11 , wherein each of the options includes an intra-option policy, a termination condition, an initiation state set, and an option prototype.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the option prototype is defined by a sub-trajectory generated by the experts.
18. The non-transitory computer-readable storage medium of claim 11 , wherein each of the one or more prototypical option embeddings is assigned with a respective closest segment embedding in a training set.