Meta imitation learning with structured skill discovery
A method for acquiring skills through imitation learning by employing a meta imitation learning framework with structured skill discovery (MILD) is presented. The method includes learning behaviors or tasks, by an agent, from demonstrations: by learning to decompose the demonstrations into segments, via a segmentation component, the segments corresponding to skills that are transferrable across different tasks, learning relationships between the skills that are transferrable across the different tasks, employing, via a graph generator, a graph neural network for learning implicit structures of the skills from the demonstrations to define structured skills, and generating policies from the structured skills to allow the agent to acquire the structured skills for application to one or more target tasks.
1. A method for acquiring skills of medication through imitation learning, the method comprising:
learning behaviors or tasks, by an agent, from medical treatment demonstrations for a given disease by:
learning to decompose the demonstrations into segments, via a segmentation component, the segments corresponding to skills that are transferrable across different tasks;
learning relationships between the skills;
employing, via a graph generator, a graph neural network for learning implicit structures of the skills from the demonstrations to define structured skills; and
generating policies for medication from the structured skills to allow the agent to acquire the structured skills for application to one or more target tasks by optimizing an objective function:
min
π
θ
ℒ
(
p
π
E
p
π
θ
)
-
H
[
c
]
+
H
[
c
❘
"\[LeftBracketingBar]"
X
,
g
]
+
H
[
X
❘
"\[LeftBracketingBar]"
c
]
-
H
[
g
]
+
H
[
g
❘
"\[LeftBracketingBar]"
s
,
c
]
wherein p π θ is a generated policy with parameters π θ , p π E is an expert policy, is a distance function, H is a Shannon entropy function, c is a set of skills, X is an implicit structure, g is a segmentation, and s is a state; and
providing a medication to a patient in accordance with the generated policies to treat the given disease.
2. The method of claim 1 , wherein the demonstrations are expert trajectories generated by an expert policy for medication.
3. The method of claim 2 , wherein each of the expert trajectories includes state-action pairs.
4. The method of claim 3 , wherein the skills correspond to subsequences of the state-action pairs extracted from the expert trajectories.
5. The method of claim 1 , wherein the relationship between the skills is learned by employing a graph generator decoder to predict a graph probability distribution over latent interaction of the skills and a graph decoder to generate skills conditioned on a graph structure.
6. A non-transitory computer-readable storage medium comprising a computer-readable program for acquiring skills of medication through imitation learning, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
learning behaviors or tasks, by an agent, from medical treatment demonstrations of a given disease by:
learning to decompose the demonstrations into segments, via a segmentation component, the segments corresponding to skills that are transferrable across different tasks;
learning relationships between the skills;
employing, via a graph generator, a graph neural network for learning implicit structures of the skills from the demonstrations to define structured skills; and
generating policies for medication from the structured skills to allow the agent to acquire the structured skills for application to one or more target tasks by optimizing an objective function:
min
π
θ
ℒ
(
p
π
E
p
π
θ
)
-
H
[
c
]
+
H
[
c
❘
"\[LeftBracketingBar]"
X
,
g
]
+
H
[
X
❘
"\[LeftBracketingBar]"
c
]
-
H
[
g
]
+
H
[
g
❘
"\[LeftBracketingBar]"
s
,
c
]
wherein p π θ is a generated policy with parameters π θ , p π E is an expert policy, is a distance function, H is a Shannon entropy function, c is a set of skills, X is an implicit structure, g is a segmentation, and s is a state; and
providing a medication to a patient in accordance with the generated policies to treat the given disease.
7. The non-transitory computer-readable storage medium of claim 6 , wherein the demonstrations are expert trajectories generated by an expert policy for medication.
8. The non-transitory computer-readable storage medium of claim 7 , wherein each of the expert trajectories includes state-action pairs.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the skills correspond to subsequences of the state-action pairs extracted from the expert trajectories.
10. The non-transitory computer-readable storage medium of claim 6 , wherein the relationship between the skills is learned by employing a graph generator decoder to predict a graph probability distribution over latent interaction of the skills and a graph decoder to generate skills conditioned on a graph structure.
11. A system for acquiring skills of medication through imitation learning, the system comprising:
an imitation component to minimize a measure of discrepancy between a learned policy and an expert policy for medication by optimizing an objective function:
min
π
θ
ℒ
(
p
π
E
p
π
θ
)
-
H
[
c
]
+
H
[
c
❘
"\[LeftBracketingBar]"
X
,
g
]
+
H
[
X
❘
"\[LeftBracketingBar]"
c
]
-
H
[
g
]
+
H
[
g
❘
"\[LeftBracketingBar]"
s
,
c
]
wherein p π θ is a generated policy with parameters π θ , p π E is an expert policy, is a distance function, H is a Shannon entropy function, c is a set of skills, X is an implicit structure g is a segmentation, and s is a state;
a graph neural network to learn the implicit structure of skills from medical treatment demonstrations to define structured skills;
a meta controller to learn predictable skills;
a segmentation component to learn to decompose the demonstrations into segments corresponding to skills that are transferrable across different tasks concurrently with learning relationships between the skills; and
a treatment component to provide a medication to a patient in accordance with the learned policies to treat the given disease.
12. The system of claim 11 , wherein the demonstrations are expert trajectories generated by an expert policy for medication.
13. The system of claim 12 , wherein each of the expert trajectories includes state-action pairs.
14. The system of claim 13 , wherein the skills correspond to subsequences of the state-action pairs extracted from the expert trajectories.
15. The system of claim 11 , wherein the relationship between the skills is learned by employing a graph generator decoder to predict a graph probability distribution over latent interaction of the skills and a graph decoder to generate skills conditioned on a graph structure.
16. The system of claim 11 , wherein the learned policy for medication is optimized based on gradients of an imitation learning loss with respect to expert demonstrations.