Meta imitation learning with structured skill discovery
A method for acquiring skills through imitation learning by employing a meta imitation learning framework with structured skill discovery (MILD) is presented. The method includes learning behaviors or tasks, by an agent, from demonstrations: by learning to decompose the demonstrations into segments, via a segmentation component, the segments corresponding to skills that are transferrable across different tasks, learning relationships between the skills that are transferrable across the different tasks, employing, via a graph generator, a graph neural network for learning implicit structures of the skills from the demonstrations to define structured skills, and generating policies from the structured skills to allow the agent to acquire the structured skills for application to one or more target tasks.
1. A method for acquiring skills through imitation learning, the method comprising:
learning behaviors or tasks, by an agent, from state-action pairs of medical treatment for a given disease by:
learning to decompose the state-action pairs into segments, via a segmentation component, the segments corresponding to skills that are transferrable across different tasks;
learning relationships between the skills;
employing, via a graph generator, a graph neural network for learning implicit structures of the skills from the state-action pairs to define structured skills; and
generating policies from the structured skills to allow the agent to acquire the structured skills for application to one or more target tasks by optimizing an objective function:
min
π
θ
ℒ
(
p
π
E
p
π
θ
)
-
H
[
c
]
+
H
[
c
❘
"\[LeftBracketingBar]"
X
,
g
]
+
H
[
X
❘
"\[LeftBracketingBar]"
c
]
-
H
[
g
]
+
H
[
g
❘
"\[LeftBracketingBar]"
s
,
c
]
wherein p πθ is a generated policy with parameters π θ , p π E is an expert policy, is a distance function, H is a Shannon entropy function, c is a set of skills, X is an implicit structure, g is a segmentation, and s is a state; and
providing a medication to a patient in accordance with the generated policies to treat the given disease.
2. The method of claim 1 , wherein the relationship between the skills is learned by employing a graph generator decoder to predict a graph probability distribution over latent interaction of the skills and a graph decoder to generate skills conditioned on a graph structure.
3. A non-transitory computer-readable storage medium comprising a computer-readable program for acquiring skills through imitation learning, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
learning behaviors or tasks, by an agent, from state-action pairs of medical treatment for a given disease by:
learning to decompose the state-action pairs into segments, via a segmentation component, the segments corresponding to skills that are transferrable across different tasks;
learning relationships between the skills;
employing, via a graph generator, a graph neural network for learning implicit structures of the skills from the state-action pairs to define structured skills; and
generating policies from the structured skills to allow the agent to acquire the structured skills for application to one or more target tasks by optimizing an objective function:
min
π
θ
ℒ
(
p
π
E
p
π
θ
)
-
H
[
c
]
+
H
[
c
❘
"\[LeftBracketingBar]"
X
,
g
]
+
H
[
X
❘
"\[LeftBracketingBar]"
c
]
-
H
[
g
]
+
H
[
g
❘
"\[LeftBracketingBar]"
s
,
c
]
wherein p π θ is a generated policy with parameters π θ , p π E is an expert policy, is a distance function, H is a Shannon entropy function, c is a set of skills, X is an implicit structure, g is a segmentation, and s is a state; and
providing a medication to a patient in accordance with the generated policies to treat the given disease.
4. The non-transitory computer-readable storage medium of claim 3 , wherein the relationship between the skills is learned by employing a graph generator decoder to predict a graph probability distribution over latent interaction of the skills and a graph decoder to generate skills conditioned on a graph structure.
5. A system for acquiring skills through imitation learning, the system comprising:
an imitation component to minimize a measure of discrepancy between a learned policy and an expert policy by optimizing an objective function:
min
π
θ
ℒ
(
p
π
E
p
π
θ
)
-
H
[
c
]
+
H
[
c
❘
"\[LeftBracketingBar]"
X
,
g
]
+
H
[
X
❘
"\[LeftBracketingBar]"
c
]
-
H
[
g
]
+
H
[
g
❘
"\[LeftBracketingBar]"
s
,
c
]
wherein p π θ is a generated policy with parameters π θ , p π E is an expert policy, is a distance function, H is a Shannon entropy function, c is a set of skills, X is an implicit structure, g is a segmentation, and s is a state;
a graph neural network to learn the implicit structure of skills from state-action pairs of medical treatment to define structured skills;
a meta controller to learn predictable skills;
a segmentation component to learn to decompose the state-action pairs into segments corresponding to skills that are transferrable across different tasks concurrently with learning relationships between the skills; and
a treatment component to provide a medication to a patient in accordance with the learned policies to treat the given disease.
6. The system of claim 5 , wherein the relationship between the skills is learned by employing a graph generator decoder to predict a graph probability distribution over latent interaction of the skills and a graph decoder to generate skills conditioned on a graph structure.
7. The system of claim 5 , wherein the learned policy is optimized based on gradients of an imitation learning loss with respect to expert state-action pairs.