Automatic learning of relevant actions in mobile robots
Described is a system and method for robotic perception and action. The system includes a mobile autonomous platform programmed to receive a visual input related to an environment and perform a movement sequence that affects the environment based on the visual input. A neural network is in communication with the mobile autonomous platform. The neural network includes a vision autoencoder trained, based on movements of the mobile autonomous platform, to encode visual features of an object in the visual input onto a visual latent space. In addition, the neural network includes an action autoencoder trained, based on movements of the mobile autonomous platform, to encode action features of the movement sequence onto an action latent space. The neural network also includes a mapping layer between the vision autoencoder and the action autoencoder. The mapping layer decodes the visual inputs into an action output of the mobile autonomous platform.
1. A system for automatic learning of relevant actions in mobile autonomous platforms, comprising:
a mobile autonomous platform programmed to receive a visual input related to an environment and perform a movement sequence that affects the environment based on the visual input;
a neural network in communication with the mobile autonomous platform, the neural network comprising:
a vision autoencoder trained, based on movements of the mobile autonomous platform, to encode visual features of a training object in the visual input onto a visual latent space;
an action autoencoder trained, based on movements of the mobile autonomous platform by the mobile autonomous platform, to encode action features of the movement sequence onto an action latent space; and
a mapping layer between the vision autoencoder and the action autoencoder, wherein the mapping layer maps the visual latent space to the action latent space and, given a new object that shares visual features with the training object and given a new objective, translates the visual features of the new object to action features and associated action possibilities of the mobile autonomous platform to achieve the objective.
2. The system as set forth in claim 1 , wherein the mapping layer encodes a series of action output controls for causing the autonomous platform to reach an object based on visual inputs from the autonomous platform to the training object.
3. The system as set forth in claim 1 , wherein the visual input comprises visual features specifying action possibilities for reaching the training object in the environment.
4. The system as set forth in claim 1 , wherein the visual features comprise texture gradients and edge features of the training object.
5. A method for automatic learning of relevant actions in mobile autonomous platforms, the method comprising:
receiving, by a mobile autonomous platform, a visual input related to an environment;
performing, by the mobile autonomous platform, a movement sequence that affects the environment;
training a vision autoencoder of a neural network to encode visual features of a training object in the visual input onto a visual latent space;
training an action autoencoder of the neural network to encode action features of the movement sequence onto an action latent space;
learning a mapping from the visual latent space to the action latent space;
using the mapping and given an objective, translating visual features from an instance of a new object that shares visual features with the training object to action features that specify action possibilities for the mobile autonomous platform to achieve the objective.
6. The method as set forth in claim 5 , wherein the visual features comprise texture gradients and edge features of the training object.
7. The method as set forth in claim 5 , further comprising training each of the vision autoencoder and the action autoencoder with a model loss function.
8. The method as set forth in claim 5 , further comprising triggering training of a mapping layer of the neural network with a model loss function when the mobile autonomous platform physically contacts the training object in the environment.
9. The method as set forth in claim 8 , wherein model loss is computed separately for the vision autoencoder, the action autoencoder, and the mapping layer.
10. The method as set forth in claim 9 , further comprising determining model loss functions for vision, action, and mapping, respectively, according to the following:
L
V
(
x
,
y
)
=
1
N
∑
n
N
y
n
log
σ
(
x
n
)
+
(
1
-
y
n
)
log
(
1
-
σ
(
x
n
)
)
L
A
(
x
,
y
)
=
1
N
∑
n
N
❘
"\[LeftBracketingBar]"
x
n
-
y
n
❘
"\[RightBracketingBar]"
L
M
(
x
,
y
)
=
1
N
∑
n
N
(
x
n
-
y
n
)
2
,
where for each loss function, x and y correspond to a decoded image and an input image for L V , a decoded action and an input action for L A , a mapped vision and encoded action for L M , N is a number of elements in x or y, and n is an index of a specific element.