METHODS FOR ESTABLISHING AND UTILIZING SENSORIMOTOR PROGRAMS
A method for establishing sensorimotor programs includes specifying a concept relationship that relates a first concept to a second concept and establishes the second concept as higher-order than the first concept; training a first sensorimotor program to accomplish the first concept using a set of primitive actions; and training a second sensorimotor program to accomplish the second concept using the first sensorimotor program and the set of primitive actions.
1 . A method for establishing sensorimotor programs, comprising:
determining a first environment that represents a first concept; and
training a first sensorimotor program to accomplish the first concept by interacting with the first environment, comprising training the first sensorimotor program using a first reward function that rewards the first sensorimotor program when the first sensorimotor program successfully accomplishes the first concept and correctly returns a value indicating that the first sensorimotor program has successfully accomplished the first concept.
2 . The method of claim 1 , wherein the first concept is a bring-about concept; wherein the first reward function rewards when the first sensorimotor program successfully brings about the first concept and correctly returns a value indicating that the first sensorimotor program has successfully brought about the first concept.
3 . The method of claim 1 , wherein the first concept is a classification concept; wherein the first reward function that rewards when the first sensorimotor program correctly returns a value consistent with the presence or non-presence of the first concept in the first environment.
4 . The method of claim 1 , wherein training the first sensorimotor program to accomplish the first concept further comprises training the first sensorimotor program using a second reward function different from the first reward function that rewards when the first sensorimotor program successfully accomplishes the first concept but fails to return a value indicating that the first sensorimotor program has successfully accomplished the first concept.
5 . The method of claim 4 , wherein the second reward function is a shaping reward function.
6 . The method of claim 1 , wherein training the first sensorimotor program to accomplish the first concept comprises using a set of primitive actions to interact with the first environment.
7 . The method of claim 7 , wherein using the set of primitive actions to interact with the first environment comprises pushing an object of the first environment.
8 . The method of claim 7 , further comprising executing the trained first sensorimotor program on a robotic arm system comprising a robotic arm actuator, wherein an action of the set of primitive actions actuates the robotic arm actuator.
9 . A method for establishing sensorimotor programs, comprising:
generating a first plurality of environments that represents a first concept, wherein the first concept is a bring-about concept; and
training a first sensorimotor program to accomplish the first concept using a reward function in each environment of the first plurality, wherein the first sensorimotor program executes actions of a set of primitive actions to accomplish the first concept.
10 . The method of claim 9 , wherein the first plurality of environments is generated based on recurring content that enables re-use of learned concepts.
9 . method of claim 9 , wherein each environment of the first plurality of environments is associated with a dynamics model that collectively simulates the actions executed by the first sensorimotor program.
12 . The method of claim 9 , wherein generating the first plurality of environments comprises generating a superset of environments and filtering the superset of environments to determine the first plurality of environments.
13 . The method of claim 9 , further comprising:
generating a second plurality of environments that represent a second concept higher-order than the first concept; and
training, using the second plurality of environments, a second sensorimotor program to accomplish the second concept using the first sensorimotor program.
14 . The method of claim 13 , wherein the second sensorimotor program is trained using the set of primitive actions, wherein the second sensorimotor program calls the first sensorimotor program as an additional action.
15 . The method of claim 13 , wherein the second sensorimotor program calls the first sensorimotor program as an observation.
16 . The method of claim 9 , wherein the first sensorimotor program is trained based on a set of actionable lower-level sensorimotor programs associated with different respective bring-about concepts and a set of conceptual lower-level sensorimotor programs associated with different respective classification concepts.
17 . The method of claim 16 , wherein the set of actionable lower-level sensorimotor programs and the set of conceptual lower-level sensorimotor programs are trained before the first sensorimotor program is trained.
18 . The method of claim 16 , wherein training the first sensorimotor program comprises automatically determining which programs of the actionable lower-level sensorimotor programs and the conceptual lower-level sensorimotor programs enable the first sensorimotor program to accomplish the first concept.
19 . The method of claim 9 , wherein the first sensorimotor program is represented by a recurrent neural network.
20 . The method of claim 9 , wherein the first sensorimotor program is trained using natural policy optimization.