System and method of efficient, continuous, and safe learning using first principles and constraints
A computer implemented method for self-learning of a control system. The method includes creating an initial knowledge base. The method learns first principles using the knowledge base. The method creates initial control commands derived from the knowledge base. The method generates constraints for the control commands. The method performs constrained reinforcement learning by executing the control commands with the constraints and observing feedback to improve the control commands. The method enriches the knowledge base based on the feedback.
1. A computer implemented method for self-learning of a control system, the method comprising:
creating a knowledge base comprising data obtained from real-world experiments;
learning first principles using the knowledge base, wherein the first principles are foundational principles that cannot be deduced from other principles, and wherein learning the first principles is based on kinematic parameters of motion of an object, estimated structural parameters of the object, and a relationship between the kinematic parameters and control parameters of the object based on the estimated structural parameters of the object;
creating control commands for controlling the object based on the first principles derived from the knowledge base;
generating constraints for the control commands;
performing constrained reinforcement learning by executing the control commands with the constraints,
utilizing feedback from the constrained reinforcement learning to improve the control commands; and
enriching the knowledge base based on the feedback.
2. The method of claim 1 , wherein the object is a vehicle, and wherein the first principles comprises first principals of vehicle motion.
3. The method of claim 1 , wherein creating the control commands derived from the knowledge base comprises receiving a current state, a target state, and a reference to the knowledge base as input parameters.
4. The method of claim 3 , wherein creating the control commands derived from the knowledge base comprises:
breaking a control task down into separate components according to the current state and the target state;
creating a query for each of the separate components of the control task;
retrieving a corresponding query result from the knowledge base for each query to generate a plurality of query results; and
combining the query results to generate a control command for the control task.
5. The method of claim 4 , wherein the query results are combined according to corresponding weights assigned to the separate components of the control task.
6. The method of claim 1 , wherein the constraints for the control commands include hard constraints that specify conditions that cannot be exceeded.
7. The method of claim 1 , wherein the constraints for the control commands include soft constraints that specify preferable conditions.
8. The method of claim 1 , wherein generating constraints for the control commands comprises:
generating a first subset of constraint items based on a state of an operating environment;
generating a second subset of constraint items for a set of filtered moving objects based on a state of a host machine and a target state;
generating a third subset of constraint items for a set of filtered stationary obstacles; and
combining the first subset of constraint items, the second subset of constraint items, and the third subset of constraint items.
9. The method of claim 1 , wherein performing the constrained reinforcement learning to improve the control commands comprises:
decomposing the control commands into multiple categories and dimensions to enable learning of the control commands in each category and dimension separately;
generating control command candidates for a control command based on a current state;
applying the constraints to the control command candidates;
refining the control commands based on past experiences when the current state is a learned state; and
adapting results from learned states to new environments when the current state is not the learned state.
10. The method of claim 1 , wherein enriching the knowledge base based on the feedback comprises refining a manifestation of dynamics and kinematics models of the knowledge base.
11. A system comprising
a memory storage unit comprising instructions; and
one or more processors in communication with the memory storage unit, wherein the one or more processors execute the instructions to:
create a knowledge base comprising data obtained from real-world experiments;
learn first principles using the knowledge base, wherein the first principles are foundational principles that cannot be deduce from other principles, and wherein learning the first principles is based on kinematic parameters of motion of an object, estimated structural parameters of the object, and a relationship between the kinematic parameters and control parameters of the object based on the estimated structural parameters of the object;
create control commands for controlling the object based on the first principles derived from the knowledge base;
generate constraints for the control commands;
perform constrained reinforcement learning by executing the control commands with the constraints,
utilize feedback from the constrained reinforcement learning to improve the control commands; and
enrich the knowledge base based on the feedback.
12. The system of claim 11 , wherein the object is a vehicle, and wherein the first principles comprises first principals of vehicle motion.
13. The system of claim 11 , wherein the one or more processors further execute the instructions to receive a current state, a target state, and a reference to the knowledge base as input parameters.
14. The system of claim 13 , wherein the one or more processors further execute the instructions to:
break a control task down into separate components according to the current state and the target state;
create a query for each of the separate components of the control task;
retrieve a corresponding query result from the knowledge base for each query to generate a plurality of query results; and
combine the query results to generate a control command for the control task.
15. The system of claim 14 , wherein the one or more processors further execute the instructions to combine the query results according to corresponding weights assigned to the separate components of the control task.
16. The system of claim 11 , wherein the constraints for the control commands include hard constraints that specify conditions that cannot be exceeded.
17. The system of claim 11 , wherein the constraints for the control commands include soft constraints that specify preferable conditions.
18. The system of claim 11 , wherein the one or more processors further execute the instructions to:
generate a first subset of constraint items based on a state of an operating environment;
generate a second subset of constraint items for a set of filtered moving objects based on a state of a host machine and a target state;
generate a third subset of constraint items for a set of filtered stationary obstacles; and
combine the first subset of constraint items, the second subset of constraint items, and the third subset of constraint items.
19. The system of claim 11 , wherein the one or more processors further execute the instructions to:
decompose the control commands into multiple categories and dimensions to enable learning of the control commands in each category and dimension separately;
generate control command candidates for a control command based on a current state;
apply the constraints to the control command candidates;
refine the control commands based on past experiences when the current state is a learned state; and
adapt results from learned states to new environments when the current state is not the learned state.
20. A non-transitory computer readable medium storing computer instructions, that when executed by one or more processors, cause the one or more processors to perform the steps of:
creating a knowledge base comprising data obtained from real-world experiments;
learning first principles using the knowledge base, wherein the first principles are foundational principles that cannot be deduce from other principles, and wherein learning the first principles is based on kinematic parameters of motion of an object, estimated structural parameters of the object, and a relationship between the kinematic parameters and control parameters of the object based on the estimated structural parameters of the object;
creating control commands for controlling the object based on the first principles derived from the knowledge base;
generating constraints for the control commands;
performing constrained reinforcement learning by executing the control commands with the constraints,
utilizing feedback from the constrained reinforcement learning to improve the control commands; and
enriching the knowledge base based on the feedback.