Control systems using deep reinforcement learning
Data indicative of a plurality of observations of an environment are received at a control system. Machine learning using deep reinforcement learning is applied to determine an action based on the observations. The deep reinforcement learning applies a convolutional neural network or a deep auto encoder to the observations and applies a training set to locate one or more regions having a higher reward. The action is applied to the environment. A reward token indicative of alignment between the action and a desired result is received. A policy parameter of the control system is updated based on the reward token. The updated policy parameter is applied to determine a subsequent action responsive to a subsequent observation.
1. A method comprising:
receiving data indicative of a plurality of observations of an environment of a coldspray application system and/or a component refurbishing system at a control system;
applying an action to the environment in accordance with a control policy of the control system, wherein the action controls a nozzle in the coldspray application and/or the refurbishing process and the observations determine a deposit characteristic on the surface;
receiving a plurality of observations of a simulated version of the environment;
applying machine learning using deep reinforcement learning to determine an action in the simulated version of the environment based on the observations;
applying the action to the simulated version of the environment;
receiving a reward token indicative of alignment between the action and a desired finishing of a surface in the simulated version of the environment;
updating a policy parameter of the control system based on the reward token; and
applying the updated policy parameter to determine a subsequent action responsive to a subsequent observation.
2. The method of claim 1 , wherein the deep reinforcement learning applies a guided policy search.
3. The method of claim 1 , wherein the deep reinforcement learning applies a deep Q-network.
4. The method of claim 1 , wherein the action comprises a classification of one or more defects in a structure, and the method further comprises:
outputting a visual indication of the classification of the one or more of defects in the structure.
5. The method of claim 1 , wherein a training set for machine learning is configured at least in part based on expert knowledge input.
6. The method of claim 1 , wherein the action comprises controlling a speed and an angle of the nozzle.
7. The method of claim 6 , wherein controlling the speed and the angle of the nozzle is based on determining a control sequence such that a final material profile matches a prescribed profile computed using a weighted sum of a time to complete a pass of the nozzle and a penalty on angular rate.
8. A system comprising:
a control system; and
a processing system configured to:
receive data indicative of a plurality of observations of an environment of a coldspray application system and/or a component refurbishing system at the control system;
apply an action to the environment in accordance with a control policy of the control system, wherein the action controls a nozzle in the coldspray application and/or the refurbishing process and the observations determine a deposit characteristic on the surface;
receive a plurality of observations of a simulated version of the environment;
apply machine learning using deep reinforcement learning to determine an action in the simulated version of the environment based on the observations;
apply the action to the simulated version of the environment;
receive a reward token indicative of alignment between the action and a desired finishing of a surface in the simulated version of the environment;
update a policy parameter of the control system based on the reward token; and
apply the updated policy parameter to determine a subsequent action responsive to a subsequent observation.
9. The system of claim 8 , wherein the deep reinforcement learning applies a guided policy search.
10. The system of claim 8 , wherein the deep reinforcement learning applies a deep Q-network.
11. The system of claim 8 , wherein the action comprises a classification of one or more defects in a structure, the system further comprises a display, and a visual indication of the classification of the one or more of defects in the structure is output to the display.
12. The system of claim 8 , wherein a training set for machine learning is configured at least in part based on expert knowledge input.
13. The system of claim 8 , wherein the action comprises controlling a speed and an angle of the nozzle.
14. The system of claim 13 , wherein controlling the speed and the angle of the nozzle is based on determining a control sequence such that a final material profile matches a prescribed profile computed using a weighted sum of a time to complete a pass of the nozzle and a penalty on angular rate.