Automated rate of penetration optimization for drilling
Systems and methods for controlling drilling operations are provided. A controller for a drilling system may provide drilling parameters such as weight-on-bit and rotation rate parameters to the drilling system, based on a machine-learned reward policy and a model-based prediction. The machine-learned reward policy may be generated during drilling operations and used to modify recommended values from the model-based prediction for subsequent drilling operations to achieve a desired rate-of-penetration.
1. A method of controlling a drill string having a drill bit, the method comprising:
obtaining a current rate-of-penetration for the drill bit, while the drill bit is rotated at a current rotation rate and with a current weight-on-bit in a formation;
determining, based on the current rate-of-penetration and a model, a recommended rotation rate and a recommended weight-on-bit for a recommended rate-of-penetration;
determining, with a machine-learned reward policy for the drill string, an updated rotation rate and an updated weight-on-bit to achieve the recommended rate-of-penetration, based on the recommended rotation rate and the recommended weight-on-bit; and
modifying the current rotation rate and the current weight-on-bit of the drill bit to the updated rotation rate and the updated weight-on-bit.
2. The method of claim 1 , wherein the updated rotation rate is different from the recommended rotation rate.
3. The method of claim 1 , wherein the updated weight-on-bit is different from the recommended weight-on-bit.
4. The method of claim 3 , wherein the updated rotation rate is different from the recommended rotation rate.
5. The method of claim 1 , wherein the machine-learned reward policy comprises at least one of a Q-table or a machine learning model generated with a reinforcement learning process for the drill string.
6. The method of claim 5 , wherein the reinforcement learning process comprises comparisons of previous actual rates-of-penetration with previous recommended rates- of-penetration evaluated prior to obtaining the current rate-of-penetration.
7. The method of claim 6 , wherein each of the comparisons comprises an evaluation of a reward function.
8. The method of claim 1 , wherein the model comprises a neural network model and wherein determining the recommended rotation rate and the recommended weight-on-bit comprises determining the recommended rotation rate and the recommended weight-on-bit based on the neural network model and a Bayesian optimization operation.
9. A control system for drilling system having a drill sting operatively coupled to a drill bit, the control system comprising:
at least one processor; and
memory comprising instructions that, when executed cause the at least one processor to:
obtain a current rate-of-penetration for the drill bit, while the drill bit is rotated at a current rotation rate and with a current weight-on-bit in a formation;
determine, based on the current rate-of-penetration and a model, a recommended rotation rate and a recommended weight-on-bit for a recommended rate-of-penetration;
determine an updated rotation rate and an updated weight-on-bit to achieve the recommended rate-of-penetration, based on a machine-learned reward policy, the recommended rotation rate and the recommended weight-on-bit; and
modify the current rotation rate and the current weight-on-bit of the drill bit to the updated rotation rate and the updated weight-on-bit.
10. The control system of claim 9 , wherein the updated weight-on-bit is different from the recommended weight-on-bit.
11. The control system of claim 10 , wherein the updated rotation rate is different from the recommended rotation rate.
12. The control system of claim 9 , wherein the machine- learned reward policy comprises at least one of a Q-table or a machine learning model generated with a reinforcement learning process for the drill string.
13. The control system of claim 12 , wherein the reinforcement learning process comprises comparisons of previous actual rates-of-penetration with previous recommended rates-of-penetration evaluated prior to obtaining the current rate-of-penetration.
14. The control system of claim 13 , wherein each of the comparisons comprises an evaluation of a reward function.
15. The control system of claim 9 , wherein the model comprises a neural network model and wherein the at least one processor is configured to determine the recommended rotation rate and the recommended weight-on-bit by determining the recommended rotation rate and the recommended weight-on-bit based on the neural network model and a Bayesian optimization operation.
16. A method of configuring a robotic controller for a drill string and associated drill bit of a drilling system, the method comprising:
obtaining, with a prediction engine, a current rate-of-penetration for the drill bit, while the drill bit is rotated at a current rotation rate and with a current weight-on-bit in a formation;
determining, with the prediction engine and based on the current rate-of-penetration and a model, a recommended rotation rate, a recommended weight-on-bit, and a recommended rate-of-penetration;
providing the recommended rotation rate, the recommended weight-on-bit, and the recommended rate-of-penetration from the prediction engine to the robotic controller; and
generating a machine-learned reward policy for future control of the drill string, by performing a reinforcement learning process with the robotic controller using the recommended rotation rate, the recommended weight-on-bit, and the recommended rate-of-penetration.
17. The method of claim 16 , wherein performing the reinforcement learning process comprises:
applying a plurality of modifications to the recommended rotation rate and the recommended weight-on-bit; and
for each modification:
obtaining a new current rate-of-penetration; and
evaluating a reward function that compares the new current rate-of-penetration to the recommended rate-of-penetration.
18. The method of claim 17 , wherein the machine-learned reward policy comprises a reward table, and wherein performing the reinforcement learning process further comprises populating the reward table based on each evaluation of the reward function.
19. The method of claim 17 , further comprising determining each of the plurality of modifications using an epsilon greedy policy of the reinforcement learning process.
20. The method of claim 16 , further comprising:
obtaining, with the prediction engine, a new current rate-of-penetration for the drill bit, while the drill bit is rotated at a new current rotation rate and with a new current weight-on-bit in a formation;
determining, with the prediction engine and based on the new current rate-of-penetration and the model, a new recommended rotation rate, a new recommended weight-on-bit, and a new recommended rate-of-penetration;
providing the new recommended rotation rate, the new recommended weight-on-bit, and the new recommended rate-of-penetration from the prediction engine to the robotic controller; and
updating the machine-learned reward policy for future control of the drill string, by performing the reinforcement learning process with the robotic controller using the new recommended rotation rate, the new recommended weight-on-bit, and the new recommended rate-of-penetration.