IP Library Granted Patent US 12,444,040
Granted Patent B2
US 12,444,040 · App. 18/663,733 · Granted Oct 14, 2025

Systems and methods for oilfield drilling operations using computer vision

Inventors: George Michalopulos (Tulsa, OK); Richard Kulavik (Frisco, TX); Todd W. Benson (Dallas, TX)
Assignee: MOTIVE DRILLING TECHNOLOGIES, INC.
G06T7/0004E21B41/0021G06T7/20G06T7/70G06T17/00G06V10/764G06V10/82G06V40/10G06V40/103G08B7/06G08B21/02H04N23/90E21B7/04G06T2207/30196G06V20/52
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,444,040
App. No.
18/663,733
Granted
Oct 14, 2025
Kind
B2
Abstract

Computer vision drilling systems and methods may be used with a drilling rig. The computer vision systems and methods may be used during drilling of a well to monitor the drilling equipment and personnel on the drilling site to provide safer drilling operations. The results from the computer vision drilling system may be used to cause corrective actions to be performed if a safety condition arises. In addition, computer vision systems and methods are provided to automatically monitor the drilling site and drilling operations, such as by tallying pipe in the drill string and by monitoring equipment for anomalous drilling conditions, and automatically taking corrective action as may be needed.

Claims (38)

1. A computer vision system for a drilling rig, the computer vision system comprising:

a processor adapted to receive image data from one or more cameras, wherein each of the one or more cameras is trained on one or more locations of a drilling rig or equipment associated with the drilling rig, thereby defining a drilling rig area and is adapted to provide image data associated with the drilling rig area during drilling operations;

a memory coupled to the processor, wherein the memory comprises instructions executable by the processor to:

receive the image data from the one or more cameras;

determine, responsive to the image data, whether a predetermined safety condition exists or is predicted to occur;

determine a confidence value associated with the determination of the existence or predicted occurrence of the predetermined safety condition;

determine a corrective action responsive to the determination of the existence or predicted occurrence of the predetermined safety condition;

determine, based on the confidence value, a time of implementation for the determined corrective action; and

generate one or more control signals to implement the corrective action at the determined time of implementation.

2. The computer vision system of claim 1 , wherein the instructions further comprise instructions executable by the processor to provide or display the confidence value on a display device.

3. The computer vision system of claim 1 , wherein the predetermined safety condition comprises a person on the rig and one or more drilling rig components occupying a same space within a time period.

4. The computer vision system of claim 1 , wherein the processor is configured to determine the confidence value responsive to a consistency of the image data over a series of time intervals.

5. The computer vision system of claim 1 , wherein the processor is configured to determine the confidence value responsive to a consistency of elements determined by the computer vision system in the image data over a series of observations.

6. The computer vision system of claim 1 , wherein the processor is configured to determine the confidence value responsive to an accuracy and/or resolution of the image data obtained from the one or more cameras.

7. The computer vision system of claim 1 , wherein the processor is configured to determine the confidence value responsive to a comparison of a determined result with one or more results stored in a database.

8. The computer vision system of claim 1 , wherein the time of implementation of the corrective action is longer when the confidence value is low and shorter when the confidence value is high.

9. A non-transitory computer readable storage medium comprising a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions which, when executed by the one or more data processors, cause the one or more data processors to perform actions including:

receiving image data associated with a drilling rig area during drilling operations from one or more cameras;

determining, responsive to the image data, whether a predetermined safety condition exists or is predicted to occur;

determining a confidence value associated with the determination of the existence or predicted occurrence of the predetermined safety condition;

determining a corrective action responsive to the determination of the existence or predicted occurrence of the predetermined safety condition; determine, based on the confidence value, a time of implementation for the determined corrective action based on the confidence value; and

generating one or more control signals to implement the corrective action at the determined time of implementation.

10. The non-transitory computer readable storage medium of claim 9 , wherein the instructions further comprise instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including displaying the confidence value on a display device.

11. The non-transitory computer readable storage medium of claim 9 , wherein the instructions further comprise instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including determining the confidence value responsive to a comparison of a determined result with one or more results stored in a database.

12. The non-transitory computer readable storage medium of claim 9 , wherein the instructions further comprise instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including determining the confidence value responsive to an accuracy and/or resolution of the image data obtained from the one or more cameras.

13. The non-transitory computer readable storage medium of claim 9 , wherein the instructions further comprise instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including determining the confidence value responsive to a consistency of elements determined by a computer vision system in the image data over a series of observations.

14. The non-transitory computer readable storage medium of claim 9 , wherein the instructions further comprise instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including determining the confidence value responsive to a consistency of elements determined by the computer vision system in the image data over a series of time intervals.

15. A method comprising:

receiving image data associated with a drilling rig area during drilling operations from one or more cameras;

determining, responsive to the image data, whether a predetermined safety condition exists or is predicted to occur;

determining a confidence value associated with the determination of the existence or predicted occurrence of the predetermined safety condition;

determining a corrective action responsive to the determination of the existence or predicted occurrence of the predetermined safety condition; determining, based on the confidence value, a time of implementation for the determined corrective action; and

generating one or more control signals to implement the corrective action at the determined time of implementation.

16. The method of claim 15 , further comprising displaying the confidence value on a display device.

17. The method of claim 15 , further comprising determining the confidence value responsive to a comparison of a determined result with one or more results stored in a database.

18. The method of claim 15 , further comprising determining the confidence value responsive to an accuracy and/or resolution of the image data obtained from the one or more cameras.

19. The method of claim 15 , further comprising determining the confidence value responsive to a consistency of elements determined by the computer vision system in the image data over a series of observations.

20. The method of claim 15 , further comprising determining the confidence value responsive to a consistency of elements determined by the computer vision system in the image data over a series of time intervals.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 14, 2024
From: MICHALOPULOS, GEORGE; KULAVIK, RICHARD; BENSON, TODD W.
To: MOTIVE DRILLING TECHNOLOGIES, INC.
Reel/Frame 067410/0375 →
Continuity (5)
Continuation 17664004 · May 18, 2022
Continuation 17182100 · Feb 22, 2021
Continuation 16660250 · Oct 22, 2019
Provisional Application 62748996 · Oct 22, 2018
Related Publication 20240296541A1 · Sep 5, 2024
References Cited (135)
US 3831680A · Edwards et al. · 1974 [cited by applicant]
US 4610005A · Utasi · 1986 [cited by applicant]
US 4658903A · Tateishi · 1987 [cited by applicant]
US 4914591A · Warren et al. · 1990 [cited by applicant]
US 6469734B1 · Nichani et al. · 2002 [cited by applicant]
US 6868920B2 · Hoteit et al. · 2005 [cited by applicant]
US 7802634B2 · Boone · 2010 [cited by applicant]
US 7823655B2 · Boone et al. · 2010 [cited by applicant]
US 7860593B2 · Boone · 2010 [cited by applicant]
US 7874351B2 · Hampton et al. · 2011 [cited by applicant]
US 7933166B2 · Goodman · 2011 [cited by applicant]
US 7938197B2 · Boone et al. · 2011 [cited by applicant]
US 8210283B1 · Benson et al. · 2012 [cited by applicant]
US 8218826B2 · Ciglenec et al. · 2012 [cited by applicant]
US 8233667B2 · Helgason et al. · 2012 [cited by applicant]
US 8360171B2 · Boone et al. · 2013 [cited by applicant]
US 8363101B2 · Gschwendtner et al. · 2013 [cited by applicant]
US 8395661B1 · Olsson et al. · 2013 [cited by applicant]
US 8510081B2 · Boone et al. · 2013 [cited by applicant]
US 8528663B2 · Boone · 2013 [cited by applicant]
US 8547428B1 · Olsson et al. · 2013 [cited by applicant]
US 8622128B2 · Hegeman · 2014 [cited by applicant]
US 8812236B1 · Freeman et al. · 2014 [cited by applicant]
US 8873806B2 · Kiest, Jr. · 2014 [cited by applicant]
US 9041794B1 · Olsson et al. · 2015 [cited by applicant]
US 9134255B1 · Olsson et al. · 2015 [cited by applicant]
US 9279319B2 · Savage · 2016 [cited by applicant]
US 9410877B2 · Maxey et al. · 2016 [cited by applicant]
US 9464492B2 · Austefjord et al. · 2016 [cited by applicant]
US 9494030B2 · Benson et al. · 2016 [cited by applicant]
US 9518817B2 · Baba et al. · 2016 [cited by applicant]
US 9651468B2 · Rowe et al. · 2017 [cited by applicant]
US 9664011B2 · Kruspe et al. · 2017 [cited by applicant]
US 9677882B2 · Kiest, Jr. · 2017 [cited by applicant]
US 9706185B2 · Ellis · 2017 [cited by applicant]
US 9869145B2 · Jones et al. · 2018 [cited by applicant]
US 9912918B2 · Samuel · 2018 [cited by applicant]
US 9915112B2 · Geehan et al. · 2018 [cited by applicant]
US 10227859B2 · Richards et al. · 2019 [cited by applicant]
US 10328503B2 · Osawa et al. · 2019 [cited by applicant]
US 10567735B2 · Ellis et al. · 2020 [cited by applicant]
US 10672154B2 · Gillen et al. · 2020 [cited by applicant]
US 10760403B2 · Ellis et al. · 2020 [cited by applicant]
US 10900341B2 · Brannigan et al. · 2021 [cited by applicant]
US 10957177B2 · Michalopulos et al. · 2021 [cited by applicant]
US 10958877B2 · Torrione et al. · 2021 [cited by applicant]
US 10982950B2 · Torrione · 2021 [cited by applicant]
US 11136876B1 · Coley · 2021 [cited by applicant]
US 11361646B2 · Michalopulos et al. · 2022 [cited by applicant]
US 12014482B2 · Michalopulos et al. · 2024 [cited by applicant]
US 12049822B2 · Michalopulos et al. · 2024 [cited by applicant]
US 20030118230A1 · Song et al. · 2003 [cited by applicant]
US 20040124009A1 · Hoteit et al. · 2004 [cited by applicant]
US 20090110107A1 · Abdallah · 2009 [cited by applicant]
US 20090159294A1 · Abdollahi et al. · 2009 [cited by applicant]
US 20090192731A1 · De Jesus et al. · 2009 [cited by applicant]
US 20110093876A1 · Belz et al. · 2011 [cited by applicant]
US 20130076907A1 · Hobbs et al. · 2013 [cited by applicant]
US 20130236064A1 · Li et al. · 2013 [cited by applicant]
US 20130265409A1 · Tjhang et al. · 2013 [cited by applicant]
US 20130275100A1 · Ellis et al. · 2013 [cited by applicant]
US 20130340998A1 · Flusche · 2013 [cited by applicant]
US 20140002617A1 · Zhang et al. · 2014 [cited by applicant]
US 20140116776A1 · Marx et al. · 2014 [cited by applicant]
US 20140326505A1 · Davis et al. · 2014 [cited by applicant]
US 20140333754A1 · Graves et al. · 2014 [cited by applicant]
US 20150075866A1 · Tjhang · 2015 [cited by applicant]
US 20150138337A1 · Tjhang et al. · 2015 [cited by applicant]
US 20150188984A1 · Mullins · 2015 [cited by applicant]
US 20150218936A1 · Maher et al. · 2015 [cited by applicant]
US 20150235544A1 · Hernandez et al. · 2015 [cited by applicant]
US 20150345261A1 · Kruspe et al. · 2015 [cited by applicant]
US 20150365635A1 · Jose et al. · 2015 [cited by applicant]
US 20160130889A1 · Torrione · 2016 [cited by applicant]
US 20160130917A1 · Torrione · 2016 [cited by applicant]
US 20160134843A1 · Torrione et al. · 2016 [cited by applicant]
US 20160292513A1 · Kozicz et al. · 2016 [cited by applicant]
US 20160305230A1 · Benson et al. · 2016 [cited by applicant]
US 20170089153A1 · Teodorescu · 2017 [cited by applicant]
US 20170138171A1 · Richards et al. · 2017 [cited by applicant]
US 20170145807A1 · Wendorf et al. · 2017 [cited by applicant]
US 20170161885A1 · Parmeshwar et al. · 2017 [cited by applicant]
US 20170167853A1 · Zheng et al. · 2017 [cited by applicant]
US 20170314369A1 · Rosano et al. · 2017 [cited by applicant]
US 20170322086A1 · Luharuka et al. · 2017 [cited by applicant]
US 20180180524A1 · Francois et al. · 2018 [cited by applicant]
US 20190003297A1 · Brannigan et al. · 2019 [cited by applicant]
US 20190024485A1 · Stromme et al. · 2019 [cited by applicant]
US 20190048707A1 · Benson et al. · 2019 [cited by applicant]
US 20190100988A1 · Ellis et al. · 2019 [cited by applicant]
US 20190102612A1 · Takemoto et al. · 2019 [cited by applicant]
US 20190136650A1 · Zheng et al. · 2019 [cited by applicant]
US 20190141294A1 · Thörn · 2019 [cited by examiner]
US 20190206068A1 · Stark et al. · 2019 [cited by applicant]
US 20200126386A1 · Michalopulos et al. · 2020 [cited by applicant]
US 20210002995A1 · Botnan et al. · 2021 [cited by applicant]
US 20210115742A1 · Agarwal et al. · 2021 [cited by applicant]
US 20210285315A1 · Coley · 2021 [cited by applicant]
GB 2522922A · 2015 [cited by applicant]
GB 2532272A · 2016 [cited by applicant]
WO 2016147045A1 · 2016 [cited by applicant]
WO 2017042677A1 · 2017 [cited by applicant]
WO 2017132297A2 · 2017 [cited by applicant]
WO 2017176689A1 · 2017 [cited by applicant]
WO 2018093273A1 · 2018 [cited by applicant]
WO 2018131485A1 · 2018 [cited by applicant]
WO 2018148832A1 · 2018 [cited by applicant]
WO 2018157513A1 · 2018 [cited by applicant]
U.S. Appl. No. 16/660,250, “Non-Final Office Action”, Apr. 28, 2020, 20 pages. [cited by applicant]
U.S. Appl. No. 16/660,250, “Notice of Allowance”, Nov. 20, 2020, 17 pages. [cited by applicant]
U.S. Appl. No. 16/660,250, “Notice of Allowance”, Sep. 16, 2020, 17 pages. [cited by applicant]
U.S. Appl. No. 17/182,100, “Final Office Action”, Oct. 21, 2021, 24 pages. [cited by applicant]
U.S. Appl. No. 17/182,100, “Non-Final Office Action”, Jun. 30, 2021, 22 pages. [cited by applicant]
U.S. Appl. No. 17/182,100, “Notice of Allowance”, Feb. 18, 2022, 7 pages. [cited by applicant]
U.S. Appl. No. 17/444,454, “Corrected Notice of Allowability”, Apr. 3, 2024, 2 pages. [cited by applicant]
U.S. Appl. No. 17/444,454, “Final Office Action”, May 4, 2023, 21 pages. [cited by applicant]
U.S. Appl. No. 17/444,454, “Non-Final Office Action”, Dec. 22, 2022, 17 pages. [cited by applicant]
U.S. Appl. No. 17/444,454, “Non-Final Office Action”, Aug. 24, 2023, 20 pages. [cited by applicant]
U.S. Appl. No. 17/444,454, “Notice of Allowance”, Mar. 27, 2024, 15 pages. [cited by applicant]
U.S. Appl. No. 17/664,004, “Advisory Action”, Jul. 24, 2023, 4 pages. [cited by applicant]
U.S. Appl. No. 17/664,004, “Final Office Action”, May 5, 2023, 17 pages. [cited by applicant]
U.S. Appl. No. 17/664,004, “Non Final Office Action”, Aug. 4, 2023, 15 pages. [cited by applicant]
U.S. Appl. No. 17/664,004, “Non-Final Office Action”, Dec. 19, 2022, 18 pages. [cited by applicant]
U.S. Appl. No. 17/664,004, “Notice of Allowance”, Feb. 20, 2024, 7 pages. [cited by applicant]
Chengtian et al., “Automatic Identification of Oil Sucker Pipe Using Character Coding”, IEEE Youth Conference on Information, Computing and Telecommunication, Sep. 1, 2009, pp. 379-382. [cited by applicant]
EP19874928.5, “Extended European Search Report”, Mar. 29, 2022, 9 pages. [cited by applicant]
EP19874928.5, “Office Action”, Jan. 27, 2023, 4 pages. [cited by applicant]
EP19874928.5, “Office Action”, Jun. 26, 2024, 4 pages. [cited by applicant]
EP19874928.5, “Office Action”, Jan. 26, 2024, 5 pages. [cited by applicant]
Harold et al., “Literature Survey and Background Studies Report (Task V)”, RPSEA, Jul. 15, 2014, pp. 1-90. [cited by applicant]
PCT/US2019/057454, “International Preliminary Report on Patentability”, May 6, 2021, 12 pages. [cited by applicant]
PCT/US2019/057454, “International Search Report and Written Opinion”, Jan. 13, 2020, 13 pages. [cited by applicant]
PCT/US2021/071103, “International Preliminary Report on Patentability”, Feb. 15, 2024, 11 pages. [cited by applicant]
PCT/US2021/071103, “International Search Report and Written Opinion”, Nov. 15, 2021, 12 pages. [cited by applicant]
EP19874928.5, “Intention to Grant”, Apr. 2, 2025, 9 pages. [cited by applicant]