Granted Patent B2

US 12,675,966 · App. 17/872,409 · Granted Jul 7, 2026

Inventors: Beau Travis Rollins (Oklahoma City, OK); Amos James Hall (Sparks, OK); Jared Lee Markes (Edmond, OK); Steven Boyd Jackson, II (Dallas, TX)

Assignee: DEVON ENERGY CORPORATION

G06V10/25G06V10/751G06V10/771

Abstract

Claims (154)

1 . A method of analyzing an image to detect at least one object of interest (OOI) therein, wherein the image depicts a scene, the method comprising:

detecting a horizon in the image, wherein the detected horizon defines a boundary of a candidate area;

detecting the at least one OOI in the candidate area by a plurality of image processing techniques, wherein each of the plurality of the image processing techniques is configured to yield resultant candidate OOI, such that an area of overlap of resultant candidate OOI from the plurality of image processing techniques corresponds to detection of the at least one OOI in the area of overlap;

wherein detecting the horizon in the image comprises analyzing the image via at least two image processing techniques to yield a detected candidate horizon per image processing technique, and selecting one of the detected candidate horizons, wherein the selected detected horizon corresponds to a best representation of the horizon in the image and the selection is based, in part, on a plurality of comparisons of each of the detected candidate horizons, wherein a first image processing technique for analyzing the image to detect a first detected candidate horizon is applied to a grayscale format of the image, and wherein a second image processing technique for analyzing the image to detect a second detected candidate horizon is applied to a red (R), green (G), blue (B), format of the image; and wherein the first image processing technique comprises:

cropping the grayscale format of the image to at least an upper 60% portion;

blurring the cropped grayscale format image by a Gaussian blur technique;

executing an edge-detection algorithm via a Sobel operation;

inverting pixel values of the executed Sobel operation edge-detection algorithm;

capturing a maximum pixel value of the inverted pixel values;

thresholding the inverted cropped image on at least 20% of the maximum pixel value;

capturing a white pixel percentage value of the thresholded image and creating contours around white pixels of the thresholded image;

calculating a width of each created contour;

locating one or more widest contours that meet predefined criteria to isolate, wherein the predefined criteria includes: a single contour having a width at least 40% of the image width, or two widest contours having a combined width of at least 40% of the image width;

isolating the one or more widest contours;

converting the one or more widest contours to points; and

rendering a fitted line through the points on the RGB format of the image, wherein the fitted line is representative of a resultant detected horizon of the first image processing technique.

2 . The method of claim 1 , wherein the second image processing technique comprises:

cropping the RGB format of the image to at least an upper 60% portion;

blurring the cropped RGB format image by a Gaussian blur technique;

executing a Canny edge-detection algorithm on the blurred cropped RGB format image;

inverting pixel values of the executed Canny edge-detection algorithm;

reducing image noise using an open morphological transformation on the inverted pixel value image;

masking HUD from the reduced image;

blurring the masked HUD image by a Gaussian blur technique;

executing an edge-detection algorithm via a Sobel operation on the blurred masked HUD image;

inverting pixel values of the executed Sobel operation edge-detection algorithm;

capturing a maximum pixel value of the inverted pixel values;

thresholding the inverted cropped image on at least 20% of the maximum pixel value;

capturing a white pixel percentage value of the thresholded image and creating contours around white pixels of the thresholded image;

calculating a width of each created contour;

locating one or more widest contours that meet predefined criteria to isolate, wherein the predefined criteria includes: a single contour having a width at least 40% of the image width, or two widest contours having a combined width of at least 40% of the image width;

isolating the one or more widest contours;

converting the one or more widest contours to points; and

rendering a fitted line through the points on the RGB format of the image, wherein the fitted line is representative of the resultant detected horizon of the second image processing technique.

3 . The method of claim 2 , wherein the plurality of comparisons for selecting the candidate horizon includes:

assessing of a percentage value of white pixels and a number of contours in a Sobel threshold for each of the at least two image processing techniques;

determining a validity of the detected candidate horizon for each of the at least two image processing techniques;

comparing similarities of the detected candidate horizon for each of the at least two image processing techniques;

comparing grayscale pixel intensity standard deviations above and below each of the candidate horizons.

4 . The method of claim 1 , wherein the at least two image processing techniques are performed simultaneously with each other or sequentially to each other.

5 . The method of claim 1 , wherein the step of detecting the at least one OOI in the candidate area includes:

capturing one or more candidate OOI in the candidate area;

applying filtering criteria to the captured one or more candidate OOI; and

classifying any filtered one or more candidate OOI to yield resultant candidate OOI.

6 . The method of claim 1 , wherein the at least two image processing techniques configured to yield resultant candidate OOI include:

inputting color statistics of contours of the candidate area into a support vector machine (SVM) model to detect and classify candidate OOI using the SVM model and thereby yield resultant candidate OOI by a first image processing technique;

inputting color statistics of contours of the candidate area into an XGBoost model to detect and classify candidate OOI using the XGBoost model and thereby yield resultant candidate OOI by a second image processing technique; and

inputting the image into a convolutional neural network (CNN) to detect and classify candidate OOI using the CNN and thereby yield resultant candidate OOI by a third image processing technique.

7 . The method of claim 1 , further comprising:

generating an annotated image of the scene, wherein the generated image includes an information graphic associated with the at least one detected OOI.

8 . The method of claim 1 , wherein the at least one OOI is smoke.

9 . The method of claim 1 , wherein the detected candidate horizon is a lower boundary of the candidate area.

10 . A computing system configured to analyze an image to detect at least one object of interest (OOI) therein, wherein the image depicts a scene, the computing system comprising:

at least one processor;

at least one non-transitory computer readable storage media operably coupled to the at least one processor; and

program instructions stored on the at least one non-transitory computer readable storage media for execution by the at least one processor that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

detecting a horizon in the image, wherein the detected horizon defines a boundary of a candidate area; and

detecting at least one OOI in the candidate area by a plurality of image processing techniques, wherein each of the plurality of image processing techniques is configured to yield resultant candidate OOI, such that an area of overlap of resultant candidate OOI from the plurality of image processing techniques corresponds to detection of the at least one OOI in the area of overlap, wherein detecting the horizon in the image comprises analyzing the image via at least two image processing techniques to yield a detected candidate horizon per image processing technique, and selecting from one of the detected candidate horizons, wherein the selected detected horizon corresponds to a best representation of the horizon in the image and the selection is based, in part, on a plurality of comparisons of each of the detected candidate horizons, wherein a first image processing technique for analyzing the image to detect a first detected candidate horizon is applied to a grayscale format of the image, and wherein a second image processing technique for analyzing the image to detect a second detected candidate horizon is applied to a red (R), green (G), blue (B) format of the image, wherein the first image processing technique comprises:

cropping the grayscale format of the image to at least an upper 60% portion;

blurring the cropped grayscale format image by a Gaussian blur technique;

executing an edge-detection algorithm via a Sobel operation;

inverting pixel values of the executed Sobel operation edge-detection algorithm;

capturing a maximum pixel value of the inverted pixel values;

thresholding the inverted cropped image on at least 20% of the maximum pixel value;

capturing a white pixel percentage value of the thresholded image and creating contours around white pixels of the thresholded image;

calculating a width of each created contour;

locating one or more widest contours that meet predefined criteria to isolate, wherein the predefined criteria includes: a single contour having a width at least 40% of the image width, or two widest contours having a combined width of at least 40% of the image width;

isolating the one or more widest contours;

converting the one or more widest contours to points; and

rendering a fitted line through the points on the RGB format of the image, wherein the fitted line is representative of a resultant detected horizon of the first image processing technique.

11 . The computing system of claim 10 , wherein the second image processing technique comprises:

cropping the RGB format of the image to at least an upper 60% portion;

blurring the cropped RGB format image by a Gaussian blur technique;

executing a Canny edge-detection algorithm on the blurred cropped RGB format image;

inverting pixel values of the executed Canny edge-detection algorithm;

reducing image noise using an open morphological transformation on the inverted pixel value image;

masking HUD from the reduced image;

blurring the masked HUD image by a Gaussian blur technique;

executing an edge-detection algorithm via a Sobel operation on the blurred masked HUD image;

inverting pixel values of the executed Sobel operation edge-detection algorithm;

capturing a maximum pixel value of the inverted pixel values;

thresholding the inverted cropped image on at least 20% of the maximum pixel value;

capturing a white pixel percentage value of the thresholded image and creating contours around white pixels of the thresholded image;

calculating a width of each created contour;

locating one or more widest contours that meet predefined criteria to isolate, wherein the predefined criteria includes: a single contour having a width at least 40% of the image width, or two widest contours having a combined width of at least 40% of the image width;

isolating the one or more widest contours;

converting the one or more widest contours to points; and

rendering a fitted line through the points on the RGB format of the image, wherein the fitted line is representative of the resultant detected horizon of the second image processing technique.

12 . The computing system of claim 11 , wherein the plurality of comparisons for selecting the candidate horizon includes:

assessing of a percentage value of white pixels and a number of contours in a Sobel threshold for each of the at least two image processing techniques;

determining a validity of the detected candidate horizon for each of the at least two image processing techniques;

comparing similarities of the detected candidate horizon for each of the at least two image processing techniques; and

comparing grayscale pixel intensity standard deviations above and below each of the candidate horizons.

13 . The computing system of claim 10 , wherein the at least two image processing techniques are performed simultaneously with each other or sequentially to each other.

14 . The computing system of claim 10 , wherein the operation of detecting the at least one OOI in the candidate area includes:

capturing one or more candidate OOI in the candidate area;

applying filtering criteria to the captured one or more candidate OOI; and

classifying any filtered one or more candidate OOI to yield resultant candidate OOI.

15 . The computing system of claim 10 , wherein the at least two image processing techniques configured to yield resultant candidate OOI include:

inputting color statistics of contours of the candidate area into a support vector machine (SVM) model to detect and classify candidate OOI using the SVM model and thereby yield resultant candidate OOI by a first image processing technique;

inputting color statistics of contours of the candidate area into an XGBoost model to detect and classify candidate OOI using the XGBoost model and thereby yield resultant candidate OOI by a second image processing technique; and

inputting the image into a convolutional neural network (CNN) to detect and classify candidate OOI using the CNN and thereby yield resultant candidate OOI by a third image processing technique.

16 . The computing system of claim 10 , wherein the operations further comprise generating an annotated image of the scene, wherein the generated image includes an information graphic associated with the at least one detected OOI.

17 . The computing system of claim 10 , wherein the at least one OOI is smoke.

18 . The computing system of claim 10 , wherein the detected candidate horizon is a lower boundary of the candidate area.

19 . A computer program product including one or more non-transitory computer readable storage mediums having instructions encoded thereon that when executed by at least one processor, cause a process to be carried out for analyzing an image to detect at least one object of interest (OOI) therein, wherein the image depicts a scene, the process comprising:

detecting a horizon in the image, wherein the detected horizon defines a boundary of a candidate area; and detecting the at least one OOI in the candidate area by a plurality of image processing techniques, wherein each of the plurality of image processing techniques is configured to yield resultant candidate OOI, such that an area of overlap of resultant candidate OOI from the plurality of image processing techniques corresponds to detection of the at least one OOI in the area of overlap, wherein detecting a horizon in the image comprises analyzing the image via at least two image processing techniques to yield a detected candidate horizon per image processing technique, and selecting one of the detected candidate horizons, wherein the selected detected horizon corresponds to a best representation of the horizon in the image and the selection is based, in part, on a plurality of comparisons of each of the detected candidate horizons, wherein a first image processing technique for analyzing the image to detect a first detected candidate horizon is applied to a grayscale format of the image, and wherein a second image processing technique for analyzing the image to detect a second detected candidate horizon is applied to a red (R) green (G), blue (B) format of the image, and wherein the first image processing technique comprises:

cropping the grayscale format of the image to at least an upper 60% portion;

blurring the cropped grayscale format image by a Gaussian blur technique;

executing an edge-detection algorithm via a Sobel operation;

inverting pixel values of the executed Sobel operation edge-detection algorithm;

capturing a maximum pixel value of the inverted pixel values;

thresholding the inverted cropped image on at least 20% of the maximum pixel value;

capturing a white pixel percentage value of the thresholded image and creating contours around white pixels of the thresholded image;

calculating a width of each created contour;

locating one or more widest contours that meet predefined criteria to isolate, wherein the predefined criteria includes: a single contour having a width at least 40% of the image width, or two widest contours having a combined width of at least 40% of the image width;

isolating the one or more widest contours;

converting the one or more widest contours to points; and

rendering a fitted line through the points on the RGB format of the image, wherein the fitted line is representative of a resultant detected horizon of the first image processing technique.

20 . The computer program product of claim 19 , wherein the second image processing technique comprises:

cropping the RGB format of the image to at least an upper 60% portion;

blurring the cropped RGB format image by a Gaussian blur technique;

executing a Canny edge-detection algorithm on the blurred cropped RGB format image;

inverting pixel values of the executed Canny edge-detection algorithm;

reducing image noise using an open morphological transformation on the inverted pixel value image;

masking HUD from the reduced image;

blurring the masked HUD image by a Gaussian blur technique;

executing an edge-detection algorithm via a Sobel operation on the blurred masked HUD image;

inverting pixel values of the executed Sobel operation edge-detection algorithm;

capturing a maximum pixel value of the inverted pixel values;

thresholding the inverted cropped image on at least 20% of the maximum pixel value;

capturing a white pixel percentage value of the thresholded image and creating contours around white pixels of the thresholded image;

calculating a width of each created contour;

locating one or more widest contours that meet predefined criteria to isolate, wherein the predefined criteria includes: a single contour having a width at least 40% of the image width, or two widest contours having a combined width of at least 40% of the image width;

isolating the one or more widest contours;

converting the one or more widest contours to points; and

rendering a fitted line through the points on the RGB format of the image, wherein the fitted line is representative of the resultant detected horizon of the second image processing technique.

21 . The computer program product of claim 20 , wherein the plurality of comparisons for selecting the candidate horizon includes:

assessing of a percentage value of white pixels and a number of contours in a Sobel threshold for each of the at least two image processing techniques;

determining a validity of the detected candidate horizon for each of the at least two image processing techniques;

comparing similarities of the detected candidate horizon for each of the at least two image processing techniques;

comparing grayscale pixel intensity standard deviations above and below each of the candidate horizons.

22 . The computer program product of claim 19 , wherein the at least two image processing techniques are performed simultaneously with each other or sequentially to each other.

23 . The computer program product of claim 19 , wherein detecting the at least one OOI in the candidate area includes:

capturing one or more candidate OOI in the candidate area;

applying filtering criteria to the captured one or more candidate OOI; and

classifying any filtered one or more candidate OOI to yield resultant candidate OOI.

24 . The computer program product of claim 19 , wherein the at least two image processing techniques configured to yield resultant candidate OOI include:

inputting color statistics of contours of the candidate area into a support vector machine (SVM) model to detect and classify candidate OOI using the SVM model and thereby yield resultant candidate OOI by a first image processing technique;

inputting color statistics of contours of the candidate area into an XGBoost model to detect and classify candidate OOI using the XGBoost model and thereby yield resultant candidate OOI by a second image processing technique; and

inputting the image into a convolutional neural network (CNN) to detect and classify candidate OOI using the CNN and thereby yield resultant candidate OOI by a third image processing technique.

25 . The computer program product of claim 19 , wherein the process further comprises generating an annotated image of the scene, wherein the generated image includes an information graphic associated with the at least one detected OOI.

26 . The computer program product of claim 19 , wherein the at least one OOI is smoke.

27 . The computer program product of claim 19 , wherein the detected candidate horizon is a lower boundary of the candidate area.

Assignments (1)

Continuity (2)

References Cited (32)