IP Library › Granted Patent US 10,217,236
Granted Patent B2
US 10,217,236 · App. 15/952,112 · Granted Feb 26, 2019

Remote determination of containers in geographical region

Inventors: Adam Wiggen Kraft (Mountain View, CA); Boris Aleksandrovich Babenko (Mountain View, CA); Alexander Bogdanov Avtanski (San Jose, CA); Daniel Michael Sammons (Fremont, CA); Jasper Lin (Fremont, CA); Jason D. Lohn (Los Gatos, CA)
Assignee: ORBITAL INSIGHT, INC.
G06T7/62G06K9/00637G06K9/2054G06K9/3241G06K9/4609G06K9/6202G06K9/6247G06K9/6256G06K9/6277G06T7/13G06T7/194G06T7/73G06T2207/10032G06T2207/20081G06T2207/20221G06T2207/30184G06T2207/30192
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,217,236
App. No.
15/952,112
Granted
Feb 26, 2019
Kind
B2
Abstract

Disclosed is a method and system for processing images from an aerial imaging device. The method includes receiving a first image of a geographical area having a first resolution. The method transmits the first image to a machine learning model to identify an area of interest containing an object of interest. The method receives a second image of the geographical area having a second resolution higher than the first resolution. The method transmits the second image to the machine learning model to determine a likelihood that the area of interest contains the object of interest. The method trains the machine learning model to filter out features corresponding to the area of interest in images having the first resolution if the likelihood is below a threshold. The method transmits a visual representation of the object of interest to a user device if the likelihood exceeds the threshold.

Claims (66)

1. A method for processing images from an aerial imaging device, the method comprising:

detecting by a machine learning model, using an image of an object of interest, an amount of cloud cover present in the image of the object of interest, the machine learning model configured to differentiate features corresponding to the cloud cover from background features and features corresponding to the object of interest;

responsive to the amount of cloud cover present in the image of the object of interest being less than a cloud detection threshold, extracting, from the image of the object of interest, parameters describing at least an azimuth angle of the aerial imaging device;

generating, from the parameters, idealized images of the object of interest, wherein each idealized image corresponds to a distinct filled volume of the object of interest;

matching the image of the object of interest to each idealized image to determine a filled volume of the object of interest; and

transmitting information corresponding to the determined filled volume of the object of interest to a user device.

2. The method of claim 1 , further comprising:

creating training sets comprising labeled images of a geographical area, the labeled images of the geographical area comprising distinct amounts of cloud cover; and

configuring the machine learning model, based on the training sets, to detect an amount of cloud cover present in an unlabeled image by differentiating features in the unlabeled image corresponding to the cloud cover present in the unlabeled image from background features present in the unlabeled image.

3. The method of claim 1 , further comprising extracting a feature vector from the image of the object of interest, the feature vector comprising features representing attributes of pixels in the image of the object of interest, wherein the detecting of the amount of cloud cover present in the image of the object of interest comprises transmitting the feature vector to the machine learning model.

4. A method for processing images from an aerial imaging device, the method comprising:

detecting by a machine learning model, using an image of an object of interest, an amount of cloud cover present in the image of the object of interest;

responsive to the amount of cloud cover present in the image of the object of interest being less than a threshold, extracting, from the image of the object of interest, parameters describing at least an azimuth angle of the aerial imaging device;

determining, from the parameters, a filled volume of the object of interest; and

transmitting information corresponding to the determined filled volume of the object of interest to a user device.

5. The method of claim 4 , further comprising:

creating training sets comprising labeled images of a geographical area, the labeled images of the geographical area comprising distinct amounts of cloud cover; and

configuring the machine learning model, based on the training sets, to detect an amount of cloud cover present in an unlabeled image by differentiating features in the unlabeled image corresponding to the cloud cover present in the unlabeled image from background features present in the unlabeled image.

6. The method of claim 4 , further comprising extracting a feature vector from the image of the object of interest, the feature vector comprising features representing attributes of pixels in the image of the object of interest, wherein the detecting of the amount of cloud cover present in the image of the object of interest comprises transmitting the feature vector to the machine learning model.

7. The method of claim 4 , wherein the parameters further describe one or more of:

a time of capture of the received image;

an elevation angle of the sun; and

an azimuth angle of the sun.

8. The method of claim 4 , wherein the determining of the filled volume of the object of interest comprises:

generating, from the parameters, idealized images of the object of interest, wherein each idealized image corresponds to a distinct filled volume of the object of interest; and

matching the image of the object of interest to each idealized image to determine the filled volume of the object of interest.

9. The method of claim 8 , further comprising rescaling each idealized image by:

setting pixels corresponding to shadows and inner surfaces of the object of interest in the idealized image to negative values; and

setting pixels corresponding to a roof of the object of interest in the idealized image to positive values.

10. The method of claim 8 , wherein the generating of each idealized image comprises:

generating a first circle corresponding to a top rim of the object of interest using the parameters;

generating a second circle corresponding to an arc of a shadow on an inner surface of the object of interest using the parameters;

generating a third circle corresponding to a roof of the object of interest using the parameters; and

synthesizing the idealized image by performing a convolution on the three circles.

11. The method of claim 10 , wherein the performing of the convolution comprises performing unions and intersections between the first circle corresponding to the top rim, the second circle corresponding to the arc of the shadow, and the third circle corresponding to the roof.

12. The method of claim 8 , wherein the matching of the image of the object of interest to each idealized image comprises:

superimposing the idealized image on different locations of the image of the object of interest;

performing a dot product between pixels of the image of the object of interest and pixels of the idealized image at each of the different locations to determine a maximum value of the dot product at each of the different locations.

13. The method of claim 8 , wherein the matching of the image of the object of interest to each idealized image comprises:

performing edge analysis on the idealized image to obtain an outline of a top rim of the object of interest in the idealized image; and

performing a dot product between pixels of an image gradient of the image of the object of interest and pixels of the outline of the top rim.

14. The method of claim 8 , wherein the matching of the image of the object of interest to each idealized image comprises:

performing edge analysis on the idealized image to obtain an outline of a shadow on an inner surface of the object of interest in the idealized image; and

performing a dot product between pixels of an image gradient of the image of the object of interest and pixels of the outline of the shadow.

15. The method of claim 8 , wherein the matching of the image of the object of interest to each idealized image comprises:

performing a dot product between pixels of the image of the object of interest and pixels of the idealized image to obtain a first response map;

performing a dot product between pixels of an image gradient of the image of the object of interest and pixels of an outline of a top rim of the object of interest in the idealized image to obtain a second response map;

performing a dot product between pixels of the image gradient and pixels of an outline of a shadow on an inner surface of the object of interest in the idealized image to obtain a third response map; and

summing up the three response maps.

16. A non-transitory computer-readable storage medium comprising instructions executable by a processor, the instructions when executed by the processor cause the processor to:

detect by a machine learning model, using an image of an object of interest, an amount of cloud cover present in the image of the object of interest;

responsive to the amount of cloud cover present in the image of the object of interest being less than a cloud detection threshold, extract, from the image of the object of interest, parameters describing at least an azimuth angle of the aerial imaging device;

determine, from the parameters, a filled volume of the object of interest; and

transmit information corresponding to the determined filled volume of the object of interest to a user device.

17. The non-transitory computer-readable storage medium of claim 16 , the instructions further causing the processor to:

create training sets comprising labeled images of a geographical area, the labeled images of the geographical area comprising distinct amounts of cloud cover; and

configure the machine learning model, based on the training sets, to detect an amount of cloud cover present in an unlabeled image by differentiating features in the unlabeled image corresponding to the cloud cover present in the unlabeled image from background features present in the unlabeled image.

18. The non-transitory computer-readable storage medium of claim 16 , the instructions further causing the processor to extract a feature vector from the image of the object of interest, the feature vector comprising features representing attributes of pixels in the image of the object of interest, wherein the detecting of the amount of cloud cover present in the image of the object of interest comprises transmitting the feature vector to the machine learning model.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the determining of the filled volume of the object of interest comprises:

generating, from the parameters, idealized images of the object of interest, wherein each idealized image corresponds to a distinct filled volume of the object of interest; and

matching the image of the object of interest to each idealized image to determine the filled volume of the object of interest.

20. The method of claim 19 , wherein the generating of each idealized image comprises:

generating a first circle corresponding to a top rim of the object of interest using the parameters;

generating a second circle corresponding to an arc of a shadow on an inner surface of the object of interest using the parameters;

generating a third circle corresponding to a roof of the object of interest using the parameters; and

synthesizing the idealized image by performing a convolution on the three circles.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 14, 2018
From: KRAFT, ADAM WIGGEN; BABENKO, BORIS ALEKSANDROVICH; AVTANSKI, ALEXANDER BOGDANOV; SAMMONS, DANIEL MICHAEL; LIN, JASPER; LOHN, JASON D.
To: ORBITAL INSIGHT, INC.
Reel/Frame 047495/0867 →
Continuity (3)
Continuation In Part 15470543 · Mar 27, 2017
Provisional Application 62320387 · Apr 8, 2016
Related Publication 20180232900A1 · Aug 16, 2018
Cited By (1)
US 12,205,031