IP Library Granted Patent US 9,639,755
Granted Patent B2
US 9,639,755 · App. 14/163,008 · Granted May 2, 2017

Automated compound structure characterization in overhead imagery

Inventor: Lionel Gueguen (Longmont, CO)
Assignee: DigitalGlobe, Inc.
G06K9/0063G06K9/4676
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,639,755
App. No.
14/163,008
Granted
May 2, 2017
Kind
B2
Abstract

A system for automatically characterizing areas of interest (e.g., urban areas, forests, and/or other compound structures) in high resolution overhead imagery through manipulation of a dictionary of visual words. The pixels of an input overhead image are initially clustered into a plurality of hierarchically-arranged connected components of a first hierarchical data structure. Image descriptors (e.g., shape, spectral, etc.) of the connected components are then clustered into a plurality of hierarchically-arranged nodes of a second hierarchical data structure. The nodes at a particular level in the second hierarchical data structure become a dictionary of visual words. Subsets of the visual words may be used to label the cells of a grid over the geographic area as falling into one of a number of areas of interest. Categorization information from the grid may be mapped into a resultant image whereby pixels depict their respective type of area of interest.

Claims (70)

1. A method for use in identifying compound structures in overhead imagery, comprising:

organizing, using a processor, a plurality of pixels of at least one input overhead image into a plurality of components of a first hierarchical data structure, wherein the input image is associated with a geographic area;

deriving, using the processor, at least one image descriptor for each of the components;

constructing, using the processor with at least some of the image descriptors, a second hierarchical data structure that includes a plurality of hierarchically-arranged levels of nodes, wherein the nodes of the second hierarchical data structure are arranged into a plurality of leaf paths that each extend from a root node to one of a plurality of leaf nodes, wherein each of the image descriptors depends from one of the plurality of leaf nodes, wherein at least one node at a particular one of the levels is a respective visual word, and wherein all of the visual words at the particular one of the levels collectively comprise a visual dictionary;

selecting a type of compound structure to be identified in the at least one input overhead image;

overlaying a grid of cells onto the at least one input overhead image, wherein the overlaying includes selecting a width of each cell in the grid to be approximately equal to a spatial extent of the selected compound structure type;

determining, for at least some of the cells of the grid of cells, a frequency of each of one or more of the visual words in the cell;

generating, for each of the at least some of the cells, a vector that includes a plurality of entries, wherein each entry includes a value that represents a respective one of the frequencies; and

for each of the at least some of the cells, using its vector to categorize the cell as representing or not representing the selected compound structure type.

2. The method of claim 1 , wherein the determining step includes:

identifying at least one of the plurality of components within the cell;

ascertaining a smallest distance between the at least one image descriptor of the identified component and each of the at least one image descriptor depending from each of a plurality of the visual words; and

classifying, based on the ascertaining step, the component as a particular one of the visual words associated with the smallest distance, wherein the classifying step contributes to the frequency of the particular one of the visual words in the cell vector.

3. The method of claim 2 , wherein the determining step further includes:

identifying another of the plurality of components within the cell;

ascertaining a smallest distance between the at least one image descriptor of the identified another component and each of the at least one image descriptor depending from each of the plurality of the visual words; and

classifying, based on the ascertaining step, the another component as another particular one of the visual words associated with the smallest distance, wherein the classifying step contributes to the frequency of the another particular one of the visual words in the cell vector.

4. The method of claim 1 , wherein the grid of cells is defined by an overlap parameter that sets forth a displacement between adjacent cells of the grid of cells.

5. The method of claim 4 , wherein the compound structure type is a first compound structure type and the grid of cells is a first grid, wherein the method further includes:

selecting a second type of compound structure to be identified in the at least one input overhead image;

overlaying a second grid of cells onto the at least one input overhead image, wherein the overlaying includes selecting a width of each cell in the second grid to be approximately equal to a spatial extent of the second selected compound structure type;

generating, for at least some of the cells of the second grid, a vector that includes a plurality of entries, wherein each entry includes a value that represents a respective frequency of one of the visual words in the cell; and

for each of the at least some of the cells of the second grid, using its vector to categorize the cell as representing or not representing the second selected compound structure type.

6. The method of claim 5 , wherein one of the entries of the cell vectors of the first grid includes a value that represents a frequency of a first of the visual words in the cell, wherein one of the entries of the cell vectors of the second grid includes a value that represents a frequency of a second of the visual words in the cell, and wherein the first and second visual words are different.

7. The method of claim 1 , wherein the using includes:

identifying a frequency of one or more of the visual words describing the selected compound structure type;

analyzing the entries of the vector in relation to the identified visual word frequencies of the selected compound structure type.

8. A method for use in characterizing areas of interest in overhead imagery, comprising:

organizing, using a processor, a plurality of pixels of at least one input overhead image into a plurality of components of a first hierarchical data structure, wherein the input image is associated with a geographic area;

deriving, using the processor, at least one image descriptor for each of the components;

constructing, using the processor with at least some of the image descriptors, a second hierarchical data structure that includes a plurality of hierarchically-arranged levels of nodes, wherein the nodes of the second hierarchical data structure are arranged into a plurality of leaf paths that each extend from a root node to one of a plurality of leaf nodes, wherein each of the image descriptors depends from one of the plurality of leaf nodes, wherein at least one node at a particular one of the levels is a respective visual word, and wherein all of the visual words at the particular one of the levels collectively comprise a visual dictionary;

determining, for each cell of at least some cells of a grid of cells disposed over the geographic area, a frequency of each of one or more of the visual words in the cell;

generating, for the cell, a vector that includes a plurality of entries, wherein each entry includes a value that represents a respective one of the frequencies

identifying a subset of visual words of the visual dictionary that corresponds to an area of interest in the input overhead image;

identifying a subset of the plurality of components classified by one of the visual words in the subset of visual words; and

identifying a subset of the plurality of cells that each fully contain at least one of the components in the subset of components, wherein the determining and generating steps are performed only with respect to the subset of visual words and the subset of cells.

9. The method of claim 8 , wherein each of at least some of the components in the subset of components is fully contained within at least two of the cells.

10. The method of claim 8 , further including:

assigning respective weighting values to each of the visual words in the subset of visual words, wherein each weighting value represents a degree to which the respective visual word indicates an area of interest;

for each cell of the subset of cells:

obtaining, for each entry in its cell vector, a product of the entry and a corresponding one of the weighting values to obtain a plurality of products;

summing the plurality of products to obtain a sum;

identifying the cell as the area of interest if the sum has one of i) reached or exceeded a threshold and b) not reached the threshold; and

identifying the cells as not the area of interest of the sum has the other of i) reached or exceeded the threshold and b) not reached the threshold.

11. The method of claim 8 , further including:

classifying each of the plurality of components as one of the plurality of visual words;

storing, in a storage device, each of the plurality of components in a data structure that is indexed according to the plurality of visual words;

extracting the components indexed according to the subset of visual words, wherein the extracted components are the subset of components; and

loading the extracted components and subset of visual words into memory, wherein the determining and generating steps are performed with respect to the extracted components and subset of visual words as loaded into memory.

12. The method of claim 1 , further including:

classifying each of the plurality of components as one of the plurality of visual words;

storing, in a storage device, each of the plurality of components in a data structure that is indexed according to the plurality of visual words.

13. The method of claim 1 , wherein the visual dictionary is a first visual dictionary, and wherein all of the visual words at another particular one of the levels collectively comprise a second visual dictionary at least partially different than the first visual dictionary.

14. A method of characterizing areas of interest in overhead imagery, comprising:

organizing, with a processor, an input overhead image associated with a geographic area into a plurality of components of a first hierarchical data structure;

clustering, with the processor, a plurality of image descriptors derived from the plurality of components into a plurality of visual words of a visual dictionary;

selecting a type of compound structure to be identified in the input overhead image;

overlaying a grid of cells onto the input overhead image, wherein the overlaying includes selecting a width of each cell in the grid to be approximately equal to a spatial extent of the selected compound structure type;

categorizing, with the processor, each of the cells as representing or not representing the selected compound structure type based on one or more respective ones of the visual words in the cells.

15. The method of claim 14 , wherein the categorizing includes:

determining, for each of the cells, a frequency of the one or more visual words in the cell;

identifying frequencies of the one or more of the visual words that describe the selected compound structure type;

comparing the frequencies of the one or more visual words in the cell to the identified frequencies of the one or more visual words that describe the selected compound structure type to categorize the cell as representing or not representing the selected compound structure type.

16. The method of claim 15 , wherein the characterizing step utilizes less than all of the entries in the vector.

17. The method of claim 15 , further including:

mapping the plurality of cells into a resultant image that is associated with the geographic area, wherein each cell in the resultant image indicates whether or not it represents the selected compound structure type.

18. The method of claim 15 , wherein the identifying includes:

selecting a sample of a portion of the input overhead image that includes the selected compound structure type;

determining, for the sample, the frequencies of the one or more of the visual words in the sample; and

generating, for the sample, a vector that includes a plurality of entries, wherein each entry includes a value that represents a respective one of the visual word frequencies, wherein the comparing uses the sample vectors and the cell vectors.

Assignments (18)
RELEASE OF SECURITY INTEREST Recorded Mar 3, 2026
From: SIXTH STREET LENDING PARTNERS, ACTING IN ITS CAPACITY AS AGENT
To: AURORA INSIGHT INC.; VANTOR INC. (F/K/A MAXAR INTELLIGENCE INC.); VANTOR SERVICES INC. (F/K/A MAXAR MISSION SOLUTIONS INC.); LANTERIS SPACE LLC (F/K/A MAXAR SPACE LLC); SPATIAL ENERGY, LLC; LANTERIS SPACE ROBOTICS LLC (F/K/A MAXAR SPACE ROBOTICS LLC); VANTOR HOLDINGS INC. (F/K/A MAXAR TECHNOLOGIES HOLDINGS INC.)
Reel/Frame 075021/0624 →
RELEASE (REEL 060389/FRAME 0720) Recorded May 12, 2023
From: ROYAL BANK OF CANADA
To: MAXAR INTELLIGENCE INC.; MAXAR SPACE LLC
Reel/Frame 063633/0431 →
INTELLECTUAL PROPERTY SECURITY AGREEMENT Recorded May 5, 2023
From: MAXAR INTELLIGENCE INC. (F/K/A DIGITALGLOBE, INC.); AURORA INSIGHT INC.; MAXAR MISSION SOLUTIONS INC. ((F/K/A RADIANT MISSION SOLUTIONS INC. (F/K/A THE RADIANT GROUP, INC.)); MAXAR SPACE LLC (F/K/A SPACE SYSTEMS/LORAL, LLC); SPATIAL ENERGY, LLC; MAXAR SPACE ROBOTICS LLC ((F/K/A SSL ROBOTICS LLC) (F/K/A MDA US SYSTEMS LLC)); MAXAR TECHNOLOGIES HOLDINGS INC.
To: SIXTH STREET LENDING PARTNERS, AS ADMINISTRATIVE AGENT
Reel/Frame 063660/0138 →
TERMINATION AND RELEASE OF PATENT SECURITY AGREEMENT - RELEASE OF REEL/FRAME 060389/0782 Recorded May 4, 2023
From: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT
To: MAXAR INTELLIGENCE INC.; MAXAR SPACE LLC
Reel/Frame 063544/0074 →
TERMINATION AND RELEASE OF PATENT SECURITY AGREEMENT - RELEASE OF REEL/FRAME 053866/0412 Recorded May 4, 2023
From: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT
To: MAXAR INTELLIGENCE INC.; MAXAR SPACE LLC
Reel/Frame 063544/0011 →
TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS AND TRADEMARKS - RELEASE OF REEL/FRAME 051258/0465 Recorded May 4, 2023
From: ROYAL BANK OF CANADA, AS AGENT
To: MAXAR INTELLIGENCE INC.; MAXAR SPACE LLC
Reel/Frame 063542/0300 →
TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS AND TRADEMARKS - RELEASE OF REEL/FRAME 044167/0396 Recorded May 4, 2023
From: ROYAL BANK OF CANADA, AS AGENT
To: MAXAR INTELLIGENCE INC.; MAXAR SPACE LLC
Reel/Frame 063543/0001 →
CHANGE OF NAME Recorded Feb 15, 2023
From: DIGITALGLOBE, INC.
To: MAXAR INTELLIGENCE INC.
Reel/Frame 062760/0832 →
RELEASE OF SECURITY INTEREST Recorded Jun 21, 2022
From: WILMINGTON TRUST, NATIONAL ASSOCIATION
To: DIGITALGLOBE, INC.; SPACE SYSTEMS/LORAL, LLC; RADIANT GEOSPATIAL SOLUTIONS LLC
Reel/Frame 060390/0282 →
SECURITY AGREEMENT Recorded Jun 17, 2022
From: MAXAR INTELLIGENCE INC.; MAXAR SPACE LLC
To: WILMINGTON TRUST, NATIONAL ASSOCIATION
Reel/Frame 060389/0782 →
SECURITY AGREEMENT Recorded Jun 16, 2022
From: MAXAR INTELLIGENCE INC.; MAXAR SPACE LLC
To: ROYAL BANK OF CANADA
Reel/Frame 060389/0720 →
PATENT SECURITY AGREEMENT Recorded Sep 23, 2020
From: DIGITALGLOBE, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 053866/0412 →
SECURITY AGREEMENT (NOTES) Recorded Dec 12, 2019
From: DIGITALGLOBE, INC.; RADIANT GEOSPATIAL SOLUTIONS LLC; SPACE SYSTEMS/LORAL, LLC (F/K/A SPACE SYSTEMS/LORAL INC.)
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, - AS NOTES COLLATERAL AGENT
Reel/Frame 051262/0824 →
AMENDED AND RESTATED U.S. PATENT AND TRADEMARK SECURITY AGREEMENT Recorded Dec 11, 2019
From: DIGITALGLOBE, INC.
To: ROYAL BANK OF CANADA, AS COLLATERAL AGENT
Reel/Frame 051258/0465 →
SECURITY INTEREST Recorded Oct 5, 2017
From: DIGITALGLOBE, INC.; MACDONALD, DETTWILER AND ASSOCIATES LTD.; MACDONALD, DETTWILER AND ASSOCIATES CORPORATION; MACDONALD, DETTWILER AND ASSOCIATES INC.; MDA GEOSPATIAL SERVICES INC.; SPACE SYSTEMS/LORAL, LLC; MDA INFORMATION SYSTEMS LLC
To: ROYAL BANK OF CANADA, AS THE COLLATERAL AGENT
Reel/Frame 044167/0396 →
RELEASE OF SECURITY INTEREST IN PATENTS FILED AT R/F 041069/0910 Recorded Oct 5, 2017
From: BARCLAYS BANK PLC
To: DIGITALGLOBE, INC.
Reel/Frame 044363/0524 →
SECURITY INTEREST Recorded Jan 23, 2017
From: DIGITALGLOBE, INC.
To: BARCLAYS BANK PLC, AS COLLATERAL AGENT
Reel/Frame 041069/0910 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 24, 2014
From: GUEGUEN, LIONEL
To: DIGITALGLOBE, INC.
Reel/Frame 032037/0170 →
Continuity (2)
Provisional Application 61911073 · Dec 3, 2013
Related Publication 20150154465A1 · Jun 4, 2015