System and method for visual localization
A computer-implemented method for processing a geo-location of an aerial platform, comprises: (a) providing a set of classification stages, which is trained end-to-end to provide a set of cascade decisions, wherein each cascade decision is optimized for a different classification modality and for different spatial resolution; (b) receiving a geo-location query comprising an aerial sensor image, a reference bird's eye view image-map with coordinate system and sensor pose angles measures with respect to the coordinate system; (c) projecting the aerial sensor image onto the coordinate system of the reference image-map using the pose angles measures; (d) matching between the projected sensor image and corresponding image patches in the reference satellite image-map, by applying said set of classification stages in accordance with features extraction from said images and similarity criteria; (e) generating a cascade sequencing, wherein a classification score is aggregated along a cascade path among said set of cascade stages; (f) generating a cascade decision at each stage along the cascade path based on the aggregated classification score and on defined decision logic between successive stages; (g) retrieving a geo-location associated with said aggregated classification decision; and (h) sending the retrieved geo-location to a geo-location requester.
1 . A computer-implemented method for processing a geo-location of an aerial platform, comprising:
providing a set of classification stages, which is trained end-to-end to provide a set of cascade decisions, wherein each cascade decision is optimized for a different classification modality and for different spatial resolution;
receiving a geo-location query comprising an aerial sensor image, a reference bird's eye view image-map with coordinate system and sensor pose angles measures with respect to the coordinate system;
projecting the aerial sensor image onto the coordinate system of the reference bird's eye view image-map using the pose angles measures;
matching between the projected sensor image and corresponding image patches in the reference bird's eye view image-map, by applying said set of classification stages in accordance with features extraction from said images and similarity criteria;
generating a cascade sequencing, wherein a classification score is aggregated along a cascade path among said set of cascade stages;
generating a cascade decision at each stage along the cascade path based on the aggregated classification score and on defined decision logic between successive stages;
retrieving a geo-location associated with said aggregated classification decision; and
sending the retrieved geo-location to a geo-location requester.
2 . The method according to claim 1 , comprising:
matching between the projected sensor image and corresponding image patches in the reference bird's eye view image-map, by applying said set of regression stages in accordance with features extraction from said images and similarity criteria;
defining a set of basic anchor shapes each refers to specific height of the sensor image with respect to reference coordinate system of the reference bird's eye view image-map;
generating classification score, where classification refers to a detection type and to an anchor shape;
generating a cascade regression at each stage along the cascade path, providing an estimation of scaling between shapes of defined anchors and the projection of the sensor-image;
deriving the shape of the projected sensor-image based on classification decision of anchor shape and on regression scaling; and
deriving the sensor height based on the regression scaling.
3 . The method according to claim 1 , wherein the cascade stages are based on a Siamese-RPN architecture with two inputs, namely the projected image of the sensor-image and the reference bird's eye view image-map, and for each input on a multi feature pyramid extraction design, which comprises parallel paths of feature extraction stages, each of which is tuned for a different detection modality.
4 . The method according to claim 1 , wherein feature extraction for reference bird's eye view image-map is executed in advance, and the entire process during sensor-image capture.
5 . The method of claim 1 , wherein modality refers to the amount of extracted features, where small amount refers to semantic, and large to texture.
6 . The method of claim 1 , wherein the extracted features are adapted to be used as natural semantic representations of the image.
7 . The method according to claim 1 , wherein the classification stages are trained end-to-end according to an Optimal Configuration Cascade (OCC), which applies the cascade detection loss for each stage of the cascade.
8 . The method of claim 7 , wherein OCC is applied subject to a constraint on computation complexity.
9 . The method according to claim 1 , wherein the cascade decision is based on a successive classification logic.
10 . The method according to claim 9 , wherein the successive classification logic comprises defined disagreement of decisions with respect to previous stage or non-target decision at a given stage results with non-target decision.
11 . The method according to claim 1 , wherein the cascade path comprises parallel paths and sequential paths.
12 . The method according to claim 1 , wherein each stage refers to different expressivity of function approximation and to different spatial resolution.
13 . The method according to claim 1 , wherein the set of cascade stages is based on a cascade Siamese localization, which comprises:
a) a feature extraction sub-branch that includes a single Backbone network and multiple feature pyramid generators; and
b) an RPN subnet, comprised of a cascade of RPN detectors, which provides an estimate of detected class (cls) and of the bounding box shape (reg).
14 . The method according to claim 13 , wherein a feature pyramid generator is characterized by increased expressivity of function approximation and by increased spatial resolution along the pyramid outputs.
15 . The method of claim 1 , wherein the aerial platform is an Unmanned Aerial Vehicle (UAE).
16 . A system for processing a geo-location of an aerial platform, comprising:
Circuitry adapted to operate a set of classification stages, which is trained end-to-end to provide a set of cascade decisions, wherein each cascade decision is optimized for a different classification modality;
Circuitry adapted to receive a geo-location query comprising an aerial sensor image, a reference bird's eye view image-map with coordinate system and measures of the sensor pose angles;
Circuitry adapted to project the aerial sensor image onto the coordinate system of the reference bird's eye view image-map based on pose angles measures;
Logical means adapted to match between the projected sensor image and corresponding image patches in the reference bird's eye view image-map, by applying said set of classification stages in accordance with features extraction from said images and similarity criteria operating on these features;
Circuitry adapted to generate a cascade sequencing, wherein a classification score is aggregated along a cascade path among said set of cascade stages;
Circuitry adapted to generate a cascade decision at each stage along the cascade path based on the aggregated classification score and on defined detection logic between successive stages;
Circuitry adapted to retrieve a geo-location associated with said aggregated classification decision; and
Communication apparatus configured to send the retrieved geo-location to a geo-location requester.