Mapping objects using unmanned aerial vehicle data in GPS-denied environments
A method for identifying, locating, and mapping targets of interest using unmanned aerial vehicle (UAV) camera footage in GPS-denied environments. In one embodiment, the method comprises obtaining UAV visual data, passing the UAV visual data through a convolutional neural network (CNN) in order to detect targets of interest based on visual features disposed in the UAV visual data, wherein the detection by the CNN defines reference points and pixel coordinates for the UAV visual data, applying a geometric transformation to known and defined pixel coordinates to obtain real-world orthogonal positions; and projecting the detected targets of interest onto an orthogonal map based on the obtained real-world orthogonal positions, all without GPS data.
1. A method for identifying, locating, and mapping targets of interest using UAV camera footage in GPS-denied environments, the method comprising:
(A) obtaining UAV visual data, wherein the UAV visual data comprises aerial video footage comprising a plurality of frames, wherein one of the plurality of frames comprises at least four manually-selected reference points, wherein the at least four manually-selected reference points comprise known pixel coordinates and real-world orthogonal positions;
(B) passing the UAV visual data through a convolutional neural network (CNN) to detect targets of interest based on visual features disposed in each of the plurality of frames, wherein the detection by the CNN defines at least four new reference points for each of the remaining plurality of frames, and wherein the CNN defines pixel coordinates for the targets of interest for each of the remaining plurality of frames;
(C) applying a geometric transformation to the known and defined pixel coordinates to obtain real-world orthogonal positions; and
(D) projecting the detected targets of interest onto an orthogonal map based on the obtained real-world orthogonal positions.
2. The method of claim 1 , wherein the GPS-denied environments comprise heavily urbanized areas, areas surrounded by dense vegetation, indoor facilities, GPS-jammed locations, unknown territories, areas experiencing severe weather, or any combinations thereof.
3. The method of claim 1 , wherein the UAV visual data comprised geophysical, meteorological, hydrological, and/or climatological events.
4. The method of claim 1 , wherein the UAV visual data is obtained via crowdsourcing.
5. The method of claim 1 , wherein the CNN is trained on relevant datasets to effectively detect the targets of interest and output the pixel coordinates of the targets of interest.
6. The method of claim 1 , wherein the CNN may be YOLO, Mask R-CNN, PSPNet, or RetinaNet.
7. The method of claim 1 , wherein the geometric transformation 1s a homography transformation.
8. The method of claim 1 , wherein the at least four manually-selected reference points are chosen by a UAV operator.
9. The method of claim 1 , wherein the at least four manually-selected reference points are suggested to a UAV operator by the CNN.
10. The method of claim 1 , wherein three of the at least four reference points are non-collinear.
11. The method of claim 1 , further comprising selecting matching keypoints between two consecutive frames of the plurality of frames via SIFT and RANSAC methods.
12. The method of claim 11 , wherein the selecting of matching keypoints is repeated for each of the plurality of frames.
13. The method of claim 11 , wherein the SIFT method comprises identifying matching keypoint candidates between the two consecutive frames.
14. The method of claim 11 , wherein the RANSAC method comprises eliminating outliers from a set of matching keypoint candidates identified by the SIFT method while removing incorrectly matched keypoints.
15. The method of claim 1 , further comprising reconciling accumulated mapping error when a UAV traverses an area or revisits the same points twice via iterative closest point (ICP) method.
16. The method of claim 1 , wherein the method is performed without using data from GPS or inertial measurement sensors of the UAV.