System and method for real-time overlay of map features onto a video feed

1. A method for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“UAV”) to a user interface (“UI”), the method comprising:

obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle;

obtaining supplemental data based on the sequence of video images and the sensor metadata;

correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique;

creating, by a processor, a geographically-referenced scene model based on a coordinate system that is registered to the sequence of video images;

overlaying the supplemental data onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model;

creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and

providing the video stream to a UI to be rendered onto a display, and

wherein the correcting the error in the sensor metadata further comprises:

performing a first registration between a video image in the sequence of video images and a corresponding map data;

determining an anchor frame from the sequence of video images; and

performing a second registration between the video image and the corresponding map data using the anchor frame by minimizing a reprojection error.

2. The method of claim 1 , wherein the supplemental data comprises one or more of: static geo-referenced datasets, dynamical geo-referenced datasets, traffic conditions, elevation data, terrain data, social media information, waypoint data, light detection and ranging (“LIDAR”) data, airspace symbology data, 3D model data, and road maps.

3. The method of claim 1 , wherein the error in the sensor metadata comprises one or more of: missing data, temporal drift, and spatial drift.

4. The method of claim 1 , wherein the correcting the error in the sensor metadata for one image frame is based on another image frame or map data corresponding to a scene that is represented in the one image.

5. The method of claim 1 , further comprising:

constructing a depth map using light detection and ranging (“LIDAR”) or digital elevation maps (“DEM”);

determining that one or more pixels representing overlay objects has low or no visibility for a video frame based the depth map; and

rendering the one or more pixels in a manner to represent that the one or more pixels are occluded.

6. The method of claim 1 , wherein the minimizing the reprojection error is performed using the reconstruction error minimization technique according to:

min( H ) R g(v,m,H) =min( H )Σ j=1 n (| p j v −Hp j m |+|p j m −H −1 p j v |)/ n , where

subject to H to be close to rigid body geometry, and v is a frame of input image, m is cropped map imagery, R g is reconstruction error of feature points, j where j=1, . . . , n, where n is a number of corresponding points.

7. A method for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“UAV”) to a user interface (“UI”), the method comprising:

obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle;

obtaining supplemental data based on the sequence of video images and the sensor metadata;

correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique;

creating, by a processor, a geographically-referenced scene model based on a coordinate system that is registered to the sequence of video images;

overlaying the supplemental data onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model;

creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and

providing the video stream to a UI to be rendered onto a display and, wherein the reconstruction error is minimized using a Lagrange optimization technique to obtain a final homography between map data and the video images.

8. The method of claim 1 , wherein the supplemental data comprises information beyond a field-of-view of the camera of the aerial vehicle.

9. The method of claim 1 , further comprising:

obtaining a series of waypoint data, wherein the series comprises a beginning waypoint, one or more intermediate waypoints, and a destination waypoint;

generating a flight path based on the series of waypoint data; and

outputting the flight path to the UI.

10. The method of claim 1 , further comprising generating the UI that shows the supplemental data overlaid over the video stream and the sensor metadata.

11. A device for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“UAV”) to a user interface (“UI”), the device comprising:

a memory containing instructions; and

at least one processor, operably connected to the memory, which executes the instructions to perform a method for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“UAV”) to a user interface (“UI”), comprising:

obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle;

obtaining supplemental data based on the sequence of video images and the sensor metadata;

correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique;

creating, by a processor, a geographically-referenced scene model based on a coordinate system that is registered to the sequence of video images;

overlaying the supplemental data onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model;

creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying;

providing the video stream to a UI to be rendered onto a display, and

constructing a depth map using light detection and ranging (“LIDAR”) or digital elevation maps (“DEM”);

determining that one or more pixels representing overlay objects has low or no visibility for a video frame based the depth map; and

rendering the one or more pixels in a manner to represent that the one or more pixels are occluded, and

wherein the correcting the error in the sensor metadata further comprises:

performing a first registration between a video image in the sequence of video images and a corresponding map data;

determining an anchor frame from the sequence of video images; and

performing a second registration between the video image and the corresponding map data using the anchor frame by minimizing a reprojection error.

12. The device of claim 11 , wherein the minimizing the reprojection error is performed using the reconstruction error minimization technique according to:

min( H ) R g(v,m,H) =min( H )Σ j=1 n (| p j v −Hp j m |+|p j m −H −1 p j v |)/ n , where

subject to H to be close to rigid body geometry, and v is a frame of input image, m is cropped map imagery, R g is reconstruction error of feature points, j where j=1, . . . , n, where n is a number of corresponding points.

13. The device of claim 12 , wherein the reconstruction error is minimized using a Lagrange optimization technique to obtain a final homography between map data and the video images.

14. A computer non-transitory readable storage medium comprising instructions for causing one or more processors to perform a method, the method for discerning a vehicle at an access control point, the method comprising:

obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle;

obtaining supplemental data based on the sequence of video images and the sensor metadata;

correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique;

creating, by a processor, a geographically-referenced scene model based on a coordinate system that is registered to the sequence of video images;

overlaying the supplemental data onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model;

creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and

providing the video stream to a UI to be rendered onto a display, and

wherein the correcting the error in the sensor metadata further comprises:

performing a first registration between a video image in the sequence of video images and a corresponding map data;

determining an anchor frame from the sequence of video images; and

performing a second registration between the video image and the corresponding map data using the anchor frame by minimizing a reprojection error.

15. The computer readable storage medium of claim 14 , wherein the minimizing the reprojection error is performed using the reconstruction error minimization technique according to:

min( H ) R g(v,m,H) =min( H )Σ j=1 n (| p j v −Hp j m |+|p j m −H −1 p j v |)/ n , where

subject to H to be close to rigid body geometry, and v is a frame of input image, m is cropped map imagery, R g is reconstruction error of feature points, j where j=1, . . . , n, where n is a number of corresponding points.