Video sensor fusion and model based virtual and augmented reality systems and methods
Techniques are disclosed for systems and methods for video based sensor fusion with respect to mobile structures. A mobile structure may include at least one imaging module and multiple navigational sensors and/or receive navigational data from various sources. A navigational database may be generated that includes data from the imaging module, navigational sensors, and/or other sources. Aspects of the navigational database may then be used to generate an integrated model, forecast weather conditions, warn of dangers, identify hard to spot items, and generally aid in the navigation of the mobile structure.
1. An apparatus comprising:
a logic device configured to communicate with a plurality of navigational sensors, at least one orientation and/or position sensor (OPS), and an imaging module coupled to a mobile structure, wherein each navigational sensor is configured to provide navigational data associated with the mobile structure, and wherein the logic device is configured to:
receive at least one image from the imaging module;
receive orientation, position, and/or navigation data corresponding to the at least one image from the OPS and/or the plurality of navigational sensors;
generate an augmented reality (AR) display view for viewing by a user of the mobile structure, the AR display view comprising the at least one image and an AR graphics overlay based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image;
determine a boundary of a cross track error corresponding to a track traversed by the mobile structure; and
render, in the AR display view, a cross track boundary line configured to identify the boundary of the cross track error within the at least one image and a traversal direction associated with the track.
2. The apparatus of claim 1 , wherein:
the logic device comprises a graphics processing unit configured to perform image processing on the at least one image and apply image data slant stabilization to the at least one image from the imaging module based, at least in part, on a mounting of a user interface configured to display AR imagery to the user of the mobile structure, wherein the image data slant stabilization is performed based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image; and
the generating the AR display view comprises:
determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigation data and/or the at least one received image;
rotating pixels of the at least one received image to at least partially compensate for the determined slant;
translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and
cropping the rotated and translated at least one received image symmetrically about a reference slant and/or a reference elevation to compensate for an expected extent of motion of the mobile structure.
3. The apparatus of claim 2 , wherein the generating the AR display view comprises:
generating one or more pixel intensity profiles corresponding to one or more features of the at least one received image and/or the rotated and translated at least one received image;
comparing the one or more pixel intensity profiles to one or more prior pixel intensity profiles generated from a prior received image and/or a prior rotated and translated received image to determine a vertical jitter damping adjustment; and
adjusting the rotating and/or translating the pixels of the at least one received image according to the determined vertical jitter damping adjustment, prior to the cropping the at least one received image.
4. The apparatus of claim 2 , wherein the generating the AR display view comprises:
detecting that the determined elevation of the imaging module is outside a predetermined allowable range of elevations;
retrieving valid pixels from a prior horizon-stabilized image corresponding to a prior determined elevation of the imaging module being substantially within the predetermined allowable range of elevations; and
copying the retrieved valid pixels into corresponding unpopulated pixel locations in the rotated and translated at least one received image, prior to the cropping the at least one received image.
5. The apparatus of claim 2 , wherein the AR display view comprises a plurality of horizon-stabilized images, and wherein the logic device is configured to:
identify a first pixel position and a first pixel area of an object detected in a first one of the plurality of horizon-stabilized images;
identify a second pixel position and a second pixel area of the object in a second one of the plurality of horizon-stabilized images, wherein a first position of the imaging module corresponding to the first horizon-stabilized image is different from a second position of the imaging module corresponding to the second horizon-stabilized image; and
determine an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the first and second pixel positions, pixel areas, and positions of the imaging module.
6. The apparatus of claim 2 , wherein the AR display view comprises a plurality of horizon-stabilized images, and wherein the logic device is configured to:
identify a first plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in a first one of the plurality of horizon-stabilized images;
identify a second plurality of pixel positions and pixel areas corresponding to the plurality of features of the object in a second one of the plurality of horizon-stabilized images, wherein a first position of the imaging module corresponding to the first horizon-stabilized image is different from a second position of the imaging module corresponding to the second horizon-stabilized image; and
generate a three dimensional point cloud model of the object based, at least in part, on the first and second pluralities of pixel positions and pixel areas and the first and second positions of the imaging module.
7. The apparatus of claim 1 , wherein the logic device is configured to:
receive user selection of at least one of the waypoint flags within the AR display view at a first position;
receive user input selecting a second position of the selected waypoint flag within the AR display view;
render the selected waypoint flag at the second position; and
render a lead line from the selected waypoint flag to the first position.
8. The apparatus of claim 1 , further comprising the imaging module and the OPS, wherein the mobile structure comprises a watercraft, and wherein the logic device is configured to:
determine a radius, solid angle, and/or orientation of a field of view (FOV) indicator corresponding to a FOV of the imaging module; and
render the FOV indicator in a chart data portion of the AR display view on the display of the user interface of the mobile structure.
9. The apparatus of claim 1 , wherein the AR display view comprises a plurality of slant and/or elevation stabilized images, and wherein the logic device is configured to:
detect one or more objects in the plurality of horizon-stabilized images and/or the received navigational data;
determine one or more object characteristics of each of the detected objects based, at least in part, on the plurality of horizon-stabilized images and/or the received navigational data;
segregate the detected objects into predetermined types based, at least in part, on the determined object characteristics;
identify one or more of the objects based, at least in part, on a navigational database; and
generate a synthetic Automatic Identification System (AIS) signature for at least one unidentified object, wherein the synthetic AIS signature comprises at least one object characteristic corresponding to the at least one unidentified object.
10. The apparatus of claim 1 , wherein the AR graphics overlay comprises one or more waypoint flags superimposed on the at least one image, each waypoint flag comprising an indication of a characteristic of an object identified in the at least one image.
11. A method comprising:
receiving at least one image from an imaging module coupled to a mobile structure;
receiving orientation, position, and/or navigation data corresponding to the at least one image from an orientation and/or position sensor (OPS) and/or a plurality of navigational sensors, wherein each navigational sensor is configured to provide navigational data associated with the mobile structure;
generating an augmented reality (AR) display view for viewing by a user of the mobile structure, the AR display view comprising the at least one image and an AR graphics overlay based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image;
determining a boundary of a cross track error corresponding to a track traversed by the mobile structure; and
rendering, in the AR display view, a cross track boundary line configured to identify the boundary of the cross track error within the at least one image and a traversal direction associated with the track.
12. The method of claim 11 , further comprising applying image data slant stabilization to the at least one image from the imaging module based, at least in part, on a mounting of a user interface configured to display AR imagery to the user of the mobile structure, wherein the image data slant stabilization is performed based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image, and wherein the generating the AR display view comprises:
determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigation data and/or the at least one received image;
rotating pixels of the at least one received image to at least partially compensate for the determined slant;
translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and
cropping the rotated and translated at least one received image symmetrically about a reference slant and/or a reference elevation to compensate for an expected extent of motion of the mobile structure.
13. The method of claim 12 , wherein the generating the AR display view comprises:
generating one or more pixel intensity profiles corresponding to one or more features of the at least one received image and/or the rotated and translated at least one received image;
comparing the one or more pixel intensity profiles to one or more prior pixel intensity profiles generated from a prior received image and/or a prior rotated and translated received image to determine a vertical jitter damping adjustment; and
adjusting the rotating and/or translating the pixels of the at least one received image according to the determined vertical jitter damping adjustment.
14. The method of claim 12 , wherein the generating the AR display view comprises:
detecting that the determined elevation of the imaging module is outside a predetermined allowable range of elevations;
retrieving valid pixels from a prior horizon-stabilized image corresponding to a prior determined elevation of the imaging module being substantially within the predetermined allowable range of elevations; and
copying the retrieved valid pixels into corresponding unpopulated pixel locations in the rotated and translated at least one received image, prior to the cropping the at least one received image.
15. The method of claim 12 , wherein the AR display view comprises a plurality of slant and/or elevation stabilized images, the method further comprising:
identifying a first pixel position and a first pixel area of an object detected in a first one of the plurality of horizon-stabilized images;
identifying a second pixel position and a second pixel area of the object in a second one of the plurality of horizon-stabilized images, wherein a first position of the imaging module corresponding to the first horizon-stabilized image is different from a second position of the imaging module corresponding to the second horizon-stabilized image; and
determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the first and second pixel positions, pixel areas, and positions of the imaging module.
16. The method of claim 12 , wherein the AR display view comprises a plurality of slant and/or elevation stabilized images, the method further comprising:
identifying a first plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in a first one of the plurality of horizon-stabilized images;
identifying a second plurality of pixel positions and pixel areas corresponding to the plurality of features of the object in a second one of the plurality of horizon-stabilized images, wherein a first position of the imaging module corresponding to the first horizon-stabilized image is different from a second position of the imaging module corresponding to the second horizon-stabilized image; and
generating a three dimensional point cloud model of the object based, at least in part, on the first and second pluralities of pixel positions and pixel areas and the first and second positions of the imaging module.
17. The method of claim 11 , further comprising:
receiving user selection of at least one of the waypoint flags within the AR display view at a first position;
receiving user input selecting a second position of the selected waypoint flag within the AR display view;
rendering the selected waypoint flag at the second position; and
rendering a lead line from the selected waypoint flag to the first position.
18. The method of claim 11 , wherein the mobile structure comprises a watercraft, the method further comprising:
determining a radius, solid angle, and/or orientation of a field of view (FOV) indicator corresponding to a FOV of the imaging module; and
rendering the FOV indicator in a chart data portion of the AR display view on the display of the user interface of the mobile structure.
19. The method of claim 11 , wherein the AR display view comprises a plurality of slant and/or elevation stabilized images, the method further comprising:
detecting one or more objects in the plurality of horizon-stabilized images and/or the received navigational data;
determining one or more object characteristics of each of the detected objects based, at least in part, on the plurality of horizon-stabilized images and/or the received navigational data;
segregating the detected objects into predetermined types based, at least in part, on the determined object characteristics;
identifying one or more of the objects based, at least in part, on a navigational database; and
generating a synthetic Automatic Identification System (AIS) signature for at least one unidentified object, wherein the synthetic AIS signature comprises at least one object characteristic corresponding to the at least one unidentified object.
20. The method of claim 11 , wherein the AR graphics overlay comprises one or more waypoint flags superimposed on the at least one image, each waypoint flag comprising an indication of a characteristic of an object identified in the at least one image.