Video sensor fusion and model based virtual and augmented reality systems and methods

1. An apparatus comprising:

a logic device configured to communicate with a navigational sensor, at least one orientation and/or position sensor (OPS), and an imaging module coupled to a mobile structure, wherein the navigational sensor is configured to provide navigational data associated with the mobile structure, and wherein the logic device is configured to:

receive at least one image from the imaging module;

receive orientation, position, and/or navigation data corresponding to the at least one image from the OPS and the navigational sensor;

generate an information display view comprising at least one augmented image corresponding to the received at least one image based, at least in part, on the received orientation, position, and navigation data and the received at least one image; and

generate a synthetic viewpoint elevated image corresponding to the at least one augmented image by:

identifying object pixels and a water contact reference pixel position corresponding to a detected object in the at least one augmented image;

removing the identified object pixels from the at least one augmented image;

backfilling unpopulated pixel locations corresponding to the removed object pixels;

generating a vertically stretched image from the at least one augmented image; and

copying the identified object pixels into the vertically stretched image according to the water contact reference pixel position to generate the synthetic viewpoint elevated image.

2. The apparatus of claim 1 , wherein the generating the at least one augmented image comprises:

identifying a pixel position and a pixel area of an object detected in the at least one image from the imaging module;

determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the pixel position, pixel area, and a position of the imaging module; and

rendering a graphical indicator and/or information graphic overlaying the at least one image from the imaging module configured to indicate the estimated range to the object, the estimated velocity of the object, and/or the estimated relative position of the object, wherein the graphical indicator and/or information graphic are rendered to have a perspective roughly following a surface of a medium surrounding the mobile structure.

3. The apparatus of claim 1 , wherein the generating the at least one augmented image comprises generating at least one horizon-stabilized image by:

determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigation data and/or the at least one received image;

rotating pixels of the at least one received image to at least partially compensate for the determined slant;

translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and

cropping the rotated and translated at least one received image symmetrically about a reference slant and/or a reference elevation to compensate for an expected extent of motion of the mobile structure.

4. The apparatus of claim 3 , wherein the generating the at least one horizon-stabilized image comprises:

generating one or more pixel intensity profiles corresponding to one or more features of the at least one received image and/or the rotated and translated at least one received image;

comparing the one or more pixel intensity profiles to one or more prior pixel intensity profiles generated from a prior received image and/or a prior rotated and translated received image to determine a vertical jitter damping adjustment; and

adjusting the rotating and/or translating the pixels of the at least one received image according to the determined vertical jitter damping adjustment, prior to the cropping the at least one received image.

5. The apparatus of claim 1 , wherein the logic device is configured to synthetically remove foreground obstructions in the at least one received image, before generating the at least one augmented image, by:

identifying foreground pixels in the at least one received image based on at least two images received prior to the at least one received image;

selecting a prior received image to provide background pixels corresponding to the foreground pixels in the at least one received image, based on orientation, position, and/or navigational data corresponding to the prior received image and the at least one received image; and

replacing the foreground pixels in the at least one received image with the background pixels from the selected prior received image.

6. The apparatus of claim 1 , wherein the at least one augmented image comprises a plurality of augmented images, and wherein the logic device is configured to:

identify a first pixel position and a first pixel area of an object detected in a first one of the plurality of augmented images;

identify a second pixel position and a second pixel area of the object in a second one of the plurality of augmented images, wherein a first position of the imaging module corresponding to the first augmented image is different from a second position of the imaging module corresponding to the second augmented image; and

determine an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the first and second pixel positions, pixel areas, and positions of the imaging module.

7. The apparatus of claim 1 , further comprising the imaging module and the OPS, wherein the imaging module is coupled to the mobile structure and the OPS is coupled to the mobile structure or within the imaging module, wherein the at least one augmented image comprises a plurality of augmented images, and wherein the logic device is configured to:

identify a first plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in a first one of the plurality of augmented images;

identify a second plurality of pixel positions and pixel areas corresponding to the plurality of features of the object in a second one of the plurality of augmented images, wherein a first position of the imaging module corresponding to the first augmented image is different from a second position of the imaging module corresponding to the second augmented image; and

generate a three dimensional point cloud model of the object based, at least in part, on the first and second pluralities of pixel positions and pixel areas and the first and second positions of the imaging module.

8. The apparatus of claim 1 , further comprising the imaging module and the OPS, wherein the imaging module is coupled to the mobile structure by an active or passive mechanically horizon-stabilized mount, wherein the mobile structure comprises a watercraft, and wherein the logic device is configured to:

combine at least a portion of the navigational data with the at least one augmented image or a synthetic viewpoint elevated image derived from the at least one augmented image to produce a combined image; and

render the combined image on a display of a user interface of the mobile structure, wherein the combined image comprises:

the at least one augmented image or the synthetic viewpoint elevated image overlaid with navigational information corresponding to the mobile structure or a detected object; or

a navigational chart comprising navigational data derived, at least in part, from the at least one augmented image or the synthetic viewpoint elevated image.

9. The apparatus of claim 1 , wherein the at least one augmented image comprises a plurality of augmented images, and wherein the logic device is configured to:

detect one or more objects in the plurality of augmented images and/or the received navigational data;

determine one or more object characteristics of each of the detected objects based, at least in part, on the plurality of augmented images and/or the received navigational data;

segregate the detected objects into predetermined types based, at least in part, on the determined object characteristics;

identify one or more of the objects based, at least in part, on a navigational database; and

generate a synthetic Automatic Identification System (AIS) signature for at least one unidentified object, wherein the synthetic AIS signature comprises at least one object characteristic corresponding to the at least one unidentified object.

10. An Apparatus comprising:

a logic device configured to communicate with a navigational sensor, at least one orientation and/or position sensor (OPS), and an imaging module coupled to a mobile structure, wherein the navigational sensor is configured to provide navigational data associated with the mobile structure, and wherein the logic device is configured to:

receive at least one image from the imaging module;

receive orientation, position, and/or navigation data corresponding to the at least one image from the OPS and the navigational sensor; and

generate an information display view comprising at least one augmented image corresponding to the received at least one image based, at least in part, on the received orientation, position, and navigation data and the received at least one image;

wherein the generating the at least one horizon-stabilized image comprises:

generating one or more pixel intensity profiles corresponding to one or more features of the at least one received image;

comparing the one or more pixel intensity profiles to one or more prior pixel intensity profiles generated from a prior received image to determine a vertical jitter damping adjustment; and

adjusting positioning of pixels within the at least one received image according to the determined vertical jitter damping adjustment.

11. A method comprising:

receiving at least one image from an imaging module coupled to a mobile structure;

receiving orientation, position, and navigation data corresponding to the at least one image from an orientation and/or position sensor (OPS) and a plurality of navigational sensors, wherein each navigational sensor is configured to provide navigational data associated with the mobile structure; and

generating an information display view comprising at least one augmented image corresponding to the received at least one image based, at least in part, on the received orientation, position, and navigation data and the received at least one image; and

synthetically removing foreground obstructions in the at least one received image by:

identifying foreground pixels in the at least one received image based on at least two images received prior to the at least one received image;

selecting a prior received image to provide background pixels corresponding to the foreground pixels in the at least one received image; and

replacing the foreground pixels in the at least one received image with the background pixels from the selected prior received image.

12. The method of claim 11 , wherein the generating the at least one augmented image comprises:

identifying a pixel position and a pixel area of an object detected in the at least one image from the imaging module;

determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the pixel position, pixel area, and a position of the imaging module; and

rendering a graphical indicator and/or information graphic overlaying the at least one image from the imaging module configured to indicate the estimated range to the object, the estimated velocity of the object, and/or the estimated relative position of the object, wherein the graphical indicator and/or information graphic are rendered to have a perspective roughly following a surface of a medium surrounding the mobile structure.

13. The method of claim 11 , wherein the generating the at least one augmented image comprises generating at least one horizon-stabilized image by:

determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigation data and/or the at least one received image;

rotating pixels of the at least one received image to at least partially compensate for the determined slant;

translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and

cropping the rotated and translated at least one received image symmetrically about a reference slant and/or a reference elevation to compensate for an expected extent of motion of the mobile structure.

14. The method of claim 13 , wherein the generating the at least one horizon-stabilized image comprises:

detecting that the determined elevation of the imaging module is outside a predetermined allowable range of elevations;

retrieving valid pixels from a prior horizon-stabilized image corresponding to a prior determined elevation of the imaging module being substantially within the predetermined allowable range of elevations; and

copying the retrieved valid pixels into corresponding unpopulated pixel locations in the rotated and translated at least one received image, prior to the cropping the at least one received image.

15. The method of claim 11 , further comprising generating a synthetic viewpoint elevated image corresponding to the at least one augmented image by:

identifying object pixels and a water contact reference pixel position corresponding to a detected object in the at least one augmented image;

removing the identified object pixels from the at least one augmented image;

backfilling unpopulated pixel locations corresponding to the removed object pixels;

generating a vertically stretched image from the at least one augmented image; and

copying the identified object pixels into the vertically stretched image according to the water contact reference pixel position to generate the synthetic viewpoint elevated image.

16. The method of claim 11 , wherein the imaging module is coupled to the mobile structure and the OPS is coupled to the mobile structure or within the imaging module, wherein the at least one augmented image comprises a plurality of augmented images, the method further comprising:

identifying a first pixel position and a first pixel area of an object detected in a first one of the plurality of augmented images;

identifying a second pixel position and a second pixel area of the object in a second one of the plurality of augmented images, wherein a first position of the imaging module corresponding to the first augmented image is different from a second position of the imaging module corresponding to the second augmented image; and

determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the first and second pixel positions, pixel areas, and positions of the imaging module.

17. The method of claim 11 , wherein the at least one augmented image comprises a plurality of augmented images, the method further comprising:

identifying a first plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in a first one of the plurality of augmented images;

identifying a second plurality of pixel positions and pixel areas corresponding to the plurality of features of the object in a second one of the plurality of augmented images, wherein a first position of the imaging module corresponding to the first augmented image is different from a second position of the imaging module corresponding to the second augmented image; and

generating a three dimensional point cloud model of the object based, at least in part, on the first and second pluralities of pixel positions and pixel areas and the first and second positions of the imaging module.

18. The method of claim 11 , wherein the imaging module is coupled to the mobile structure by an active or passive mechanically horizon-stabilized mount, wherein the mobile structure comprises a watercraft, the method further comprising:

combining at least a portion of the navigational data with the at least one augmented image or a synthetic viewpoint elevated image derived from the at least one augmented image to produce a combined image; and

rendering the combined image on a display of a user interface of the mobile structure, wherein the combined image comprises:

the at least one augmented image or the synthetic viewpoint elevated image overlaid with object identification information, channel depth information, distance measurement information, object course and/or speed information, and/or other navigational information corresponding to the mobile structure or a detected object; or

a navigational chart comprising navigational data derived, at least in part, from the at least one augmented image or the synthetic viewpoint elevated image.

19. The method of claim 11 , wherein the at least one augmented image comprises a plurality of augmented images, the method further comprising:

detecting one or more objects in the plurality of augmented images and/or the received navigational data;

determining one or more object characteristics of each of the detected objects based, at least in part, on the plurality of augmented images and/or the received navigational data;

segregating the detected objects into predetermined types based, at least in part, on the determined object characteristics;

identifying one or more of the objects based, at least in part, on a navigational database; and

generating a synthetic Automatic Identification System (AIS) signature for at least one unidentified object, wherein the synthetic AIS signature comprises at least one object characteristic corresponding to the at least one unidentified object.