Water non-water segmentation systems and methods
Techniques are disclosed for systems and methods for water non-water segmentation of navigational imagery to assist in the autonomous navigation of mobile structures. An imagery based navigation system includes a logic device configured to communicate with an imaging module coupled to a mobile structure and/or configured to capture images of an environment about the mobile structure. The logic device may be configured to receive at least one image from the imaging module; determine a water/non-water segmented image based, at least in part, on the received at least one image, and generate a range chart corresponding to the environment about the mobile structure based, at least in part, on the determined water/non-water segmented image and/or the received at least one image.
1. A system comprising:
a logic device configured to communicate with an imaging module configured to capture images of an environment about a mobile structure, wherein the logic device is configured to 1 :
receive at least one image from the imaging module; and
initialize (A) a convolutional autoencoder or (B) a self-supervised neural network arrangement via a dual network training process comprising a semantic segmentation network and a transfer network, wherein the semantic segmentation network is trained to identify water and non-water portions of imagery, wherein the transfer network is trained to implement (A) the convolutional autoencoder or (B) the self-supervised neural network arrangement via the trained semantic segmentation network to perform operations comprising:
(C) determining a (D) water/ 2 (E) non-water segmented image based, at least in part, on the received at least one image, wherein the (D) water/(E) non-water segmented image comprises water pixels and non-water pixels, and/or
(F) generating a range chart corresponding to the environment about the mobile structure based, at least in part, on the determined (D) water/(E) non-water segmented image and/or (G) the received at least one image.
2. The system of claim 1 , wherein the imagery is birds-eye view marine imagery.
3. The system of claim 1 , wherein the determining the water/non-water segmented image comprises:
generating at least one autoencoded image corresponding to the at least one received image by applying a convolutional autoencoder to the at least one received image; and
determining the water/non-water segmented image by applying a graph cut image segmenter to the generated at least one autoencoded image.
4. The system of claim 1 , wherein:
the imaging module comprises a visible spectrum imaging module configured to capture visible spectrum images of the environment about the mobile structure;
the logic device is configured to communicate with a thermal imaging module configured to capture thermal images of the environment about the mobile structure; and
the determining the water/non-water segmented image comprises:
determining a cross-spectral correspondence between at least one visible spectrum image received from the visible spectrum imaging module and at least one thermal image received from the thermal imaging module using a self-supervised neural network arrangement comprising at least two cross-spectral flow sub-networks; and
determining the water/non-water segmented image based, at least in part, on the determined cross-spectral correspondence provided by the self-supervised neural network arrangement.
5. The system of claim 4 , wherein:
the self-supervised neural network arrangement comprises at least one cycle consistency loss feedback to provide for self-supervision of the self-supervised neural network arrangement and at least one edge loss feedback to prevent trivial or zero flow field solutions from being implemented by the at least two cross-spectral flow sub-networks.
6. The system of claim 1 , wherein the generating the range chart comprises:
determining a pixel range from the imaging module to each water pixel in the water/non-water segmented image based, at least in part, on an estimated perpendicular height of the imaging module above a corresponding water surface and/or an orientation of the imaging module relative to the water surface;
determining a set of range contours comprising both water pixels and non-water pixels within the water/non-water segmented image, wherein the set of range contours correspond to a set of range chart intervals; and
generating the range chart as a plan view of the water/non-water segmented image based, at least in part, on the set of range contours.
7. The system of claim 6 , wherein the generating the plan view of the water/non-water segmented image comprises:
plotting each pixel within the set of range contours on a two-dimensional plane based, at least in part, on a corresponding pixel position within the water/non-water segmented image and/or a field of view of the imaging module; and
filling pixel gaps between the range contours by interpolation between pixels within adjacent range contours along a radial direction extending from the imaging module to the pixel gaps.
8. The system of claim 1 , further comprising a navigational sensor configured to provide navigational data associated with the mobile structure and at least one orientation and/or position sensor (OPS), wherein:
the logic device is configured to:
receive orientation, position, and/or navigation data corresponding to the at least one image from the OPS and/or the navigational sensor; and
generate an information display view comprising the range chart and at least one augmented image corresponding to the received at least one image based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image.
9. The system of claim 8 , wherein the imaging module is coupled to the mobile structure and the OPS is coupled to the mobile structure or within the imaging module, and wherein the generating the information display view comprises:
identifying a pixel position and a pixel area of an object detected in the at least one image from the imaging module;
determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the pixel position, pixel area, and a position of the imaging module; and
rendering a graphical indicator and/or information graphic overlaying the object as shown in the range chart and/or the at least one augment image configured to indicate the estimated range to the object, the estimated velocity of the object, and/or the estimated relative position of the object, wherein the graphical indicator and/or information graphic are rendered to have a perspective roughly following a surface of a medium surrounding the mobile structure.
10. The system of claim 1 , wherein the logic device is configured to:
identify a plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in the received at least one image and/or the water/non-water segmented image; and
generate a three dimensional point cloud model of the object based, at least in part, on the plurality of pixel positions and pixel areas.
11. The system of claim 1 , wherein the logic device is configured to generate at least one horizon-stabilized image corresponding to the received at least one image, prior to determining the water/non-water segmented image using the at least one horizon-stabilized image, by:
determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigation data and/or the at least one received image;
rotating pixels of the at least one received image to at least partially compensate for the determined slant;
translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and
cropping the rotated and translated at least one received image symmetrically about a reference slant and/or a reference elevation to compensate for an expected extent of motion of the mobile structure.
12. A method comprising 3 :
receiving at least one image from an imaging module coupled to a mobile structure; and
initializing (A) a convolutional autoencoder or (B) a self-supervised neural network arrangement via a dual network training process comprising a semantic segmentation network and a transfer network, wherein the semantic segmentation network is trained to identify water and non-water portions of imagery, and wherein the transfer network is trained to implement (A) the convolutional autoencoder or (B) the self-supervised neural network arrangement via the trained semantic segmentation network to perform operations comprising:
determining a (C) water/ 4 (D) non-water segmented image based, at least in part, on the received at least one image, wherein the (C) water/(D) non-water segmented image comprises water pixels and non-water pixels, and
generating a range chart corresponding to the environment about the mobile structure based, at least in part, on the determined (C) water/(D) non-water segmented image and/or (E) the received at least one image.
13. The method of claim 12 , wherein the imagery is birds-eye view marine imagery.
14. The method of claim 12 , wherein the determining the water/non-water segmented image comprises:
generating at least one autoencoded image corresponding to the at least one received image by applying a convolutional autoencoder to the at least one received image; and
determining the water/non-water segmented image by applying a graph cut image segmenter to the generated at least one autoencoded image.
15. The method of claim 12 , wherein:
the imaging module comprises a visible spectrum imaging module configured to capture visible spectrum images of the environment about the mobile structure;
the determining the water/non-water segmented image comprises:
determining a cross-spectral correspondence between at least one visible spectrum image received from the visible spectrum imaging module and at least one thermal image received from a thermal imaging module using a self-supervised neural network arrangement comprising at least two cross-spectral flow sub-networks; and
determining the water/non-water segmented image based, at least in part, on the determined cross-spectral correspondence provided by the self-supervised neural network arrangement.
16. The method of claim 15 , wherein:
the self-supervised neural network arrangement comprises at least one cycle consistency loss feedback to provide for self-supervision of the self-supervised neural network arrangement and at least one edge loss feedback to prevent trivial or zero flow field solutions from being implemented by the at least two cross-spectral flow sub-networks.
17. The method of claim 12 , wherein the generating the range chart comprises:
determining a pixel range from the imaging module to each water pixel in the water/non-water segmented image based, at least in part, on an estimated perpendicular height of the imaging module above a corresponding water surface and/or an orientation of the imaging module relative to the water surface;
determining a set of range contours comprising both water pixels and non-water pixels within the water/non-water segmented image, wherein the set of range contours correspond to a set of range chart intervals; and
generating the range chart as a plan view of the water/non-water segmented image based, at least in part, on the set of range contours.
18. The method of claim 17 , wherein the generating the plan view of the water/non-water segmented image comprises:
plotting each pixel within the set of range contours on a two-dimensional plane based, at least in part, on a corresponding pixel position within the water/non-water segmented image and/or a field of view of the imaging module; and
filling pixel gaps between the range contours by interpolation between pixels within adjacent range contours along a radial direction extending from the imaging module to the pixel gaps.
19. The method of claim 12 , further comprising:
receiving orientation, position, and/or navigation data corresponding to the at least one image from an orientation and/or position sensor (OPS) and/or a navigational sensor; and
generating an information display view comprising the range chart and at least one augmented image corresponding to the received at least one image based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image, wherein the imaging module is coupled to the mobile structure and the OPS is coupled to the mobile structure or within the imaging module, and wherein the generating the information display view comprises:
identifying a pixel position and a pixel area of an object detected in the at least one image from the imaging module;
determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the pixel position, pixel area, and a position of the imaging module; and
rendering a graphical indicator and/or information graphic overlaying the object as shown in the range chart and/or the at least one augment image configured to indicate the estimated range to the object, the estimated velocity of the object, and/or the estimated relative position of the object, wherein the graphical indicator and/or information graphic are rendered to have a perspective roughly following a surface of a medium surrounding the mobile structure.
20. The method of claim 12 , wherein the logic device is configured to:
identify a plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in the received at least one image and/or the water/non-water segmented image; and
generate a three dimensional point cloud model of the object based, at least in part, on the plurality of pixel positions and pixel areas, further comprising generating at least one horizon-stabilized image corresponding to the received at least one image, prior to determining the water/non-water segmented image using the at least one horizon-stabilized image, by:
determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigation data and/or the at least one received image;
rotating pixels of the at least one received image to at least partially compensate for the determined slant;
translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and
cropping the rotated and translated at least one received image symmetrically about a reference slant and/or a reference elevation to compensate for an expected extent of motion of the mobile structure.