Granted Patent B2

US 12,530,885 · App. 17/891,001 · Granted Jan 20, 2026

Inventors: Mark Johnson (Vannes, FR); Celyn Walters (Fareham, GB); Richard Bowden (Surrey, GB); Oscar Mendez Maldonado (Surrey, GB)

Assignee: RAYMARINE UK LIMITED

G06V20/10B63B49/00G06T3/60G06T7/162G06T7/50G06T7/73G06T11/00G06T17/05G06V10/26G06V10/776G06V10/803G06V10/82G06T2207/10028G06T2207/10048G06T2207/20084G06T2207/20132G06T2207/30181G06T2207/30244G06T2210/56

Abstract

Claims (83)

1 . A system comprising:

a navigational sensor configured to provide navigational data associated with a mobile structure and at least one orientation and/or position sensor (OPS);

a logic device configured to communicate with an imaging module configured to capture images of an environment about the mobile structure, wherein the logic device is configured to:

receive at least one image from the imaging module;

receive orientation, position, and/or navigation data corresponding to the at least one image from the OPS and/or the navigational sensor;

determine a water/non-water segmented image based, at least in part, on the received at least one image, wherein the water/non-water segmented image comprises water pixels and non-water pixels;

generate a range chart corresponding to the environment about the mobile structure based, at least in part, on the determined water/non-water segmented image and/or the received at least one image; and

generate an information display view comprising the range chart and at least one augmented image corresponding to the received at least one image based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image.

2 . A system comprising:

an imaging module configured to capture images of an environment about a mobile structure;

a logic device configured to communicate with the imaging module, wherein the logic device is configured to:

receive at least one image from the imaging module;

determine a water/non-water segmented image based, at least in part, on the received at least one image, wherein the water/non-water segmented image comprises water pixels and non-water pixels, wherein the determining the water/non-water segmented image comprises:

generating at least one autoencoded image corresponding to the at least one received image by applying a convolutional autoencoder to the at least one received image; and

determining the water/non-water segmented image by applying a graph cut image segmenter to the generated at least one autoencoded image; and

generate a range chart corresponding to the environment about the mobile structure based, at least in part, on the determined water/non-water segmented image and/or the received at least one image.

3 . The system of claim 1 , wherein:

the imaging module comprises a visible spectrum imaging module configured to capture visible spectrum images of the environment about the mobile structure;

the logic device is configured to communicate with a thermal imaging module configured to capture thermal images of the environment about the mobile structure; and

the determining the water/non-water segmented image comprises:

determining a cross-spectral correspondence between at least one visible spectrum image received from the visible spectrum imaging module and at least one thermal image received from the thermal imaging module using a self-supervised neural network arrangement comprising at least two cross-spectral flow sub-networks; and

determining the water/non-water segmented image based, at least in part, on the determined cross-spectral correspondence provided by the self-supervised neural network arrangement.

4 . The system of claim 3 , wherein:

the self-supervised neural network arrangement comprises at least one cycle consistency loss feedback to provide for self-supervision of the self-supervised neural network arrangement and at least one edge loss feedback to prevent trivial or zero flow field solutions from being implemented by the at least two cross-spectral flow sub-networks.

5 . The system of claim 1 , wherein the generating the range chart comprises:

determining a pixel range from the imaging module to each water pixel in the water/non-water segmented image based, at least in part, on an estimated perpendicular height of the imaging module above a corresponding water surface and/or an orientation of the imaging module relative to the water surface;

determining a set of range contours comprising both water pixels and non-water pixels within the water/non-water segmented image, wherein the set of range contours correspond to a set of range chart intervals; and

generating the range chart as a plan view of the water/non-water segmented image based, at least in part, on the set of range contours.

6 . The system of claim 5 , wherein the generating the plan view of the water/non-water segmented image comprises:

plotting each pixel within the set of range contours on a two-dimensional plane based, at least in part, on a corresponding pixel position within the water/non-water segmented image and/or a field of view of the imaging module; and

filling pixel gaps between the range contours by interpolation between pixels within adjacent range contours along a radial direction extending from the imaging module to the pixel gaps.

7 . The system of claim 1 , wherein the imaging module is coupled to the mobile structure and the OPS is coupled to the mobile structure or within the imaging module, and wherein the generating the information display view comprises:

identifying a pixel position and a pixel area of an object detected in the at least one image from the imaging module;

determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the pixel position, pixel area, and a position of the imaging module; and

rendering a graphical indicator and/or information graphic overlaying the object as shown in the range chart and/or the at least one augment image configured to indicate the estimated range to the object, the estimated velocity of the object, and/or the estimated relative position of the object, wherein the graphical indicator and/or information graphic are rendered to have a perspective roughly following a surface of a medium surrounding the mobile structure.

8 . The system of claim 1 , wherein the logic device is configured to:

identify a plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in the received at least one image and/or the water/non-water segmented image; and

generate a three dimensional point cloud model of the object based, at least in part, on the plurality of pixel positions and pixel areas.

9 . The system of claim 1 , wherein the logic device is configured to generate at least one horizon-stabilized image corresponding to the received at least one image, prior to determining the water/non-water segmented image using the at least one horizon-stabilized image, by:

determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigation data and/or the at least one received image;

rotating pixels of the at least one received image to at least partially compensate for the determined slant;

translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and

cropping the rotated and translated at least one received image symmetrically about a reference slant and/or a reference elevation to compensate for an expected extent of motion of the mobile structure.

10 . A method comprising:

receiving at least one image from an imaging module coupled to a mobile structure;

determine a water/non-water segmented image based, at least in part, on the received at least one image, wherein the water/non-water segmented image comprises water pixels and non-water pixels;

generating a range chart corresponding to the environment about the mobile structure based, at least in part, on the determined water/non-water segmented image and/or the received at least one image;

receiving orientation, position, and/or navigation data corresponding to the at least one image from an orientation and/or position sensor (OPS) and/or a navigational sensor; and

generating an information display view comprising the range chart and at least one augmented image corresponding to the received at least one image based, at least in part, on the received orientation, position, and/or navigation data and/or the received at least one image.

11 . The method of claim 10 , wherein the determining the water/non-water segmented image comprises:

generating at least one autoencoded image corresponding to the at least one received image by applying a convolutional autoencoder to the at least one received image; and

determining the water/non-water segmented image by applying a graph cut image segmenter to the generated at least one autoencoded image.

12 . The method of claim 10 , wherein:

the imaging module comprises a visible spectrum imaging module configured to capture visible spectrum images of the environment about the mobile structure;

the determining the water/non-water segmented image comprises:

determining a cross-spectral correspondence between at least one visible spectrum image received from the visible spectrum imaging module and at least one thermal image received from a thermal imaging module using a self-supervised neural network arrangement comprising at least two cross-spectral flow sub-networks; and

determining the water/non-water segmented image based, at least in part, on the determined cross-spectral correspondence provided by the self-supervised neural network arrangement.

13 . The method of claim 12 , wherein:

the self-supervised neural network arrangement comprises at least one cycle consistency loss feedback to provide for self-supervision of the self-supervised neural network arrangement and at least one edge loss feedback to prevent trivial or zero flow field solutions from being implemented by the at least two cross-spectral flow sub-networks.

14 . The method of claim 10 , wherein the generating the range chart comprises:

determining a pixel range from the imaging module to each water pixel in the water/non-water segmented image based, at least in part, on an estimated perpendicular height of the imaging module above a corresponding water surface and/or an orientation of the imaging module relative to the water surface;

determining a set of range contours comprising both water pixels and non-water pixels within the water/non-water segmented image, wherein the set of range contours correspond to a set of range chart intervals; and

generating the range chart as a plan view of the water/non-water segmented image based, at least in part, on the set of range contours.

15 . The method of claim 14 , wherein the generating the plan view of the water/non-water segmented image comprises:

plotting each pixel within the set of range contours on a two-dimensional plane based, at least in part, on a corresponding pixel position within the water/non-water segmented image and/or a field of view of the imaging module; and

filling pixel gaps between the range contours by interpolation between pixels within adjacent range contours along a radial direction extending from the imaging module to the pixel gaps.

16 . The method of claim 10 , wherein the imaging module is coupled to the mobile structure and the OPS is coupled to the mobile structure or within the imaging module, and wherein the generating the information display view comprises:

identifying a pixel position and a pixel area of an object detected in the at least one image from the imaging module;

determining an estimated range to the object, an estimated velocity of the object, and/or an estimated relative position of the object based, at least in part, on the pixel position, pixel area, and a position of the imaging module; and

rendering a graphical indicator and/or information graphic overlaying the object as shown in the range chart and/or the at least one augment image configured to indicate the estimated range to the object, the estimated velocity of the object, and/or the estimated relative position of the object, wherein the graphical indicator and/or information graphic are rendered to have a perspective roughly following a surface of a medium surrounding the mobile structure.

17 . The method of claim 10 , wherein the logic device is configured to:

identify a plurality of pixel positions and pixel areas corresponding to a plurality of features of an object detected in the received at least one image and/or the water/non-water segmented image; and

generate a three dimensional point cloud model of the object based, at least in part, on the plurality of pixel positions and pixel areas.

18 . The method of claim 10 , further comprising generating at least one horizon-stabilized image corresponding to the received at least one image, prior to determining the water/non-water segmented image using the at least one horizon-stabilized image, by:

determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigation data and/or the at least one received image;

rotating pixels of the at least one received image to at least partially compensate for the determined slant;

translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and

cropping the rotated and translated at least one received image symmetrically about a reference slant and/or a reference elevation to compensate for an expected extent of motion of the mobile structure.

19 . The system of claim 1 , further comprising the imaging module.

20 . The system of claim 2 , wherein the generating the range chart comprises:

determining a pixel range from the imaging module to each water pixel in the water/non-water segmented image based, at least in part, on an estimated perpendicular height of the imaging module above a corresponding water surface and/or an orientation of the imaging module relative to the water surface;

determining a set of range contours comprising both water pixels and non-water pixels within the water/non-water segmented image, wherein the set of range contours correspond to a set of range chart intervals; and

generating the range chart as a plan view of the water/non-water segmented image based, at least in part, on the set of range contours.

Assignments (2)

Continuity (8)

References Cited (14)