Granted Patent B1

US 12,617,341 · App. 19/220,930 · Granted May 5, 2026

Inventors: Joona Petrell (Tampere, FI); Thomas Carlsson (Vantaa, FI)

Assignee: Distance Technologies Oy

B60R1/24G01C21/3602G06F3/013G06T17/00G06T19/006G06V10/25G06V10/60G06V20/58H04N7/181H04N23/13B60R2300/105B60R2300/303B60R2300/8053

Abstract

Claims (100)

1 . A system for improving visibility in low-visibility conditions, the system being implemented in a vehicle, the system comprising:

at least one visible-light camera;

at least one depth camera;

a display unit;

an optical combiner arranged on an optical path of the display unit and on an optical path of a real-world light field of a real-world environment, wherein the real-world light field is incoming via a windshield of the vehicle; and

at least one processor configured to:

obtain information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of the optical combiner;

control the at least one visible-light camera and the at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;

for a given visible-light image from amongst the plurality of visible-light images, identify, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;

identify at least one corresponding image segment in the given visible-light image whose visual information is any one of: occluded, degraded, based on: (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner; and

generate an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image; and

display the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;

wherein the image is a light field image, and the display unit is a light field display unit comprising a multiscopic optical element configured to direct a first part and a second part of the synthetic light field towards a first eye and a second eye of an individual one of the at least one user, via the optical combiner, presenting a first image and a second image to the first eye and the second eye, respectively, wherein the at least one processor is configured to generate the light field image by utilising the first image and the second image.

2 . The system of claim 1 , further comprising at least one thermal imaging camera controlled to capture a plurality of thermal images of the region of the real-world environment that is front of the vehicle, wherein the at least one processor is configured to:

for the given visible-light image, identify, in a corresponding thermal image from amongst the plurality of thermal images, at least one image segment that represents the at least one object present in the region of the real-world environment; and

identify the at least one corresponding image segment in the given visible-light image, based further on the at least one image segment that is identified in the corresponding thermal image.

3 . The system of claim 1 , further comprising at least one light sensor configured to detect illuminances in sub-regions of the region in the real-world environment, when capturing the given visible-light image and the corresponding depth image, wherein the at least one processor is further configured to identify the at least one corresponding image segment in the given visible-light image, based further on the detected illuminances in the sub-regions of the region in the real-world environment.

4 . The system of claim 1 , further comprising at least one sensor configured to detect the presence of a substance on at least one portion of a surface of the windshield, wherein the at least one processor is further configured to identify the at least one corresponding image segment in the given visible-light image, based on the location of the at least one portion of the surface of the windshield.

5 . The system of claim 1 , wherein the at least one processor is further configured to:

obtain a three-dimensional (3D) model of the real-world environment, from a data repository; and

environment and a current location of the vehicle.

6 . The system of claim 1 , further comprising a geospatial navigation device configured to generate real-time navigation information for the vehicle, wherein the at least one processor is further configured to generate the at least one virtual element, based on the real-time navigation information.

7 . The system of claim 1 , wherein the at least one processor is further configured to:

obtain information indicative of a current weather condition in the real-world environment; and

perform the steps of obtaining the information indicative of the relative position, controlling the at least one visible-light camera and the at least one depth camera, identifying the at least one image segment, identifying the at least one corresponding image segment, generating the image, and displaying the image, based on the weather condition in the real-world environment.

8 . The system of claim 7 , further comprising at least one weather sensor, wherein the at least one processor is further configured to determine the current weather condition in the real-world environment, based on current sensor data collected by at least one weather sensor and at least visual information represented in the given visible-light image, by utilising at least one neural network.

9 . The system of claim 1 , wherein the at least one processor is further configured to dynamically adjust at least one of: a brightness, a colour, a contrast, an illuminance, a sharpness, a transparency, of the at least one virtual element, based on at least one of: (i) a current weather condition in the real-world environment, (ii) an optical depth of the at least one object from the eyes of the at least one user.

10 . The system of claim 1 , wherein the at least one processor is further configured to identify the at least one corresponding image segment whose visual information is degraded, based further on at least one of: a contrast level, a blur level, a brightness level, a colour desaturation level, of the given corresponding image segment.

11 . A system for improving visibility in low-visibility conditions, the system being implemented in a vehicle, the system comprising:

at least one visible-light camera;

at least one depth camera;

a display unit;

an optical combiner arranged on an optical path of the display unit and on an optical path of a real-world light field of a real-world environment, wherein the real-world light field is incoming via a windshield of the vehicle; and

at least one processor configured to:

obtain information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of the optical combiner;

control the at least one visible-light camera and the at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;

for a given visible-light image from amongst the plurality of visible-light images, identify, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;

identify at least one corresponding image segment in the given visible-light image whose visual information is any one of: occluded, degraded, based on: (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner;

generate an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image; and

display the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;

wherein when identifying the at least one image segment, the at least one processor is further configured to:

process, by utilising a movement filtering technique, the corresponding depth image; and

identify the at least one object as a moving object present in the region of the real-world environment, wherein when generating the image, the at least one processor is configured to generate a virtual element corresponding to the moving object.

12 . A system for improving visibility in low-visibility conditions, the system being implemented in a vehicle, the system comprising:

at least one visible-light camera;

at least one depth camera;

a display unit;

an optical combiner arranged on an optical path of the display unit and on an optical path of a real-world light field of a real-world environment, wherein the real-world light field is incoming via a windshield of the vehicle; and

at least one processor configured to:

obtain information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of the optical combiner;

control the at least one visible-light camera and the at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;

for a given visible-light image from amongst the plurality of visible-light images, identify, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;

identify at least one corresponding image segment in the given visible-light image whose visual information is any one of: occluded, degraded, based on: (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner; and

generate an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image; and

display the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;

the system further comprising a tracker, wherein the at least one processor is further configured to:

determine gaze directions of the eyes of the at least one user, by utilising the tracker;

determine a focus depth at which the at least one user is gazing, based on the gaze directions of the eyes;

detect when a difference between the focus depth and an optical depth at which the at least one virtual element is being presented via the synthetic light field is greater than a predefined threshold difference; and

when it is detected that said difference is greater than the predefined threshold difference, skip generating the image and displaying the image.

13 . A system for improving visibility in low-visibility conditions, the system being implemented in a vehicle, the system comprising:

at least one visible-light camera;

at least one depth camera;

a display unit;

an optical combiner arranged on an optical path of the display unit and on an optical path of a real-world light field of a real-world environment, wherein the real-world light field is incoming via a windshield of the vehicle; and

at least one processor configured to:

obtain information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of the optical combiner;

control the at least one visible-light camera and the at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;

for a given visible-light image from amongst the plurality of visible-light images, identify, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;

identify at least one corresponding image segment in the given visible-light image whose visual information is any one of: occluded, degraded, based on: (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner;

generate an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image; and

display the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;

the system further comprising a tracker, wherein the at least one virtual element comprises a plurality of virtual elements, and wherein the at least one processor is further configured to:

determine gaze directions of the eyes of the at least one user, by utilising the tracker;

determine a focus depth at which the at least one user is gazing, based on the gaze directions of the eyes;

detect a difference between the focus depth and an optical depth at which respective ones of the plurality of virtual elements are to be presented via the synthetic light field; and

perform digital superimposition of only those virtual elements on the at least one corresponding image segment for which said difference is less than a predefined threshold difference, when generating the image.

14 . A system for improving visibility in low-visibility conditions, the system being implemented in a vehicle, the system comprising:

at least one visible-light camera;

at least one depth camera;

a display unit;

an optical combiner arranged on an optical path of the display unit and on an optical path of a real-world light field of a real-world environment, wherein the real-world light field is incoming via a windshield of the vehicle; and

at least one processor configured to:

obtain information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of the optical combiner;

control the at least one visible-light camera and the at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;

for a given visible-light image from amongst the plurality of visible-light images, identify, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;

identify at least one corresponding image segment in the given visible-light image whose visual information is any one of: occluded, degraded, based on: (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner;

generate an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image; and

display the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;

wherein prior to identifying the at least one image segment in the corresponding depth image, and identifying the at least one corresponding image segment in the given visible-light image, the at least one processor is further configured to:

reproject the corresponding depth image, from a camera pose of the at least one depth camera to a head pose of the at least one user or an eye pose of a given eye of the at least one user; and

reproject the given visible-light image, from a camera pose of the at least one visible-light camera to the head pose of the at least one user or the eye pose of the given eye of the at least one user, based on depth information in the corresponding depth image.

15 . A method for improving visibility in low-visibility conditions, the method being implemented in a vehicle, the method comprising:

obtaining information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of an optical combiner, the optical combiner being arranged on an optical path of a display unit and on an optical path of a real-world light field of a real-world environment, the real-world light field incoming via a windshield of the vehicle;

controlling at least one visible-light camera and at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;

for a given visible-light image from amongst the plurality of visible-light images, identifying, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;

identifying at least one corresponding image segment in the given visible-light image whose visual information is any one of: occluded, degraded, based on: (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner; and

generating an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image; and

displaying the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;

wherein the image is a light field image, and the display unit is a light field display unit comprising a multiscopic optical element configured to direct a first part and a second part of the synthetic light field towards a first eye and a second eye of an individual one of the at least one user, via the optical combiner, presenting a first image and a second image to the first eye and the second eye, respectively, and the light field image is generated by utilising the first image and the second image.

Assignments (1)

References Cited (25)