IP Library Granted Patent US 10,943,334
Granted Patent B2
US 10,943,334 · App. 16/520,893 · Granted Mar 9, 2021

Method and system for representation of vehicle surroundings with reduced artefacts

Inventors: Markus Friebe (Gefrees, DE); Sukumar Srikanth (Bangalore, IN); Pavannag Prabhakar (Bangalore, IN)
Assignee: Continental Automotive GmbH
G06T5/005G06K9/00791G06T5/50G06T7/80H04N5/2253H04N5/23267H04N5/247H04N5/2628B60R1/00B60R2300/105B60R2300/30G06T2207/30252
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,943,334
App. No.
16/520,893
Granted
Mar 9, 2021
Kind
B2
Abstract

A system is configured to repair artefacts in image data when representing vehicle surroundings. This involves: detecting an artefact-comprising area of a vehicle camera for a first time point, creating a reconstructed version of the artefact-comprising area from image data of an adjacent image area, determining a sample point of the artefact-comprising area, transforming the sample point to a new location predicted for a second time point, determining whether the new location still lies within the artefact-comprising area, and if so updating the artefact-comprising area in the image for the second time point by using motion-compensated image data from the reconstructed version of the sample point from the first time point.

Claims (29)

1. A method of representing a vehicle surroundings in a moving vehicle, comprising steps:

with a vehicle camera of the moving vehicle, capturing a first image of the vehicle surroundings at a first time point and a second image of the vehicle surroundings at a second time point,

detecting an artefact-comprising area in the first image captured at the first time point, wherein the artefact-comprising area in the first image depicts an artefact of the vehicle camera, wherein the artefact is caused by soil or dirt present on a lens of the vehicle camera,

creating a reconstructed version of the artefact-comprising area in the first image captured at the first time point by using image data that comprises texture data of an adjacent image area that is adjacent to the artefact-comprising area in the first image captured at the first time point,

determining a sample point of the artefact-comprising area in the first image captured at the first time point,

transforming the sample point to a new location of the sample point predicted for the second time point based on a motion of the vehicle,

determining whether the new location still lies within the artefact-comprising area, and

when the new location still lies within the artefact-comprising area, updating the artefact-comprising area in the second image captured at the second time point by using reconstructed image data of the sample point in the reconstructed version of the artefact-comprising area in the first image captured at the first time point and motion-compensating the reconstructed image data to the new location for the second time point based on the motion of the vehicle.

2. The method according to claim 1 , further comprising, when the new location lies outside the artefact-comprising area, rendering the artefact-comprising area in the second image captured at the second time point by using image data of the sample point of the first image captured at the first time point.

3. The method according to claim 2 , further comprising determining, by a motion vector obtained from vehicle motion data regarding the motion of the vehicle, which particular vehicle camera of a plurality of vehicle cameras of the vehicle having different viewing areas captured the first image at the first time point.

4. The method according to claim 1 , wherein determining the sample point comprises determining a world point of the sample point in a real world space.

5. The method according to claim 1 , further comprising obtaining motion data regarding the motion of the vehicle, for performing the motion compensation, by an odometry system of the vehicle.

6. The method according to claim 4 , wherein the world point is determined based on calibration data of the vehicle camera.

7. The method according to claim 1 , further comprising iteratively repeating the step of the determining of the sample point respectively for plural sample points until all sample points of the artefact-comprising area have been determined.

8. A system for representing a vehicle surroundings in a moving vehicle, comprising:

a vehicle camera of the moving vehicle configured to capture a first image of the vehicle surroundings at a first time point and a second image of the vehicle surroundings at a second time point,

a memory configured to store the first image captured at the first time point and the second image captured at the second time point, and

a data processing unit configured to:

detect an artefact-comprising area in the first image captured at the first time point, wherein the artefact-comprising area in the first image depicts an artefact of the vehicle camera, wherein the artefact is caused by soil or dirt present on a lens of the vehicle camera,

create a reconstructed version of the artefact-comprising area in the first image captured at the first time point by using image data that comprises texture data of an adjacent image area that is adjacent to the artefact-comprising area in the first image captured at the first time point,

determine a sample point of the artefact-comprising area in the first image captured at the first time point,

transform the sample point to a new location of the sample point predicted for the second time point based on a motion of the vehicle,

determine whether the new location still lies within the artefact-comprising area, and

when the new location still lies within the artefact-comprising area, update the artefact-comprising area in the second image captured at the second time point by using reconstructed image data of the sample point in the reconstructed version of the artefact-comprising area in the first image captured at the first time point and motion-compensating the reconstructed image data to the new location for the second time point based on the motion of the vehicle.

9. The system according to claim 8 , wherein the data processing unit is further configured, when the new location lies outside the artefact-comprising area, to render the artefact-comprising area in the second image captured at the second time point by using image data of the sample point of the first image captured at the first time point.

10. The method according to claim 7 , further comprising iteratively repeating the steps of the transforming of the sample point to the new location, the determining of whether the new location lies within the artefact-comprising area, and the updating of the artefact-comprising area, respectively for all of the sample points of the artefact-comprising area.

11. The method according to claim 1 , wherein the artefact comprises a temporary static artefact of the vehicle camera caused by the soil or the dirt present on the lens of the vehicle camera.

12. The method according to claim 1 , wherein the artefact comprises a static artefact caused by the soil or the dirt present on the lens of the vehicle camera.

13. The method according to claim 1 , wherein the artefact-comprising area is an area of an imager of the vehicle camera, at which area of the imager the soil or the dirt present on the lens of the vehicle camera is imaged.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 7, 2025
From: CONTINENTAL AUTOMOTIVE GMBH
To: CONTINENTAL AUTONOMOUS MOBILITY GERMANY GMBH
Reel/Frame 070438/0643 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 1, 2019
From: FRIEBE, MARKUS; SRIKANTH, SUKUMAR; PRABHAKAR, PAVANNAG
To: CONTINENTAL AUTOMOTIVE GMBH
Reel/Frame 049931/0667 →