Granted Patent B2

US 12,333,827 · App. 18/002,354 · Granted Jun 17, 2025

Inventor: Sharath Yadav Doddamane Hemantharaja (Karnataka, IN)

Assignee: HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED

G06V20/582G06T7/55G06T7/74G06V10/82G06T2207/20084G06T2207/30244G06T2207/30252

Abstract

Claims (59)

1. A system for a vehicle, comprising:

an image sensor;

a processor; and

a non-transitory storage device storing instructions executable by the processor to:

capture, via the image sensor, an image of an environment of the vehicle;

detect an object in the image of the environment of the vehicle;

determine whether the object is a projected image of the object;

selectively control one or more vehicle systems of the vehicle to perform one or more actions wherein the object is not the projected image; and

authenticate the traffic sign board as a valid traffic sign board to determine that the object is not the projected image wherein the estimated depth is zero.

2. The system of claim 1 , wherein the non-transitory storage device further stores instructions executable by the processor to not selectively control the one or more vehicle systems of the vehicle to perform the one or more actions wherein the object is the projected image.

3. The system of claim 1 , wherein the non-transitory storage device further stores instructions executable by the processor to: estimate a depth of the object in the image, and determine that the object is not the projected image of the object wherein the estimated depth is non-zero.

4. The system of claim 3 , wherein the storage device further stores instructions executable by the processor to: classify the object in the image as a traffic sign board, and determine that the object is not the projected image of the object wherein the estimated depth is zero.

5. The system of claim 3 , wherein the non-transitory storage device stores a depth estimation neural network configured to accept the image as input and generate the estimated depth as output, wherein the non-transitory storage device further stores instructions executable by the processor to: input the image to the depth estimation neural network to estimate the depth of the object in the image.

6. The system of claim 5 , wherein the non-transitory storage device stores a relative camera pose neural network trained to accept two consecutive images from the image sensor, the two consecutive images including the image and a second image, and output a relative camera pose between the two consecutive images.

7. The system of claim 6 , wherein the non-transitory storage device further stores instructions executable by the processor to:

estimate, with the depth estimation neural network, a second depth of the second image;

estimate, with the relative camera pose neural network, the relative camera pose between the two consecutive images;

transform the depth to a warped depth based on the relative camera pose;

interpolate the second depth to an interpolated depth based on the relative camera pose; and

update the depth estimation neural network based on a depth inconsistency between the warped depth and the interpolated depth.

8. The system of claim 1 , wherein the non-transitory storage device further stores instructions executable by the processor to output a notification, to an operator of the vehicle via a user interface, indicating that the object is a projected object wherein the object is the projected image.

9. A method for a vehicle, comprising:

capturing, via an image sensor, an image of an environment of the vehicle;

detecting, with a first neural network, an object in the image;

determining, with a second neural network, whether the object is a projected image of the object;

selectively controlling one or more vehicle systems of the vehicle to perform one or more actions wherein the object is not the projected image;

not selectively controlling the one or more vehicle systems of the vehicle to perform the one or more actions wherein the object is the projected image; and

outputting a notification, to an operator of the vehicle via a user interface, indicating that the object is a projected object wherein the object is the projected image.

10. The method of claim 9 , wherein determining whether the object is the projected image of the object comprises:

estimating, with the second neural network, a depth of the object in the image; and

determining that the object is not the projected image wherein the depth is non-zero.

11. The method of claim 10 , further comprising classifying, with the first neural network, a type of the object, wherein determining whether the object is the projected image of the object further comprises determining that the object is the projected image wherein the depth is zero and the type of the object is not classified as a traffic sign board.

12. The method of claim 11 , wherein determining whether the object is the projected image of the object further comprises determining that the object is not the projected image wherein the depth is zero and the type of the object is classified as the traffic sign board.

13. The method of claim 10 , further comprising:

capturing, with the image sensor, a second image of the environment of the vehicle;

estimating, with the second neural network, depth of the second image;

estimating, with a third neural network, a relative camera pose between the image and the second image;

transforming the depth of the image and the depths of the second image to a same three-dimensional structure based on the relative camera pose;

determining a depth inconsistency between the transformed depth of the image and the transformed depths of the second image; and

updating the second neural network based on the depth inconsistency.

14. The method of claim 13 , further comprising determining, with the updated second neural network for a subsequent image acquired via the image sensor, whether an object in the subsequent image is a projected image.

15. A method for a vehicle, comprising:

capturing, via an image sensor, an image of an environment of the vehicle;

detecting, with a first neural network, an object in the image;

estimating, with a second neural network, a depth of the object in the image;

determining, based on the estimated depth of the object, whether the object is a projected image of the object;

classifying, with the first neural network, the object as a traffic sign board, and authenticating a validity of the object as the traffic sign board wherein the estimated depth is zero;

selectively controlling one or more vehicle systems of the vehicle to perform one or more actions wherein the object is not the projected image; and

not selectively controlling the one or more vehicle systems of the vehicle to perform the one or more actions wherein the object is the projected image.

16. The method of claim 15 , further comprising performing unsupervised training of the second neural network to update the second neural network in near real-time based on the estimated depth of the object in the image.

17. The method of claim 16 , wherein performing unsupervised training of the second neural network to update the second neural network in real-time based on the estimated depth of the object in the image comprises:

capturing, via the image sensor, a second image of the environment of the vehicle;

estimating, with the second neural network, depth of the second image;

determining a depth inconsistency between depth of the object in the image and the depths of the second image; and

updating the second neural network based on the depth inconsistency.

18. The method of claim 17 , further comprising:

estimating, with a third neural network, a relative camera pose between the image and the second image; and

transforming the depth of the image and the depth of the second image to a same three-dimensional structure based on the relative camera pose,

wherein determining the depth inconsistency based on the depth of the object in the image and the depths of the second image comprises determining the depth inconsistency between the transformed depth of the object in the image and the transformed depths of the second image.

Assignments (1)

Continuity (1)

References Cited (19)