Granted Patent B2

US 12,322,189 · App. 17/619,159 · Granted Jun 3, 2025

Inventors: Fabian Hüger (Wolfenbüttel, DE); Peter Schlicht (Wolfsburg, DE); Nico Maurice Schmidt (Berlin, DE); Feix Assion (Berlin, DE); Florens Fabian Gressner (Berlin, DE)

Assignees: VOLKSWAGEN AKTIENGESELLSCHAFT; DSPACE GMBH

G06V20/58G06V10/764G06V10/82

Abstract

Claims (72)

1. A method for generating disturbed input data for a neural network for analyzing sensor data of a driver assistance system, which sensor data represent a plurality of digital images, the method comprising:

generating a first metric, which indicates a magnitude of a change in the sensor data, wherein two digital images of the sensor data are compared and an image distances value is provided as part of the first metric;

detecting and classifying one or more objects;

generating a second metric, which indicates where a disturbance of the sensor data is directed to, wherein the second metric is directed at a change of a first object of a specific class;

wherein the first metric and the second metric indicate change of the one or more digital images;

generating an optimization problem from a combination of the first metric and the second metric, wherein the optimization problem comprises a loss function for a neural network which comprises as parameters a disturbance parameter and an image resulting from the disturbance according to the second metric, wherein in the optimization problem the minimum of the disturbance parameter is found under the condition that the extent of the change in the generated image relative to the initial image according to the first metric is below a predefined value;

solving the optimization problem using at least one solution algorithm, wherein the solution indicates a target disturbance of the input data; and

generating, using the target disturbance, disturbed input data comprising changed digital images for the neural network.

2. The method of claim 1 , wherein the second metric is directed at a change in the classification of objects.

3. The method of claim 1 , wherein the second metric is directed at a disappearance of objects.

4. The method of claim 1 , wherein the disturbances that are described by one or more of the first and the second metric are naturally occurring disturbances.

5. The method of claim 4 , wherein

one or more of the first and the second metrics are stored in a database; and

a data set on a naturally occurring disturbance that is measured with one or more of the first and the second metric is loaded from the database.

6. The method of claim 1 , wherein

a third metric is defined, which indicates what kind of sensor data a disturbance is applied to; and

the optimization problem is generated from a combination of at least two metrics of the first, the second, and the third metric.

7. The method of claim 6 , wherein the third metric relates to all sensor data.

8. The method of claim 6 , wherein the third metric relates only to a subset of the sensor data.

9. The method of claim 6 , wherein the third metric describes only sensor data that contain a specific object.

10. The method of claim 1 , wherein the solution algorithm comprises iterative methods using the gradients of the neural network for determining the change directions.

11. A method for checking the robustness of a neural network for analyzing sensor data against disturbed input data, which sensor data represent a plurality of digital images, comprising:

providing a neural network with an associated parameter set;

generating training data using an example sensor data set;

generating a first analysis of the example sensor data set on the basis of the training data using the neural network;

generating disturbed input data as training data for the example sensor data set using the method of claim 1 ;

generating a second analysis of the example sensor data set on the basis of the disturbed input data using the neural network;

comparing the first and second analysis; and

determining a robustness value depending on the result of the comparison of the first and second analysis.

12. A method for improving a parameter set of a neural network for analyzing sensor data in relation to disturbed input data, which sensor data represent a plurality of digital images, comprising:

providing a neural network with an associated parameter set;

generating training data using an example sensor data set;

generating a first analysis of the example sensor data set on the basis of the training data using the neural network;

generating disturbed input data as training data for the example sensor data set using the method of claim 1 ;

generating a second analysis of the example sensor data set on the basis of the disturbed input data using the neural network;

comparing the first and second analysis; and

generating an improved parameter set for the neural network on the basis of the result of the comparison of the first and second analysis.

13. A method for generating disturbed input data for a neural network for analyzing sensor data of a driver assistance system, which sensor data represent a plurality of digital images, comprising:

generating a first quantity of first metrics, which each differently indicate how a magnitude of a change in the sensor data is measured, wherein two digital images of the sensor data are compared and an image distances value is provided as part of the first metric;

detecting and classifying one or more objects;

generating a second quantity of second metrics, which each differently indicate where a disturbance of the sensor data is directed to, wherein the second metric is directed at a change of a first object of a specific class;

selecting any combination of a first metric from the first quantity and a second metric from the second quantity;

generating an optimization problem from the chosen combination of first and second metric, wherein the optimization problem comprises a loss function for a neural network which comprises as parameters a disturbance parameter and an image resulting from the disturbance according to the second metric, wherein in the optimization problem the minimum of the disturbance parameter is found under the condition that the extent of the change in the generated image relative to the initial image according to the first metric is below a predefined value;

solving the optimization problem using at least one solution algorithm, wherein the solution indicates a target disturbance of the input data; and

generating, using the target disturbance, disturbed input data from sensor data for the neural network.

14. The method of claim 13 , wherein the first quantity comprises at least two, in particular at least five, different first metrics.

15. The method of claim 13 , wherein the second quantity comprises at least two, in particular at least five, different second metrics.

16. The method of claim 13 , comprising:

using a third metric, which indicates what kind of sensor data a disturbance is applied to;

selecting any combination of a first metric of the first quantity, a second metric of the second quantity, and the third metric; and

generating an optimization problem from the chosen combination of the first, second, and third metric.

17. The method of claim 13 , comprising:

using a solution algorithm quantity that contains multiple solution algorithms that each solve the optimization problem differently in order to generate different target disturbances of the input data; and

selecting any solution algorithm of the solution algorithm quantity in order to generate disturbed input data from sensor data for the neural network.

18. A generator system for generating disturbed input data for a neural network for analyzing sensor data of a driver assistance system, which sensor data represent a plurality of digital images, the generator system comprising:

a first metric, which indicates a magnitude of a change in the sensor data, wherein two digital images of the sensor data are compared and an image distances value is provided as part of the first metric;

a second metric, which indicates where a disturbance of the sensor data is directed to, wherein the second metric is directed at a change of a first object of a specific class, wherein one or more objects are detected and classified;

wherein the first metric and the second metric indicate change of the one or more digital images;

one or more processors, which are configured to:

generate an optimization problem from the first and the second metric, wherein the optimization problem comprises a loss function for a neural network which comprises as parameters a disturbance parameter and an image resulting from the disturbance according to the second metric, wherein in the optimization problem the minimum of the disturbance parameter is found under the condition that the extent of the change in the generated image relative to the initial image according to the first metric is below a predefined value;

to solve the optimization problem by means of at least one solution algorithm, wherein the solution indicates a target disturbance of the input data from sensor data; and

using the target disturbance, to generate disturbed input data comprising changed digital images for a neural network.

19. The generator system of claim 18 , comprising

a third metric unit with a third metric, which indicates what kind of sensor data the disturbance is applied to,

wherein the processing unit is also coupled to the third metric unit and is designed to generate the optimization problem from at least two metrics of the first, the second, and the third metric.

20. A generator system for generating disturbed input data for a neural network for analyzing sensor data of a driver assistance system, which sensor data represent a plurality of digital images, with

a first quantity of first metrics, which each differently indicate how a magnitude of a change in the sensor data is measured, wherein two digital images of the sensor data are compared and an image distances value is provided as part of the first metric;

a second quantity of second metrics, which each differently indicate where a disturbance of the sensor data is directed to, wherein the second metric is directed at a change of a first object of a specific class, wherein one or more objects are detected and classified;

one or more processors, which are configured to:

generate an optimization problem from any combination of a first metric of the first quantity and a second metric of the second quantity, wherein the optimization problem comprises a loss function for a neural network which comprises as parameters a disturbance parameter and an image resulting from the disturbance according to the second metric, wherein in the optimization problem the minimum of the disturbance parameter is found under the condition that the extent of the change in the generated image relative to the initial image according to the first metric is below a predefined value;

to solve the optimization problem by means of at least one solution algorithm, wherein the solution indicates a target disturbance of the input data from sensor data, and

to generate disturbed input data from sensor data for a neural network by means of the target disturbance.

Assignments (3)

Priority Claims (1)

Continuity (1)

References Cited (11)

Cited By (1)

US 12,731,025