Apparatus and method for analyzing road surface condition based on vehicle noise

1 . A method for analyzing a road surface condition, comprising:

receiving, by a signal processing unit, an image signal of a vehicle and an audio signal that is a noise generated on a road surface by driving of the vehicle;

generating, by the signal processing unit, an audio-of-interest signal by detecting an audio signal of a frequency-of-interest band from the received audio signal; and

recognizing, by a recognition unit, the road surface condition by analyzing the image signal and the audio-of-interest signal through a learned vehicle type recognition model and one of a plurality of learned state recognition models,

wherein the method further comprises:

before receiving the audio signal,

preparing, by a learning unit, training data including a training audio-of-interest signal and a label corresponding to the training audio-of-interest signal;

inputting, by the learning unit, the training audio-of-interest signal to a state recognition model that does not complete learning;

calculating, by the state recognition model, a predictive value for training indicating a predicted probability for each of a plurality of road surface conditions, by performing a plurality of operations in which unlearned weights between layers are applied;

calculating, by the learning unit, a classification loss indicating a difference between the predictive value for training and the label; and

performing, by the learning unit, optimization of updating a weight of the state recognition model to minimize the classification loss.

2 . The method of claim 1 , wherein recognizing the road surface condition includes:

recognizing, by the recognition unit, a vehicle type by analyzing the image signal through the learned vehicle type recognition model;

selecting, by the recognition unit, a learned state recognition model corresponding to the recognized vehicle type from among the plurality of state recognition models learned for respective vehicle types;

computing, by the recognition unit, a predictive value, which is a probability predicted for each of a plurality of road surface conditions, by analyzing the audio-of-interest signal through the selected learned state recognition model; and

recognizing, by the recognition unit, the road surface condition according to the predictive value.

3 . The method of claim 2 , wherein recognizing the vehicle type includes:

inputting, by the recognition unit, the image signal to the learned vehicle type recognition model;

calculating, by the vehicle type recognition model, an output value including a bounding box indicating an area occupied by a vehicle object in the image signal and a confidence indicating a probability that the vehicle object in the bounding box belongs to each of a plurality of vehicle types by performing an operation in which a plurality of learned layer weights are applied to the image signal; and

recognizing, by the recognition unit, the vehicle type of the vehicle object according to the confidence.

4 . The method of claim 1 , further comprising:

before receiving the audio signal,

preparing, by a learning unit, training data including a training image signal and a label, the training image signal being obtained by photographing a vehicle, and the label including a ground-truth indicating an area occupied by a vehicle object contained in the training image signal and a vehicle type vector indicating the vehicle type of the vehicle object in the ground-truth;

inputting, by the learning unit, the training image signal to a vehicle type recognition model that does not complete learning;

calculating, by the vehicle type recognition model, an output value for training including a bounding box indicating an area occupied by a vehicle object in the image signal and a confidence indicating a probability that the vehicle object in the bounding box belongs to each of a plurality of vehicle types by performing a plurality of operations in which unlearned weights between layers are applied to the training image signal;

calculating, by the learning unit, a composite loss including a coordinate loss indicating a coordinate difference between the bounding box and the ground-truth and a division loss indicating a difference between the confidence and the vehicle type vector; and

performing, by the learning unit, optimization of updating a weight of the vehicle type recognition model to minimize the composite loss.

5 . An apparatus for analyzing a road surface condition, comprising:

a signal processing unit configured to receive an image signal of a vehicle and an audio signal that is a noise generated on a road surface by driving of the vehicle, and to generate an audio-of-interest signal by detecting an audio signal of a frequency-of-interest band from the received audio signal;

a recognition unit configured to recognize the road surface condition by analyzing the image signal and the audio-of-interest signal through a learned vehicle type recognition model and one of a plurality of learned state recognition models; and

a learning unit configured to:

prepare training data including a training audio-of-interest signal and a label corresponding to the training audio-of-interest signal;

input the training audio-of-interest signal to a state recognition model that does not complete learning;

when the state recognition model calculates a predictive value for training indicating a predicted probability for each of a plurality of road surface conditions, by performing a plurality of operations in which unlearned weights between layers are applied:

calculate a classification loss indicating a difference between the predictive value for training and the label; and

perform optimization of updating a weight of the state recognition model to minimize the classification loss.

6 . The apparatus of claim 5 , wherein the recognition unit is configured to:

recognize a vehicle type by analyzing the image signal through the learned vehicle type recognition model;

select a learned state recognition model corresponding to the recognized vehicle type from among the plurality of state recognition models learned for respective vehicle types;

compute a predictive value, which is a probability predicted for each of a plurality of road surface conditions, by analyzing the audio-of-interest signal through the selected learned state recognition model; and

recognize the road surface condition according to the predictive value.

7 . The apparatus of claim 6 , wherein the recognition unit is configured to:

input the image signal to the learned vehicle type recognition model; and

when the vehicle type recognition model calculates an output value including a bounding box indicating an area occupied by a vehicle object in the image signal and a confidence indicating a probability that the vehicle object in the bounding box belongs to each of a plurality of vehicle types by performing an operation in which a plurality of learned layer weights are applied to the image signal,

recognize the vehicle type of the vehicle object according to the confidence.

8 . The apparatus of claim 5 ,

wherein the learning unit is further configured to:

prepare training data including a training image signal and a label, the training image signal being obtained by photographing a vehicle, and the label including a ground-truth indicating an area occupied by a vehicle object contained in the training image signal and a vehicle type vector indicating the vehicle type of the vehicle object in the ground-truth;

input the training image signal to a vehicle type recognition model that does not complete learning;

when the vehicle type recognition model calculates an output value for training including a bounding box indicating an area occupied by a vehicle object in the image signal and a confidence indicating a probability that the vehicle object in the bounding box belongs to each of a plurality of vehicle types by performing a plurality of operations in which unlearned weights between layers are applied to the training image signal,

calculate a composite loss including a coordinate loss indicating a coordinate difference between the bounding box and the ground-truth and a division loss indicating a difference between the confidence and the vehicle type vector; and

perform optimization of updating a weight of the vehicle type recognition model to minimize the composite loss.