Audio-based technique to sense and detect the road condition for autonomous driving vehicles

1. A computer-implemented method for operating an autonomous driving vehicle (ADV), the method comprising:

receiving sound signals by one or more microphones disposed at the ADV, comprising:

detecting the sound signals from a friction between each wheel of the ADV and a surface of a road, at least two microphones of the one or more microphones being mounted at a hub of each wheel of the ADV, the sound signals from the at least two microphones having a reduced noise by applying a noise reducing algorithm;

analyzing the sound signals to extract a feature representing the road on which the ADV is driving, comprising:

extracting a power spectrogram of the road based on the sound signals, the power spectrogram representing a power level of a signal of the sound signals in each frequency band over time and corresponding to a road condition of the road;

determining the road condition of the road based on the extracted feature, comprising:

determining a pavement condition of the road based on the extracted feature, the pavement condition including an asphalt condition, a rocky condition, or a sandy condition;

determining a depth of a covering of the road based on the extracted feature, the depth of the covering of the road including a depth of water, snow, or sand on the road; and

determining a friction coefficient of the road based on at least one of a surface type, the pavement condition, or the depth of the covering of the road by using regression models in a machine learning algorithm;

performing a path planning and speed planning to generate a trajectory based on the road condition, comprising:

performing at least one of changing a path or lowering a speed of the ADV, in response to that the surface type is a wet surface and the friction coefficient is less than a predetermined threshold; and

controlling the ADV to drive autonomously according to the generated trajectory.

2. The method of claim 1 , wherein the one or more microphones are disposed inside tires or wheel hubs.

3. The method of claim 1 , wherein the one or more microphones are disposed at a bottom panel of the ADV.

4. The method of claim 1 , wherein analyzing the sound signals to extract the feature of the road comprises analyzing the sound signals to extract a frequency feature representing a roughness of the surface of the road.

5. The method of claim 1 , wherein determining the road condition comprises determining the road condition using the machine learning algorithm or a deep learning algorithm.

6. The method of claim 1 , wherein determining the road condition based on the extracted feature comprises determining at least one of a surface type of the road or the pavement condition based on the extracted feature.

7. The method of claim 1 , wherein determining the road condition based on the extracted feature comprises determining the friction coefficient representing a friction between one or more tires of the ADV and the surface of the road based on the extracted feature.

8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of operating an autonomous driving vehicle (ADV), the operations comprising:

receiving sound signals by one or more microphones disposed at the ADV, comprising:

detecting the sound signals from a friction between each wheel of the ADV and a surface of a road, at least two microphones of the one or more microphones being mounted at a hub of each wheel of the ADV, the sound signals from the at least two microphones having a reduced noise by applying a noise reducing algorithm;

analyzing the sound signals to extract a feature representing the road on which the ADV is driving, comprising:

extracting a power spectrogram of the road based on the sound signals, the power spectrogram representing a power level of a signal of the sound signals in each frequency band over time and corresponding to a road condition of the road;

determining the road condition of the road based on the extracted feature, comprising:

determining a pavement condition of the road based on the extracted feature, the pavement condition including an asphalt condition, a rocky condition, or a sandy condition;

determining a depth of a covering of the road based on the extracted feature, the depth of the covering of the road including a depth of water, snow, or sand on the road; and

determining a friction coefficient of the road based on at least one of a surface type, the pavement condition, or the depth of the covering of the road by using regression models in a machine learning algorithm;

performing a path planning and speed planning to generate a trajectory based on the road condition, comprising:

performing at least one of changing a path or lowering a speed of the ADV, in response to that the surface type is a wet surface and the friction coefficient is less than a predetermined threshold; and

controlling the ADV to drive autonomously according to the generated trajectory.

9. The machine-readable medium of claim 8 , wherein the one or more microphones are disposed inside tires or wheel hubs.

10. The machine-readable medium of claim 8 , wherein the one or more microphones are disposed at a bottom panel of the ADV.

11. The machine-readable medium of claim 8 , wherein analyzing the sound signals to extract the feature of the road comprises analyzing the sound signals to extract a frequency feature representing a roughness of the surface of the road.

12. The machine-readable medium of claim 8 , wherein determining the road condition comprises determining the road condition using the machine learning algorithm or a deep learning algorithm.

13. The machine-readable medium of claim 8 , wherein determining the road condition based on the extracted feature comprises determining at least one of a surface type of the road or the pavement condition based on the extracted feature.

14. The machine-readable medium of claim 8 , wherein determining the road condition based on the extracted feature comprises determining the friction coefficient representing a friction between one or more tires of the ADV and the surface of the road based on the extracted feature.

15. A data processing system, comprising:

a processor; and

a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations of operating an autonomous driving vehicle (ADV), the operations including

receiving sound signals by one or more microphones disposed at the ADV, comprising:

detecting the sound signals from a friction between each wheel of the ADV and a surface of a road, at least two microphones of the one or more microphones being mounted at a hub of each wheel of the ADV, the sound signals from the at least two microphones having a reduced noise by applying a noise reducing algorithm;

analyzing the sound signals to extract a feature representing the road on which the ADV is driving, comprising:

extracting a power spectrogram of the road based on the sound signals, the power spectrogram representing a power level of a signal of the sound signals in each frequency band over time and corresponding to a road condition of the road;

determining the road condition of the road based on the extracted feature, comprising:

determining a pavement condition of the road based on the extracted feature, the pavement condition including an asphalt condition, a rocky condition, or a sandy condition;

determining a depth of a covering of the road based on the extracted feature, the depth of the covering of the road including a depth of water, snow, or sand on the road; and

determining a friction coefficient of the road based on at least one of a surface type, the pavement condition, or the depth of the covering of the road;

performing a path planning and speed planning to generate a trajectory based on the road condition, comprising:

performing at least one of changing a path or lowering a speed of the ADV, in response to that the surface type is a wet surface and the friction coefficient is less than a predetermined threshold; and

controlling the ADV to drive autonomously according to the generated trajectory.

16. The system of claim 15 , wherein the one or more microphones are disposed inside tires or wheel hubs.