Machine learning methods for evaluating vehicle conditions
Techniques for using a trained machine learning (ML) model to detect presence of vehicle defects from audio acquired at least in part during operation of an engine of a vehicle. The techniques include using at least one computer hardware processor to perform: obtaining, via at least one communication network, a first audio recording that was acquired, using at least one acoustic sensor, at least in part during operation of the engine; processing the first audio recording using the trained ML model to detect, from the first audio recording, presence of at least one vehicle defect, the processing comprising: generating an audio waveform from the first audio recording, generating a two-dimensional (2D) representation of the audio waveform, and processing the audio waveform and the 2D representation of the audio waveform using the trained ML model to obtain output indicative of presence or absence of the at least one vehicle defect.
1 . A method for using a trained machine learning (ML) model to detect presence of vehicle defects from audio acquired at least in part during operation of an engine of a vehicle, the method comprising:
using at least one computer hardware processor to perform:
obtaining, via at least one communication network, a first audio recording that was acquired, using at least one acoustic sensor, at least in part during operation of the engine;
processing the first audio recording using the trained ML model to detect, from the first audio recording, presence of at least one vehicle defect, the processing comprising:
generating an audio waveform from the first audio recording,
generating a two-dimensional (2D) representation of the audio waveform, and
processing the audio waveform using the trained ML model and processing the 2D representation of the audio waveform using the trained ML model to obtain output indicative of presence or absence of the at least one vehicle defect, wherein the trained ML model comprises:
a first neural network portion comprising a plurality of one-dimensional (1D) convolutional layers configured to process the audio waveform;
a second neural network portion comprising a plurality of 2D convolutional layers configured to process the 2D representation of the audio waveform; and
a fusion neural network portion comprising one or more fully connected layers configured to combine outputs produced by the first neural network portion and the second neural network portion to obtain the output indicative of the presence or absence of the at least one vehicle defect.
2 . The method of claim 1 , wherein the audio recording comprises at least a first waveform for at least a first audio channel, and wherein generating the audio waveform from the first audio recording comprises:
resampling the first waveform to a target frequency to obtain a resampled waveform;
normalizing the resampled waveform by subtracting its mean and dividing by its standard deviation to obtain a normalized waveform; and
clipping the normalized waveform to a target maximum to obtain the audio waveform.
3 . The method of claim 1 , wherein the audio waveform is between 5 and 45 seconds long and wherein the frequency of the audio waveform is between 8 and 45 KHz.
4 . The method of claim 1 , wherein generating the two-dimensional (2D) representation of the audio waveform comprises generating a time-frequency representation of the audio waveform.
5 . The method of claim 4 , wherein generating the time-frequency representation of the audio waveform comprises using a short-time Fourier transform, a wavelet transform, a Gabor transform, or a chirplet transform to generate the time-frequency representation.
6 . The method of claim 5 , wherein generating the time-frequency representation of the audio waveform comprises generating a Mel-scale log spectrogram from the audio waveform.
7 . The method of claim 1 , further comprising:
obtaining, via the at least one communication network, metadata indicating one or more properties of the vehicle,
wherein using the trained ML model to detect the presence of the at least one vehicle defect further comprises generating metadata features from the metadata, and
wherein processing the audio waveform and the 2D representation of the audio waveform comprises processing the audio waveform, the 2D representation of the audio waveform, and the metadata features using the trained ML model to obtain the output indicative of the presence or absence of the at least one vehicle defect.
8 . The method of claim 7 , wherein the properties of the vehicle are selected from the group consisting of: a reading of the vehicle's odometer, a model of the vehicle, a make of the vehicle, an age of the vehicle, a type of drivetrain in the vehicle, a type of transmission in the vehicle, a measure of displacement of the engine, a fuel type for the vehicle, an indication of whether on-board diagnostics (OBD) codes could be obtained from the vehicle, a number of incomplete readiness monitors reported by the OBD scanner, one or more BlackBook-reported engine properties, a list of one or more OBD codes, location of the vehicle, information about weather at the location of the vehicle, and information about a seller of the vehicle.
9 . The method of claim 7 , wherein the metadata comprises text indicating at least one of the one or more properties, and generating the metadata features from the metadata comprises generating a numeric representation of the text.
10 . The method of claim 1 , wherein the output is indicative of the presence or absence of abnormal internal engine noise, timing chain noise, engine accessory noise, and/or exhaust noise.
11 . The method of claim 1 , further comprising:
obtaining, via the at least one communication network, metadata indicating one or more properties of the vehicle,
wherein using the trained ML model to detect the presence of the at least one vehicle defect further comprises generating metadata features from the metadata,
wherein processing the audio waveform and the two-dimensional representation of the audio waveform comprises processing the audio waveform, the two-dimensional representation of the audio waveform, and the metadata features, using the trained ML model to obtain output indicative of presence of the at least one vehicle defect,
wherein the trained ML model further comprises a third neural network portion comprising one or more fully connected layers configured to process the metadata features, and
wherein the one or more fully connected layers of the fusion neural network are configured to combine outputs produced by the first neural network portion, the second neural network portion, and the third neural network portion to obtain the output indicative of the presence or absence of the at least one vehicle defect.
12 . The method of claim 11 ,
wherein the trained ML model has at least one million parameters, and
wherein processing the first audio recording using the trained ML model to detect the presence of the at least one vehicle defect comprises computing the output using values of the at least one million parameters, the audio waveform, and the 2D representation of the audio waveform.
13 . The method of claim 1 , further comprising:
acquiring, using the at least one acoustic sensor, the first audio recording at least in part during operation of the engine.
14 . The method of claim 1 , further comprising:
determining, based on the output, that the at least one vehicle defect was detected using the first audio recording, and
generating an electronic vehicle condition report indicating that the at least one vehicle defect was detected using the first audio recording and a measure of confidence in that detection.
15 . The method of claim 14 , further comprising:
transmitting the electronic vehicle condition report, via the at least one communication network, to a remote device of an inspector of the vehicle.
16 . The method of claim 15 , further comprising:
receiving a second audio recording, via the at least one communication network, from the remote device of the inspector of the vehicle, the second audio recording being acquired after transmission of the electronic vehicle condition report and using the at least one acoustic sensor at least in part during operation of the engine; and
processing the second audio recording using the trained ML model to detect, from the second audio recording, presence of the at least one vehicle defect, the processing comprising:
generating a second audio waveform from the second audio recording,
generating a second two-dimensional (2D) representation of the second audio waveform, and
processing the second audio waveform and the second 2D representation of the audio waveform using the trained ML model to obtain second output indicative of presence or absence of the at least one vehicle defect.
17 . The method of claim 1 ,
wherein obtaining the first audio recording comprises receiving the first audio recording from a mobile device, via the at least one communication network, by at least one computing device at a location remote from a location of the mobile device,
wherein the processing is performed by the at least one computing device, and
wherein the mobile device comprises a smart phone or a mobile vehicle diagnostic device.
18 . A system, comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing processor executable instructions that when executed by the at least one computer hardware processor perform a method for using a trained machine learning (ML) model to detect presence of vehicle defects from audio acquired at least in part during operation of an engine of a vehicle, the method comprising:
obtaining, via at least one communication network, a first audio recording that was acquired, using at least one acoustic sensor, at least in part during operation of the engine;
processing the first audio recording using the trained ML model to detect, from the first audio recording, presence of at least one vehicle defect, the processing comprising:
generating an audio waveform from the first audio recording,
generating a two-dimensional (2D) representation of the audio waveform, and
processing the audio waveform using the trained ML model and processing the 2D representation of the audio waveform using the trained ML model to obtain output indicative of presence or absence of the at least one vehicle defect, wherein the trained ML model comprises:
a first neural network portion comprising a plurality of one-dimensional (1D) convolutional layers configured to process the audio waveform;
a second neural network portion comprising a plurality of 2D convolutional layers configured to process the 2D representation of the audio waveform; and
a fusion neural network portion comprising one or more fully connected layers configured to combine outputs produced by the first neural network portion and the second neural network portion to obtain the output indicative of the presence or absence of the at least one vehicle defect.
19 . At least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by a at least one computer hardware processor, cause the at least one processor to perform a method for using a trained machine learning (ML) model to detect presence of vehicle defects from audio acquired at least in part during operation of an engine of a vehicle, the method comprising:
obtaining, via at least one communication network, a first audio recording that was acquired, using at least one acoustic sensor, at least in part during operation of the engine;
processing the first audio recording using the trained ML model to detect, from the first audio recording, presence of at least one vehicle defect, the processing comprising:
generating an audio waveform from the first audio recording,
generating a two-dimensional (2D) representation of the audio waveform, and
processing the audio waveform using the trained ML model and processing the 2D representation of the audio waveform using the trained ML model to obtain output indicative of presence or absence of the at least one vehicle defect, wherein the trained ML model comprises:
a first neural network portion comprising a plurality of one-dimensional (1D) convolutional layers configured to process the audio waveform;
a second neural network portion comprising a plurality of 2D convolutional layers configured to process the 2D representation of the audio waveform; and
a fusion neural network portion comprising one or more fully connected layers configured to combine outputs produced by the first neural network portion and the second neural network portion to obtain the output indicative of the presence or absence of the at least one vehicle defect.