System and method for detecting and counting road anomalies
View Patent ↗A system and a non-transitory computer-readable storage medium for executing a method of detecting road anomalies includes obtaining visual data of a road from an end computing device and inputting the visual data to a machine learning (ML) model that learns to detect and classify at least one road anomaly such as a pothole, longitudinal crack, transverse crack, or alligator crack. The class and a bounding box of at least one road anomaly are output. Multiple post-detection features of the road anomaly, such as an object box area, intersection of neighboring object boxes, union of neighboring object boxes are determined. When the road anomaly is detected across multiple sequential frames, a number of skip frames is determined based on a model fidelity distance (MFD) and a vehicle speed.
1. A method of detecting road anomalies, the method comprising:
obtaining visual data of a road from an end computing device in a vehicle travelling along the road, wherein the visual data is a live feed captured using a camera mounted in or on the vehicle and sent to the end computing device;
inputting the visual data to a machine learning (ML) model in the end computing device that learns to detect and classify at least one road anomaly into a class including one of a pothole, a longitudinal crack, a transverse crack, or an alligator crack on the road;
outputting the class and a bounding box, created by the ML model, of the at least one road anomaly;
determining multiple post-detection features of the detected at least one road anomaly, wherein the determining multiple post-detection features includes determining that the at least one road anomaly is detected across multiple sequential frames;
when the at least one road anomaly is detected across multiple sequential frames, determining a number of skip frames based on a model fidelity distance (MFD) and a vehicle speed, wherein the MFD is a distance that the ML model can detect objects within a predetermined error;
determining that the detected at least one road anomaly is detected multiple times in each of the multiple sequential frames;
skipping frames based on the number of skip frames to a key frame; and
transmitting, via a roadside computing device installed proximate to the road, a notification to multiple end computing devices in a communication range of the roadside computing device, wherein the notification is an anomaly-specific notification that includes at least one of a message indicating a presence of the road anomaly, an image of the road anomaly, a location of the road anomaly, or a severity of the road anomaly.
2. The method of claim 1 , further comprising: assessing a severity of the road, wherein the severity is determined to be of a first value when an area of the bounding box exceeds a severity threshold area.
3. The method of claim 1 , further comprising:
determining multiple post-detection features of the detected at least one road anomaly, wherein the multiple post-detection features include an object box area, intersection of neighboring object boxes, union of neighboring object boxes; and
determining whether neighboring object boxes are one of
fully overlapping for objects of different classes,
partially overlapping for objects of different classes,
fully overlapping for objects of same class,
partially overlapping for objects of same class.
4. The method of claim 3 , further comprising
when it is determined that neighboring object boxes are fully overlapping for objects of different classes, count them as two different anomalies.
5. The method of claim 3 , further comprising
when it is determined that neighboring object boxes are fully overlapping for objects of same class, then count them as one anomaly.
6. The method of claim 3 , further comprising:
when it is determined that the neighboring object boxes are partially overlapping,
determining a ratio of the Intersection and the Union (IoU),
checking the IoU against a predefined threshold,
if the IoU exceeds the predefined threshold, the neighboring anomalies are counted as one large, connected anomaly, and
removing a smaller of the neighboring object boxes from a count of road anomalies.
7. The method of claim 6 , further comprising determining a density of the road anomalies.
8. The method of claim 7 , further comprising determining that the density of the anomalies is above a predetermined threshold, such that there is a high density of anomalies in the area of the road, indicating bad road health.
9. The method of claim 1 , further comprising maintaining a count of a number of road anomalies.
10. The method of claim 1 , wherein the ML model is implemented at the end computing device, wherein the end computing device is configured to stream portions of the visual data containing the detected road anomaly to the roadside computing device.
11. A system comprising:
at least one camera, mounted in a vehicle, configured to capture a live feed;
an end computing device, in or on the vehicle travelling along a road, configured to send visual data of a road, wherein the visual data is the live feed captured using the camera;
a processing circuitry configured to:
input the visual data to a machine learning (ML) model that learns to detect and classify at least one road anomaly into a class including one of a pothole, longitudinal crack, transverse crack, or alligator crack on the road;
output the class and a bounding box, created by the ML model, of the at least one road anomaly;
determine multiple post-detection features of the detected at least one road anomaly, wherein the determined multiple post-detection features includes that the at least one road anomaly is detected across multiple sequential frames,
when the at least one road anomaly is detected across multiple sequential frames, determine a number of skip frames based on a model fidelity distance (MFD) and a vehicle speed, wherein the MFD is a distance that the ML model can detect objects within a predetermined error;
determine that the detected at least one road anomaly is detected multiple times in each of the multiple sequential frames;
skip frames based on the number of skip frames to a key frame; and
a roadside computing device installed proximate to the road, configured to transmit a notification to multiple end computing devices in a communication range of the roadside computing device, wherein the notification is an anomaly-specific notification that includes at least one of a message indicating a presence of the road anomaly, an image of the road anomaly, a location of the road anomaly, or a severity of the road anomaly.
12. The system of claim 11 , wherein the processing circuitry is further configured to assess a severity of the road, wherein the severity is determined to be of a first value when an area of the bounding box exceeds a severity threshold area.
13. The system of claim 11 , wherein the processing circuitry is further configured to determine multiple post-detection features of the detected at least one road anomaly, wherein the multiple post-detection features include an object box area, intersection of neighboring object boxes, union of neighboring object boxes;
determine whether neighboring object boxes are one of
fully overlapping for objects of different classes,
partially overlapping for objects of different classes,
fully overlapping for objects of same class,
partially overlapping for objects of same class.
14. The system of claim 13 , wherein the processing circuitry is further configured to
when it is determined that neighboring object boxes are fully overlapping for objects of different classes, count them as two different anomalies.
15. The system of claim 13 , wherein the processing circuitry is further configured to when it is determined that neighboring object boxes are fully overlapping for objects of same class, then count them as one anomaly.
16. The system of claim 13 , wherein the processing circuitry is further configured to
when it is determined that the neighboring object boxes are partially overlapping, determine a ratio of the Intersection and the Union (IoU),
check the IoU against a predefined threshold,
if the IoU exceeds a predefined threshold, the neighboring object boxes are counted as one large, connected anomaly, and
remove a smaller of the neighboring object boxes from a count of road anomalies.
17. The system of claim 16 , wherein the processing circuitry is further configured to determine a density of the road anomalies.
18. The system of claim 17 , wherein the processing circuitry is further configured to determine that the density of the anomalies is above a predetermined threshold, such that there is a high density of anomalies in the area of the road, indicating bad road health.
19. The system of claim 11 , wherein the processing circuitry is further configured to accumulate a count of a number of road anomalies.
20. The system of claim 11 , wherein the ML model is implemented at the end computing device, wherein the end computing device is configured to stream portions of the visual data containing the detected road anomaly to the roadside computing device.