Refining event triggers using machine learning model feedback
A vehicle device may execute one or more neural networks (and/or other artificial intelligence), such as based on input from one or more of the cameras and/or other sensors associated with the dash cam, to intelligently detect safety events in real-time. The vehicle device may further pass the input to a backend server for further analysis and the backend server can detect safety events based on the input. The vehicle device may analyze the output of the vehicle device and the output of the backend server to determine whether the output of the vehicle device is correct. If the output of the vehicle device is incorrect, the vehicle device can adjust how the vehicle device identifies safety events.
1. A vehicle device implemented in a vehicle, the vehicle device comprising:
a computer readable storage medium having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the vehicle device to:
obtain sensor data associated with an image of a driver of the vehicle, the sensor data including one or more objects associated with the driver of the vehicle, wherein the one or more objects comprise at least one of face of the driver or one or more hands of the driver;
route the sensor data to a receiving server system using a network connection with the receiving server system, wherein the receiving server system comprises a first neural network, wherein the first neural network is configured to output a first probability of an event based on a first probability of a feature of the one or more objects, wherein the feature comprises at least one of a hand action of the driver or a face pose of the driver;
access a second neural network, wherein the second neural network is configured to output a second probability of the event based on a second probability of the feature of the one or more objects;
identify a distracted state of the driver based at least in part on the second probability of the event;
obtain, from the receiving server system, a comparison of the first probability of the event and the second probability of the event; and
determine one or more updates to the vehicle device to increase accuracy of identifying the distracted state of the driver based on the comparison of the first probability of the event and the second probability of the event.
2. A vehicle device comprising:
a computer readable storage medium having program instructions embodied therewith; and
one or more processors configured to execute the program instructions to cause the vehicle device to:
obtain sensor data associated with an image of a scene, the sensor data including one or more objects in the image;
route the sensor data to a receiving server system using a network connection with the receiving server system, wherein the receiving server system comprises a first neural network, wherein the first neural network is configured to output a first probability of an event based on a first probability of a feature of the one or more objects;
access a second neural network, wherein the second neural network is configured to output a second probability of the event based on a second probability of the feature of the one or more objects;
identify the event based at least in part on the second probability of the event;
obtain, from the receiving server system, a comparison of the first probability of the event and the second probability of the event; and
determine one or more operations to dynamically adjust how the vehicle device identifies the event based on the comparison of the first probability of the event and the second probability of the event.
3. The vehicle device of claim 2 , wherein the image of the scene corresponds to an image of a vehicle driver associated with a vehicle, the vehicle device implemented in the vehicle, wherein the one or more objects comprise at least one of a face of the vehicle driver or one or more hands of the vehicle driver, and wherein the feature of the one or more objects comprises at least one of a hand action of the vehicle driver or a face pose of the vehicle driver.
4. The vehicle device of claim 2 , wherein the event comprises a distracted state of a vehicle driver.
5. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
determine the event comprises a false positive event in response to determining that the first probability of the event exceeds the second probability of the event by at least a particular amount.
6. The vehicle device of claim 2 , wherein to identify the event, the one or more processors are configured to execute the program instructions to further cause the vehicle device to identify the event based on a prediction schedule, wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
in response to determining that the event comprises a false positive event, adjust the prediction schedule.
7. The vehicle device of claim 2 , wherein to identify the event, the one or more processors are configured to execute the program instructions to further cause the vehicle device to identify the event based on a determination that the second probability of the event is greater than a prediction threshold.
8. The vehicle device of claim 7 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
determine the event comprises a false positive event based at least one comparison of the first probability of the event and the second probability of the event.
9. The vehicle device of claim 8 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
adjust the prediction threshold in response to determining the event comprises the false positive event.
10. The vehicle device of claim 2 , wherein, to identify the event, the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
determine that the second probability of the event is less than a prediction threshold indicative of non-occurrence of the feature;
determine that the event comprises a false negative event based on the comparison of the first probability of the event and the second probability of the event; and
adjust the prediction threshold based on identifying the false negative event.
11. The vehicle device of claim 2 , wherein, to identify the event, the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
identify, using a third neural network, the event based on the second probability of the event.
12. The vehicle device of claim 2 , wherein, to identify the event, the one or more processors are configured to execute the program instructions to further cause the vehicle device to identify, using a third neural network, the event based on the second probability of the event, wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
determine the event comprises a false positive event based on comparing the first probability of the event and the second probability of the event; and
train the third neural network, wherein the first probability of the event is provided as an expected output of the third neural network, wherein one or more characteristics of the image are provided as an input to train the third neural network.
13. The vehicle device of claim 2 , wherein, to identify the event, the one or more processors are configured to execute the program instructions to further cause the vehicle device to identify, using a third neural network, the event based on the second probability of the event, wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
determine the event comprises a false positive event based on comparing the first probability of the event and the second probability of the event; and
train the third neural network, wherein the first probability of the event is provided as an expected output of the third neural network, wherein one or more characteristics of the image are provided as an input to train the third neural network, wherein the one or more characteristics of the image comprise time of day data, a time associated with generation of the event, or a confidence score of the event.
14. The vehicle device of claim 2 , wherein, to identify the event, the one or more processors are configured to execute the program instructions to further cause the vehicle device to generate, using a third neural network, the event based on the second probability of the event, wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
determine the event comprises a false positive event based on comparing the first probability of the event and the second probability of the event; and
train the third neural network, wherein the first probability of the event is provided as an expected output of the third neural network, wherein the second probability of the event is provided as an input to train the third neural network.
15. The vehicle device of claim 2 , wherein the second probability is associated with a first frame, wherein, to identify the event, the one or more processors are configured to execute the program instructions to further cause the vehicle device to identify the event based on the second probability and a third probability of the event, wherein the third probability is associated with a second frame.
16. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
obtain input from a user computing device; and
determine the event comprises a false positive event based on the input.
17. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
provide an indication of the event to a user computing device;
obtain input from the user computing device; and
determine the event comprises a false positive event based on the input.
18. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
implement the one or more operations without retraining the second neural network;
obtain additional sensor data associated with an additional image of an additional scene, the additional sensor data identifying one or more additional objects of the additional image;
access the second neural network, wherein the second neural network is further configured to output a third probability of an additional event; and
identify an additional event based on the third probability of the additional event and the one or more operations.
19. The vehicle device of claim 2 , wherein the one or more processors are configured to execute the program instructions to further cause the vehicle device to:
implement the one or more operations without retraining the second neural network; and
identify a non-event based on the second probability of the event and the one or more operations.
20. A computer-implemented method comprising:
obtaining sensor data associated with an image of a scene, the sensor data including one or more objects in the image;
routing the sensor data to a receiving server system using a network connection with the receiving server system, wherein the receiving server system comprises a first neural network, wherein the first neural network is configured to output a first probability of an event based on a first probability of a feature of the one or more objects in the image;
accessing a second neural network, wherein the second neural network is configured to output a second probability of the event based on a second probability of the feature of the one or more objects in the image;
identifying the event based at least in part on the second probability of the event;
obtaining, from the receiving server system, a comparison of the first probability of the event and the second probability of the event;
determining one or more operations to dynamically adjust how events are identified based on the comparison of the first probability of the event and the second probability of the event;
implementing the one or more operations without retraining the second neural network;
obtaining additional sensor data associated with an additional image of an additional scene, the additional sensor data identifying one or more additional objects of the additional image;
accessing the second neural network, wherein the second neural network is further configured to output a third probability of an additional event; and
identifying the additional event based on the third probability of the additional event and the one or more operations.