Systems and methods for simulating instantaneous attentive human responses to road hazards for improved early warning systems and fleet data harvesting

1. A method for predicting a hazardous event from road-scene data, the method comprising:

providing a neural network;

inputting the road-scene data, gaze-tracking data, and physiological data into the neural network, the neural network is trained with training gaze-tracking data and training physiological data associated with road-scene training data;

predicting, with the neural network, an occurrence of the hazardous event;

determining, with the neural network, one or more salient portions of the predicted hazardous event and one or more risk-weights for the one or more salient portions based on the gaze-tracking data and the physiological data; and

outputting a graded alert corresponding to the one or more risk-weights and the predicted hazardous event.

2. The method of claim 1 , wherein the neural network is trained by:

a) inputting the road-scene training data to the neural network, wherein the road-scene training data includes at least one sequence of an example hazardous event occurring in a road-scene;

b) identifying, with the neural network, one or more patterns of events present in the at least one sequence of the example hazardous event within the road-scene training data;

c) updating one or more parameters of the neural network in response to identifying the one or more patterns of events; and

d) repeating steps a-c one or more times, wherein the road-scene training data input during a first iteration of training of the neural network is different than a second iteration of training of the neural network.

3. The method of claim 2 , wherein the road-scene training data includes road-scene video data captured by a camera attached to a vehicle during a high-G event.

4. The method of claim 1 , wherein the neural network is trained by:

a) inputting the road-scene training data to the neural network;

b) inputting the training gaze-tracking data associated with the road-scene training data into the neural network;

c) inputting the training physiological response data associated with the road-scene training data into the neural network;

d) evolving the neural network to determine one or more salient portions within the road-scene data based on the training gaze-tracking data;

e) evolving the neural network to determine one or more risk-weights for the one or more salient portions within the road-scene data based on the training physiological response data associated with the road-scene data; and

f) repeating steps a-e one or more times, wherein the road-scene training data input during a first iteration of training of the neural network is different than a second iteration of training of the neural network.

5. The method of claim 4 , wherein the training physiological response data includes pupillometry data.

6. The method of claim 4 , wherein the training gaze-tracking data and the training physiological response data is generated from sensor systems configured to monitor responses of one or more individuals to the road-scene training data.

7. The method of claim 1 , further comprising generating a risk-weighted saliency map based on the predicted hazardous event.

8. The method of claim 1 , further comprising outputting an alert when the neural network predicts the occurrence of the hazardous event present in the road-scene data.

9. The method of claim 8 , wherein the alert is a graded alert corresponding to a degree of the predicted hazardous event.

10. The method of claim 1 , wherein the road-scene data includes road-scene video data.

11. The method of claim 1 , wherein the neural network is trained to identify one or more patterns preceding the hazardous event.

12. A system for predicting a hazardous event from road-scene data comprising:

an electronic control unit configured to implement a neural network; and

a camera communicatively coupled to the electronic control unit, wherein the camera generates the road-scene data, and wherein the electronic control unit is configured to:

receive the road-scene data from the camera;

input the road-scene data, gaze-tracking data, and physiological data into the neural network, the neural network is trained with training gaze-tracking data and training physiological data associated with road-scene training data;

predict, with the neural network, an occurrence of the hazardous event within the road-scene data from the camera;

determine, with the neural network, one or more salient portions of the predicted hazardous event and one or more risk-weights for the one or more salient portions based on the gaze-tracking data and the physiological data; and

output a graded alert corresponding to the one or more risk-weights and the predicted hazardous event.

13. The system of claim 12 , wherein the electronic control unit is further configured to:

generate, with the neural network, a risk-weighted saliency map based on the road-scene data.

14. The system of claim 12 , wherein the graded alert corresponds to a degree of the predicted hazardous event.

15. The system of claim 12 , wherein the neural network is trained to identify one or more patterns preceding the hazardous event.

16. The system of claim 12 , wherein the neural network is trained by the steps:

a) inputting the road-scene training data to the neural network, wherein the road-scene training data includes at least one sequence of an example hazardous event occurring in a road-scene;

b) identifying, with the neural network, one or more patterns of events present in the at least one sequence of the example hazardous event within the road-scene training data;

c) updating one or more parameters of the neural network in response to identifying the one or more patterns of events; and

d) repeating steps a-c one or more times, wherein the road-scene training data input during a first iteration of training of the neural network is different than a second iteration of training of the neural network.

17. The system of claim 12 , wherein the neural network is trained by:

a) inputting the road-scene training data to the neural network;

b) inputting the training gaze-tracking data associated with the road-scene training data into the neural network;

c) inputting the training physiological response data associated with the road-scene training data into the neural network;

d) evolving the neural network to determine one or more salient portions within the road-scene data based on the training gaze-tracking data;

e) evolving the neural network to determine one or more risk-weights for the one or more salient portions within the road-scene data based on the training physiological response data associated with the road-scene data; and

f) repeating steps a-e one or more times, wherein the road-scene training data input during a first iteration of training of the neural network is different than a second iteration of training of the neural network.

18. A vehicle comprising:

an electronic control unit configured to implement a neural network; and

a camera communicatively coupled to the electronic control unit, wherein the camera generates road-scene data, and wherein the electronic control unit is configured to:

receive the road-scene data from the camera;

input the road-scene data, gaze-tracking data, and physiological data into the neural network, the neural network is trained with training gaze-tracking data and training physiological data associated with road-scene training data;

predict, with the neural network, an occurrence of a hazardous event within the road-scene data from the camera;

determine, with the neural network, one or more salient portions of the predicted hazardous event and one or more risk-weights for the one or more salient portions based on the gaze-tracking data and the physiological data; and

output a graded alert corresponding to the one or more risk-weights and the predicted hazardous event.