Pupil dynamics entropy and task context for automatic prediction of confidence in data
A pilot monitoring system receives data of a pilot's pose such as arm/hand positions and eyes to detect their gaze and pupil dynamics, coupled with knowledge about their current task to detect what a pilot is paying attention to, and temporally predict what they may do next. The system may use interactions between the pilot and the instrumentation to estimate a probability distribution of the next intention of the pilot. Such probability distribution may be used subsequently to evaluate the performance or training effectiveness and readiness of the pilot. The system determine data that will be necessary for a later pilot action based on the probability distribution, and compile that data from avionics systems for later display.
1 . A computer apparatus comprising:
at least one camera; and
at least one processor in data communication with a memory storing processor executable code; and
wherein the processor executable code configures the at least one processor to instantiate a trained neural network to:
perform an initial calibration to determine a baseline behavior of the pilot for a task;
receive an image stream from the at least one camera;
receive data being displayed to a pilot;
process the image stream to identify eye tracking data including pupil dynamics and eyelid position;
determine a pilot pose estimate based on the eye tracking data including pupil dynamics identified in the image stream;
correlate the pilot pose estimate to the data being displayed, the data being displayed comprising instrument readings and alerts associated with the task;
compare the baseline behavior to subsequent pupil behavior to detect anomalous behavior representative of pilot skepticism with respect to the displayed data;
determine a pilot confidence level based on the pilot pose estimate and the anomalous behavior; and
when the pilot confidence level is below a predetermined threshold, retrieve supplemental data comprising a source for the data and metadata of the corresponding source, pertaining to the displayed data, and displaying the supplemental data to the pilot.
2 . The computer apparatus of claim 1 , wherein the pose estimate comprises at least a pilot eye movement, a pilot gaze, a pilot eye lid position, a pilot hand and arm position, and a pilot hand and arm movement.
3 . The computer apparatus of claim 1 , wherein behavior representative of pilot skepticism comprises dwell time and characteristic eye lid position.
4 . A method for monitoring pilot behavior via a trained neural network, the method comprising:
performing an initial calibration to determine a baseline behavior of the pilot for a task;
receiving an image stream from at least one camera;
receiving data being displayed to a pilot;
processing the image stream to identify eye tracking data including pupil dynamics and eyelid position;
determining a pilot pose estimate based on the eye tracking data including pupil dynamics identified in the image stream;
correlating the pilot pose estimate to the data being displayed, the data being displayed comprising instrument readings and alerts associated with the task;
comparing the baseline behavior to subsequent pupil behavior to detect anomalous behavior representative of pilot skepticism with respect to the displayed data;
determining a pilot confidence level based on the pilot pose estimate and the anomalous behavior; and
when the pilot confidence level is below a predetermined threshold, retrieving supplemental data comprising a source for the data and metadata of the corresponding source, pertaining to the displayed data and displaying the supplemental data to the pilot.
5 . The method of claim 4 , wherein the pose estimate comprises at least a pilot eye movement, a pilot gaze, a pilot eye lid position, a pilot hand and arm position, and a pilot hand and arm movement.
6 . The method of claim 4 , wherein behavior representative of pilot skepticism comprises dwell time and characteristic eye lid position.
7 . A pilot monitoring system comprising:
at least one camera; and
at least one processor in data communication with a memory storing processor executable code; and
wherein the processor executable code configures the at least one processor to instantiate a trained neural network to:
perform an initial calibration to determine a baseline behavior of the pilot for a task;
receive an image stream from the at least one camera;
receive data being displayed to a pilot;
process the image stream to identify eye tracking data including pupil dynamics and eyelid position;
determine a pilot pose estimate based on the eye tracking data including pupil dynamics identified in the image stream;
correlate the pilot pose estimate to the data being displayed, the data being displayed comprising instrument readings and alerts associated with the task;
compare the baseline behavior to subsequent pupil behavior to detect anomalous behavior representative of pilot skepticism with respect to the displayed data;
determine a pilot confidence level based on the pilot pose estimate and the anomalous behavior; and
when the pilot confidence level is below a predetermined threshold, retrieve supplemental data comprising a source for the data and metadata of the corresponding source, pertaining to the displayed data, and displaying the supplemental data to the pilot.
8 . The pilot monitoring system of claim 7 , wherein the pose estimate comprises at least a pilot eye movement, a pilot gaze, a pilot eye lid position, a pilot hand and arm position, and a pilot hand and arm movement.
9 . The pilot monitoring system of claim 7 , wherein behavior representative of pilot skepticism comprises dwell time and characteristic eye lid position.