Method and system for collaborative task-based allocation between human and autonomous systems

1 . A method for determining a visual attention of a human operator to an operating panel of a vehicle during operation of the vehicle, the method comprising:

receiving, by a processor, data indicative of a gaze direction of the human operator from at least one monitoring device over a first period of time and a second period of time;

determining, by the processor, an approximate location of the gaze direction of the human operator on the operating panel at different individual times over the first period of time and the second period of time;

identifying, by the processor, an individual areas-of-interest (AOI) located at each of the determined approximate locations of the gaze direction;

calculating, by the processor, a first value and a second value for at least one metric using at least the determined approximate locations on the operating panel at the different individual times and the identification of the individual AOIs located at the determined approximate locations over the first and second periods of time to determine the visual attention of the human operator to the operating panel;

assessing, by the processor using the at least one metric, a performance of the human operator during an action of the vehicle, the assessing including comparing the first value for the at least one metric to a standard value for the at least one metric;

comparing, by the processor, the first value for the at least one metric for the first period of time to the second value for the at least one metric for the second period of time to compare the performance of the human operator between the first and second periods of time; and

performing an automated task by the vehicle based on the comparison of the performance of the human operator between the first and second periods of time.

2 . The method of claim 1 , wherein the at least one metric includes: an approximate length of time that the gaze direction of the human operator fell on each of the identified individual AOI during the first period of time; an amount of time for each transition between the gaze direction falling on a first individual AOI and a second individual AOI; and/or an amount of time determined for a sequence of multiple transitions, wherein each of the multiple transitions includes an amount of time between the gaze direction falling on the first individual AOI and the second individual AOI.

3 . The method of claim 1 , wherein the at least one metric is an order in which the human operator gazed upon different ones of the individual AOI over the first period of time.

4 . The method of claim 1 , wherein determining the approximate location of the gaze direction of the human operator on the operating panel at the different individual times over the first and second periods of time includes receiving focal point data for eyes of the human operator collected by at least one of a head-mounted eye tracker and a non-head-mounted eye tracker.

5 . The method of claim 1 , further comprising identifying, by the processor, a non-AOI time period within the first and second periods of time during which the gaze direction of the human operator does not fall on any one of the identified individual AOIs.

6 . The method of claim 5 , wherein calculating the first and second values for the at least one metric further includes using the identified non-AOI time period.

7 . The method of claim 6 , further comprising assessing, by the processor, the performance of the human operator during the action of the vehicle using the at least one metric, wherein the assessing includes comparing the second value for the at least one metric to the standard value for the at least one metric.

8 . The method of claim 1 , further comprising dynamically allocating one or more vehicle operation tasks of the human operator to the vehicle based on the comparison of the performance of the human operator between the first and second periods of time.

9 . The method of claim 1 , wherein the processor processes the data received from the at least one monitoring device using a machine learning model to determine a cognitive state of the human operator and package the data into a free form data string stream for transmission to a second processor.

10 . At least one non-transitory computer-readable medium storing instructions that, when executed by a computer, perform a method for determining a visual attention of a human operator to an operating panel of a vehicle during operation of the vehicle, the method comprising:

receiving, by a processor, data indicative of a gaze direction of the human operator from at least one monitoring device over a first period of time and a second period of time;

determining, by the processor, an approximate location of the gaze direction of the human operator on the operating panel at different individual times over the first and second periods of time;

identifying, by the processor, an individual areas-of-interest (AOI) located at each of the determined approximate locations of the gaze direction;

calculating, by the processor, first and second values for at least one metric using at least the determined approximate locations on the operating panel at the different individual times and the identification of the individual AOIs located at the determined approximate locations over the first and second periods of time to determine the visual attention of the human operator to the operating panel;

assessing, by the processor using the at least one metric, a performance of the human operator during an action of the vehicle, the assessing including comparing the first value for the at least one metric to a standard value for the at least one metric;

comparing, by the processor, the first value for the at least one metric for the first period of time to the second value for the at least one metric for the second period of time to compare the performance of the human operator between the first and second periods of time; and

performing an automated task by the vehicle based on the comparison of the performance of the human operator between the first and second periods of time.

11 . The at least one non-transitory computer readable medium of claim 10 , wherein the at least one metric includes: an approximate length of time that the gaze direction of the human operator fell on each of the identified individual AOI during the first period of time; an amount of time for each transition between the gaze direction falling on a first individual AOI and a second individual AOI; an amount of time determined for a sequence of multiple transitions, wherein each of the multiple transitions includes an amount of time between the gaze direction falling on the first individual AOI and the second individual AOI; and/or an order in which the human operator gazed upon different AOI over the period of time.

12 . The at least one non-transitory computer readable medium of claim 10 , wherein determining the approximate location of the gaze direction on the operating panel at the different individual times over the first and second periods of time includes receiving focal point data for eyes of the human operator collected by at least one of a head-mounted eye tracker and a non-head-mounted eye tracker.

13 . The at least one non-transitory computer readable medium of claim 10 , the method further comprising identifying, by the processor, a non-AOI time period within the first and second periods of time during which the gaze direction does not fall on any one of the identified individual AOIs.

14 . The at least one non-transitory computer readable medium of claim 13 , wherein calculating the first and second values for the at least one metric further includes using the identified non-AOI time period.

15 . The at least one non-transitory computer readable medium of claim 14 , further comprising assessing, by the processor, the performance of the human operator during the action of the vehicle using the at least one metric, wherein the assessing includes comparing the second value for the at least one metric to the standard value for the at least one metric.

16 . A method for determining a visual attention of a human operator to an operating panel of a during vehicle operation of the vehicle, the method comprising:

receiving, by a processor, data indicative of a gaze direction of the human operator from at least one monitoring device over first and second predetermined periods of time;

determining, by the processor, an approximate location of the gaze direction of the human operator on the operating panel at different individual times over the first and second predetermined periods of time;

identifying, by the processor, an individual areas-of-interest (AOI) located at each of the determined approximate locations of the gaze direction;

calculating, by the processor, first and second values for at least one metric using at least the determined approximate locations on the operating panel at the different individual times and the identification of the individual AOIs located at the determined approximate locations during the first and second predetermined periods of time to determine the visual attention of the human operator to the operating panel;

assessing, by the processor, a performance of the human operator during an action of the vehicle, the assessing including comparing the first value for the at least one metric to a standard value for the at least one metric;

comparing, by the processor, the first value for the at least one metric to the second value for the at least one metric to compare the performance of the human operator between the first and second predetermined periods of time; and

calculating a summary statistic for the value of the at least one metric using a first value for the at least one metric calculated during the first predetermined period of time and a second value for the at least one metric calculated during the second predetermined period of time.

17 . The method of claim 16 , wherein the at least one metric is selected from a group consisting of: an approximate length of time that the human operator's gaze fell on each of the identified individual AOI during the period of time; an amount of time for each transition between the human operator's gaze falling on a first individual AOI and a second individual AOI; an amount of time determined for a sequence of multiple transitions, wherein each of the multiple transitions includes an amount of time between a human operator's gaze falling on a first individual AOI and a second individual AOI; and an order in which the human operator gazed upon different AOI over the period of time.

18 . The method of claim 16 , further comprising commanding, by the processor, one or more subsystems of the vehicle to perform one or more automated tasks based on the comparison of the performance of the human operator between the first and second predetermined periods of time.

19 . The method of claim 16 , further comprising dynamically allocating one or more vehicle operation tasks of the human operator to the vehicle based on the comparison of the performance of the human operator between the first and second predetermined periods of time.

20 . The method of claim 16 , wherein the processor processes the data received from the at least one monitoring device using a machine learning model to determine a cognitive state of the human operator and package the data into a free form data string stream for transmission to a second processor.