Granted Patent B2

US 12,567,253 · App. 18/216,299 · Granted Mar 3, 2026

Inventors: Peggy Wu (Ellicott City, MD); Andrew Radlbeck (South Glastonbury, CT)

Assignee: Rockwell Collins, Inc.

G06V20/40G06V40/18

Abstract

Claims (72)

1 . A computer apparatus comprising:

at least one camera;

a data communication device; and

at least one processor in data communication with a memory storing processor executable code; and

wherein the processor executable code configures the at least one processor to:

receive a video stream from the at least one camera;

receive a predefined baseline user measurement of at least gaze and pupil dynamics;

determine a user metric of situational awareness and/or surprise based on the video stream compared to the predefined baseline;

receive one or more contemporaneous team member metrics of situational awareness and/or surprises via the data communication device;

determine a priority associated with each of the user metric of situational awareness and/or surprise and one or more contemporaneous team member metric of situational awareness and/or surprises;

weight the user metric of situational awareness and/or surprise and one or more contemporaneous team member metric of situational awareness and/or surprises according to the associated priority; and

determine a team metric of situational awareness and/or surprise based on the weighted user metric of situational awareness and/or surprise and one or more weighted contemporaneous team member metric of situational awareness and/or surprises.

2 . The computer apparatus of claim 1 , further comprising one or more physiological data recording devices in data communication with the at least one processor, wherein:

the processor executable code further configures the at least one processor to:

receive physiological data from the one or more physiological data recording devices; and

correlate the physiological data with the video stream; and

creating the user metric of situational awareness and/or surprise includes reference to the physiological data.

3 . The computer apparatus of claim 2 , wherein:

the processor executable code further configures the at least one processor to receive a task or user specific profile of gaze, scan pattern, voice intonation, and physiological data; and

creating the user metric of situational awareness and/or surprise includes reference to the task or user specific profile.

4 . The computer apparatus of claim 1 , further comprising a display, wherein the processor executable code further configures the at least one processor to:

receive at least one video stream from a team member via the data communication device;

display the at least one video stream from the team member on the display; and

determine the user engagement with reference to the at least one video stream from the team member.

5 . The computer apparatus of claim 1 , wherein the processor executable code further configures the at least one processor as a machine learning neural network.

6 . A method comprising:

receiving a video stream from at least one camera;

receiving a predefined baseline user measurement of at least gaze and pupil dynamics;

determining a user metric of situational awareness and/or surprise based on the video stream compared to the predefined baseline;

receiving one or more contemporaneous team member metrics of situational awareness and/or surprises via a data link;

determining a priority associated with each of the user metric of situational awareness and/or surprise and one or more contemporaneous team member metric of situational awareness and/or surprises;

weighting the user metric of situational awareness and/or surprise and one or more contemporaneous team member metric of situational awareness and/or surprises according to the associated priority; and

determining a team metric of situational awareness and/or surprise based on the weighted user metric of situational awareness and/or surprise and one or more weighted contemporaneous team member metric of situational awareness and/or surprises.

7 . The method of claim 6 , further comprising:

receiving physiological data from one or more physiological data recording devices; and

correlating the physiological data with the video stream,

wherein creating the user metric of situational awareness and/or surprise includes reference to the physiological data.

8 . The method of claim 7 , further comprising receiving a task or user specific profile of gaze, scan pattern, voice intonation, and physiological data, wherein creating the user metric of situational awareness and/or surprise includes reference to the task or user specific profile.

9 . The method of claim 6 , further comprising:

receiving at least one video stream from a team member;

displaying the at least one video stream from the team member on a display; and

determining the user engagement with reference to the at least one video stream from the team member.

10 . The method of claim 6 , further comprising recording the team metric of situational awareness and/or surprise, user metric of situational awareness and/or surprise, and one or more contemporaneous team member metric of situational awareness and/or surprises associated with each of a plurality of discreet tasks over time.

11 . The method of claim 10 , further comprising determining a team composition based on the team metric of situational awareness and/or surprise, user metric of situational awareness and/or surprise, and one or more contemporaneous team member metric of situational awareness and/or surprises based on individual engagement during discreet tasks.

12 . A team monitoring system comprising:

a plurality of team member monitoring computers, each comprising:

at least one camera;

a data communication device; and

at least one processor in data communication with a memory storing processor executable code to configure the at least one processor to:

receive an video stream from the at least one camera;

receive a predefined baseline user measurement of at least gaze and pupil dynamics;

determine a user metric of situational awareness and/or surprise based on the video stream compared to the predefined baseline;

receive one or more contemporaneous team member metrics of situational awareness and/or surprises from the plurality of team member monitoring computers via the data communication device;

determine a priority associated with each of the user metric of situational awareness and/or surprise and one or more contemporaneous team member metric of situational awareness and/or surprises;

weight the user metric of situational awareness and/or surprise and one or more contemporaneous team member metric of situational awareness and/or surprises according to the associated priority; and

determine a team metric of situational awareness and/or surprise based on the weighted user metric of situational awareness and/or surprise and one or more weighted contemporaneous team member metric of situational awareness and/or surprises.

13 . The team monitoring system of claim 12 , further comprising one or more physiological data recording devices in data communication with the at least one processor, wherein:

the processor executable code further configures the at least one processor to:

receive physiological data from the one or more physiological data recording devices; and

correlate the physiological data with the video stream; and

creating the user metric of situational awareness and/or surprise includes reference to the physiological data.

14 . The team monitoring system of claim 13 , wherein:

the processor executable code further configures the at least one processor to receive a task or user specific profile of gaze, scan pattern, voice intonation, and physiological data; and

creating the user metric of situational awareness and/or surprise includes reference to the task or user specific profile.

15 . The team monitoring system of claim 12 , further comprising a display, wherein the processor executable code further configures the at least one processor to:

receive at least one video stream from a team member via the data communication device;

display the at least one video stream from the team member on the display; and

determine the user engagement with reference to the at least one video stream from the team member.

16 . The team monitoring system of claim 12 , wherein the processor executable code further configures the at least one processor as a machine learning neural network.

17 . The team monitoring system of claim 12 , wherein the processor executable code further configures the at least one processor to:

record the team metric of situational awareness and/or surprise, user metric of situational awareness and/or surprise, and one or more contemporaneous team member metric of situational awareness and/or surprises associated with each of a plurality of discreet tasks over time; and

determine a team composition based on the team metric of situational awareness and/or surprise, user metric of situational awareness and/or surprise, and one or more contemporaneous team member metric of situational awareness and/or surprises based on individual engagement during discreet tasks.

Assignments (4)

Continuity (1)

References Cited (9)