System and method for gaze and pose detection to anticipate operator intent
A system and method for inferring operator intent by detecting operator focus incorporates cameras positioned within a cockpit or control space of a vehicle and oriented at an operator of the vehicle. The cameras capture images of the operator in a control seat; the images are analyzed (either individually or sequentially) to determine a gaze and/or body pose of the operator (including, e.g., a position and orientation of the torso and limbs). By comparing the determined gaze and/or body pose to the positions and orientations of potential focus targets within the control space (e.g., windows, display units, and/or control panels that the operator may engage with visually and/or physically), the system predicts the most likely future focus target or targets: what the operator is most likely to visually and/or physically engage with next. Operator intent may be further analyzed to identify potentially abnormal or anomalous behavior.
1. A system for inferring operator intent by detecting operator focus, the system comprising:
one or more cameras positioned within a control space of a mobile platform, the control space comprising one or more interfaces, the one or more cameras oriented toward a control position within the control space and configured to capture at least one image corresponding to:
a gaze of an operator occupying the control position;
or
a body pose of the operator, the body pose comprising at least one of a position or an orientation of at least one body part of the operator;
a memory configured for storage of one or more target poses, each target pose associated with at least one of a position or an orientation of the one or more interfaces, each target pose relative to the one or more cameras;
and
one or more processors in communication with the one or more cameras and the memory, the one or more processors configured to:
receive the at least one image;
detect, based on the at least one image, at least one of the gaze or the body pose;
train the memory using training data corresponding to one or more observed parameters based on the detected gaze or the body pose;
and
based on the detected gaze or body pose and the one or more target poses, infer at least one future focus target of the operator, the future focus target comprising at least one of:
a future target of the gaze of the operator;
or
an interface associated with a future engagement by the operator,
wherein the one or more processors are configured to determine a confidence level corresponding to each future focus target.
2. The system for inferring operator intent of claim 1 , wherein the body pose is associated with at least one of:
an arm pose of the operator;
a hand pose of the operator;
a finger pose of the operator;
a posture of the operator;
or
a torso orientation of the operator.
3. The system for inferring operator intent of claim 1 , wherein the at least one future focus target is selected from a group including:
a window of the mobile platform;
a display of the mobile platform;
or
an auxiliary device associated with the operator.
4. The system for inferring operator intent of claim 1 , wherein the at least one interface is selected from a group including:
a manual control configured to be engaged by the operator;
or
a touchscreen display surface configured to be engaged by the operator.
5. The system for inferring operator intent of claim 1 , wherein the one or more processors are configured to:
receive operational context selected from a group including:
a flight segment, an identifier of the operator, a position of the mobile platform, or a heading of the mobile platform;
and
infer the at least one future focus target based at least partially on the operational context.
6. The system for inferring operator intent of claim 1 , wherein the one or more processors are configured to:
store to the memory at least one operator profile corresponding to the operator, the at least one operator profile comprising one or more of:
an inferred future gaze target;
an inferred interface;
an inferred movement pattern;
a confidence level corresponding to the inferred future gaze target, interface, or inferred movement pattern;
or
operational context associated with the inferred future gaze target, interface, or inferred movement pattern.
7. The system for inferring operator intent of claim 6 , wherein:
the one or more processors are configured to infer at least one of a second future gaze target, a second interface, or a second movement pattern associated with an operator based on the at least one operator profile.
8. The system for inferring operator intent of claim 1 , wherein:
the at least one image includes at least one sequence of one or more images; and
the one or more processors are configured to:
detect, based on the at least one sequence, one or more of:
a gaze shift from a first gaze direction to a second gaze direction;
or
a pose change from a first body pose to a second body pose;
and
infer the at least one future focus target based on the gaze shift or pose change.
9. The system for inferring operator intent of claim 8 , wherein the one or more processors are configured to:
infer, based on the one or more of a gaze shift or a pose change, at least one movement pattern of the operator;
and
store the at least one inferred movement pattern to the memory.
10. A method for inferring operator intent by detecting operator focus, the method comprising:
capturing, via one or more cameras, at least one image indicative of an operator of a mobile platform, the operator in an operating position proximate to one or more focus targets;
analyzing, via one or more processors in communication with the one or more cameras, the at least one image;
detecting, based on the analyzing, one or more of:
a gaze of the operator;
or
a body pose of the operator, the body pose comprising at least one of a position or an orientation of at least one body part of the operator;
training a memory using training data corresponding to one or more observed parameters based on the detected gaze or the body pose;
and
inferring, based on the detected gaze or body pose, at least one future focus target of the operator, the future focus target selected from the one or more focus targets and comprising at least one of:
a future target of the gaze of the operator;
or
an interface associated with a future engagement by the operator,
wherein the one or more processors are configured to determine a confidence level corresponding to each future focus target.
11. The method of claim 10 , wherein:
analyzing the at least one image includes analyzing at least one sequence of one or more images;
and
detecting one or more of a gaze of the operator or a body pose of the operator includes detecting, based on the at least one sequence, at least one of 1) A gaze shift from a first gaze direction to a second gaze direction or 2) a pose change from a first body pose to a second body pose.
12. The method of claim 11 , wherein:
inferring at least one future focus target of the operator includes inferring at least one movement pattern of the operator based on the detected gaze shift or pose change.
13. The method of claim 10 , further comprising:
receiving, via the one or more processors, operational context selected from a group including: a flight segment; an identifier of the operator; a position of the mobile platform; or a heading of the mobile platform;
and wherein
inferring, based on the detected gaze or body pose, at least one future focus target of the operator includes inferring the at least one future focus target based at least partially on the operational context.