Monitoring and optimization of human-device interactions
A real time system for monitoring and optimizing patient adherence to digital therapy. The system can measure at least one of head stability, eye stability, gaze direction, head pose, facial expression, reading related eye movements, eye alignment stability, eye blink rate, attentive engagement, general engagement, emotional state, total eye stability, yawning and distance between the user and the camera. This information is provided as input to computer vision algorithms to generate a multi-dimensional representation of user attention and engagement. The system can alert the caregiver or health care professional if the patient disengages from the treatment and if the patient is sitting at the wrong distance from the treatment device. The system includes a camera, a processor, computer vision algorithms executed by the processor (a GPU processor), at least one digital display device, a loud speaker and an optional internet connection that can enable communication with other electronic devices.
1 . A system for detecting an inattentive state of a person, comprising:
a video game system including a controller to provide user input and an electronic display at which a person's eyes are directed at configured to present a video game for viewing and playing by the person;
an image capture device positioned to capture digital video frames of the person's head and face;
a computing device configured to receive the captured digital video frames and extract visual cues of eyes and mouth of the person's head and face from each of the captured digital video frames, the computing device further configured to
increment a frame counter when
an average of an aspect ratio of a left eye and a right eye of the person exceeds a predetermined threshold in a current video frame, or
a yawn is detected from the shape of the mouth in the current video frame and the average of the aspect ratio of the left eye and the right eye of the person is less than or equal to the predetermined threshold,
reset the frame counter when no yawn is detected in the current video frame,
generate a first electronic signal when the frame counter reaches a predetermined number of frames and resetting the frame counter;
the computing device is further configured to detect
an eye gaze of the person is to the left or to the right of the screen by
extracting the eye portions from the face in the current video frame,
determining a first ratio of white pixels corresponding to the right side and the left side of the left iris,
determining a second ratio of white pixels corresponding to the right side and the left side of the right iris,
determining the eye gaze is to the right when an average of the first and second ratios is below a first eye gaze threshold, determining the eye gaze is to the left when the average is greater than a second eye gaze threshold, and determining the eye gaze is centered when the average is between the first and second eye gaze thresholds, and
the computing device generates a second electronic signal when the eye gaze is determined as being to the right or the left,
wherein the computing device is configured to change currently displayed content of the electronic display device in real time to display new content non-native to the video game in response to at least one of the first and the second electronic signals, indicating the inattentive state of the person.
2 . The system of claim 1 , further including an audio output device controlled by the computing device to generate an audio alert when at least one of the first and the second electronic signals is received.
3 . The system of claim 1 , wherein the computing device is configured to detect from the captured video frames the at least one visual cue of emotional state of the person as being one of happy, neutral, sad or angry.
4 . The system of claim 3 , wherein when the computing device detects either the sad or angry emotional states, the computing device controls the audio output device to provide an indication of an emergency state of the person.
5 . The system of claim 4 , wherein the computing device generates and transmits an alert to a mobile device of another person in response to the detected emergency state of the person.
6 . The system of claim 1 , wherein the computing device is configured to determine
a left eye alignment as a first ratio of a number of left side white pixels to a number of right side white pixels of the left eye of the person,
a right eye alignment as a second ratio of a number of left side white pixels to a number of right side white pixels of the right eye of the person,
eye alignment stability as the absolute value of the ratio of the left eye alignment to the right eye alignment, and
a classification of the eye alignment stability greater than a predetermined threshold and generating an output indicating the person exhibits strabismus.
7 . The system of claim 1 , wherein the computing device is configured to determine strabismus from a video frame as input to a convolution neural network trained with eye regions segmented from 175 each strabismus and non-strabismus eye images, with at least specifications of an epoch of 600, batch size of 32, and image size of 100×100.
8 . The system of claim 1 , wherein the new content non-native to the video game includes a graphic overlay on presented content of the video game on the electronic display.
9 . The system of claim 1 , wherein the new content non-native to the video game includes changing to a different video game.
10 . The system of claim 1 , further including pausing the video game in response to the at least one of the first and the second electronic signals.
11 . The system of claim 1 , further including to changing a difficulty level of the video game in response to the at least one of the first and the second electronic signals.
12 . A system for detecting an inattentive state of a person driving an automobile equipped with an autonomous driving system, comprising:
a windshield of a vehicle at which a person's eyes are directed at;
an audio output device;
an image capture device positioned to capture digital video frames of the person's head and face;
a computing device configured to receive the captured digital video frames and extract visual cues of eyes and mouth of the person's head and face from each of the captured digital video frames, the computing device further configured to
a) increment a frame counter when
an average of an aspect ratio of a left eye and a right eye of the person exceeds a predetermined threshold in a current video frame, or
a yawn is detected from the shape of the mouth in the current video frame and the average of the aspect ratio of the left eye and the right eye of the person is less than or equal to the predetermined threshold,
b) reset the frame counter when no yawn is detected in the current video frame,
c) generate a first electronic signal indicating a drowsy state of the person when the frame counter reaches a predetermined number of frames and resetting the frame counter; and
d) generate an audio alert to change the drowsy state of the person to an alert state, and repeating steps a) through c) in response to the first electronic signal, wherein when the drowsy state of the person is still detected after audio alert is generated the computing device is configured to control the autonomous driving system to take over control of the automobile and park the automobile in a safe area.
13 . The system of claim 12 , wherein the computing device is configured to determine pitch and yaw of the head of the person from the current video frame, and to determine that the head of the person is oriented straight when the determined pitch and yaw are within corresponding predetermined orientation thresholds.
14 . The system of claim 13 , wherein the computing device is configured to determine proper head stability of the person when a ratio of a number of frames with the person's head oriented straight towards the windshield to the total number of frames captured over a predetermined period of time exceeds a predetermined head stability threshold.
15 . The system of claim 14 , wherein the computing device is configured to control the audio output device to generate the audio alert in response to any one of the detected drowsy state of the person and when the head stability of the driver falls below the predetermined head stability threshold.
16 . The system of claim 12 , wherein the computing device is further configured to detect
an eye gaze of the person is to the left or to the right of the screen by
extracting the eye portions from the face in the current video frame,
determining a first ratio of white pixels corresponding to the right side and the left side of the left iris,
determining a second ratio of white pixels corresponding to the right side and the left side of the right iris,
determining the eye gaze is to the right when an average of the first and second ratios is below a first eye gaze threshold, determining the eye gaze is to the left when the average is greater than a second eye gaze threshold, and determining the eye gaze is centered when the average is between the first and second eye gaze thresholds, and
the computing device is configured to generate a second electronic signal indicating a distracted state of the driver when the eye gaze is determined as being to the right or the left for a predetermined duration of time, and the audio device is controlled to generate another audio alert in response to the second electronic signal.