Method and system for privacy-preserving fall detection
Various embodiments of a vision-based privacy-preserving embedded fall-detection system are disclosed. This embedded fall-detection system can include one or more cameras for capturing video images of one or more persons. Moreover, this embedded fall-detection system can include various fall-detection modules for processing the captured video images including: a pose-estimation module, an action-recognition module, and a fall-detection module, all of which can perform the intended fall-detection functionalities within the embedded system environment in real-time in order to detect falls of the one or more persons. When a fall is detected, instead of sending the original captured images, the embedded fall-detection system can transmit sanitized video images to the server, wherein each detected person is represented by a skeleton figure in place of the actual person images, thereby preserving the privacy of the detected person. The embedded fall-detection system can be implemented as an embedded vision sensor to be installed at a single fixed location.
1. A method of performing person fall detection, comprising:
receiving a sequence of video images including a person being monitored;
for each image in the sequence of video images:
detecting a pose of the person within the image; and
classifying the pose of the person within the image into an action among a set of predetermined actions;
aggregating the sequence of classified actions for the person corresponding to the sequence of video images; and
processing the aggregated sequence of classified actions to determine if a fall has occurred to the person using a state machine that includes a normal state representing normal activities, an alarm state representing a fall, and at least one intermediate state between the normal state and the alarm state, wherein using the state machine to determine if a fall has occurred further includes:
receiving a current state and an associated state score for the person in the state machine; and
sequentially processing the aggregated sequence of classified actions by:
for each classified action in the aggregated sequence of classified actions classified as a dangerous activity:
increasing the state score of the current state in the state machine to obtain an updated state score of the current state; and
if the updated state score of the current state exceeds a predetermined upper bound, causing the state machine to transition from the current state toward the alarm state; and
for each classified action in the aggregated sequence of classified actions classified as a normal activity:
decreasing the state score of the current state in the state machine to obtain an updated state score of the current state; and
if the updated state score of the current state drops below a predetermined lower bound, causing the state machine to transition from the current state toward the normal state.
2. The method of claim 1 , wherein detecting a pose of the person within the image includes:
identifying a set of locations within the image corresponding to a set of human keypoints of the person; and
connecting neighboring locations in the set of identified locations to form a skeleton diagram of the person, wherein the skeleton diagram represents a sanitized image of the person.
3. The method of claim 2 , wherein classifying the pose of the person into an action among the set of predetermined actions includes:
cropping out from the image, a two-dimensional (2-D) image of the person based on the skeleton diagram of the person;
feeding the cropped image of the person into an action classifier configured to predict probabilities of the person being in each of the set of predetermined actions; and
classifying the pose of the person into the action based on the set of probabilities corresponding to the set of predetermined actions.
4. The method of claim 1 , wherein the set of predetermined actions includes a first category of actions and a second category of actions, and wherein classifying the pose of the person into an action among the set of predetermined actions includes:
classifying the pose of the person into either the first category of actions or the second category of actions; and
for the classified first or second category of actions, further classifying the pose of the person into a predetermined action among the classified category of actions.
5. The method of claim 4 ,
wherein the first category of actions is a subset of dangerous actions among the set of predetermined actions; and
wherein the second category of actions is a subset of normal actions among the set of predetermined actions.
6. The method of claim 5 ,
wherein the subset of dangerous actions includes one or more of: a lying action and a struggling action; and
wherein the subset of normal actions includes one or more of: a standing action, one or more types of sitting actions, a bending action, and a squatting action.
7. The method of claim 1 , wherein aggregating the sequence of classified actions for the person corresponding to the sequence of video images includes:
identifying a dangerous region, such as a floor or a carpet in the sequence of video images;
determining a spatial relationship between each of the sequence of classified actions and the identified dangerous region; and
when a classified dangerous action among the sequence of classified actions is determined to be within the identified dangerous region, confirming the classified dangerous action.
8. The method of claim 1 , wherein aggregating the sequence of classified actions for the person corresponding to the sequence of video images includes:
identifying a normal region, such as a bed or a sofa in the sequence of video images;
determining a spatial relationship between each of the sequence of classified actions and the identified normal region; and
when a classified dangerous action among the sequence of classified actions is determined to be within the identified normal region, reclassifying the classified dangerous action as a normal action.
9. The method claim 2 , wherein if a fall is detected for the person, the method further includes:
generating a fall alarm;
generating a sanitized video clip depicting the fall by replacing the actual images of the person in the sequence of video images with the skeleton diagrams of the person; and
transmitting the sanitized video clip along with the fall alarm to a server, thereby preserving the privacy of the person.
10. The method claim 2 , wherein if a fall is detected for the person, the method further includes:
generating a sanitized video clip by:
identifying a common background image for the sequence of video images; and
superimposing the set of skeleton diagrams of the person corresponding to the sequence of video images onto the common background image to obtain a sequence of sanitized video images; and
transmitting the sanitized video clip composed of the sequence of sanitized video images to a server, thereby preserving the privacy of the person.
11. The method of claim 1 , wherein a fall is detected for the person when the state machine transitions into the alarm state.
12. A fall-detection system, comprising:
one or more processors;
a memory coupled to the one or more processors;
a pose-estimation module stored in the memory and configured to:
receive a sequence of video images including a person being monitored; and
detect a pose of the person within each image in the sequence of video images;
an action-recognition module stored in the memory and configured to classify, for each image in the sequence of video images, the pose of the person within the image into an action among a set of predetermined actions; and
a fall-detection module stored in the memory and configured to:
aggregate the sequence of classified actions for the person corresponding to the sequence of video images; and
process the aggregated sequence of classified actions to determine if a fall has occurred to the person using a state machine that includes a normal state representing normal activities, an alarm state representing a fall, and at least one intermediate state between the normal state and the alarm state, wherein using the state machine to determine if a fall has occurred further includes:
receiving a current state and an associated state score for the person in the state machine; and
sequentially processing the aggregated sequence of classified actions by:
for each classified action in the aggregated sequence of classified actions classified as a dangerous activity;
increasing the state score of the current state in the state machine to obtain an updated state score of the current state; and
if the updated state score of the current state exceeds a predetermined upper bound, causing the state machine to transition from the current state toward the alarm state; and
for each classified action in the aggregated sequence of classified actions classified as a normal activity:
decreasing the state score of the current state in the state machine to obtain an updated state score of the current state; and
if the updated state score of the current state drops below a predetermined lower bound, causing the state machine to transition from the current state toward the normal state.
13. The fall-detection system of claim 12 , wherein the pose estimation module is configured to detect a pose of the person within the image by:
identifying a set of locations within the image corresponding to a set of human keypoints of the person; and
connecting neighboring locations in the set of identified locations to form a skeleton diagram of the person, wherein the skeleton diagram represents a sanitized image of the person.
14. The fall-detection system of claim 13 , wherein the action recognition module is configured to classify the pose of the person into an action among the set of predetermined actions by:
cropping out from the image, a two-dimensional (2-D) image of the person based on the skeleton diagram of the person;
feeding the cropped image of the person into an action classifier configured to predict probabilities of the person being in each of the set of predetermined actions; and
classifying the pose of the person into the action based on the set of probabilities corresponding to the set of predetermined actions.
15. The fall-detection system of claim 12 , wherein the fall-detection module is configured to aggregate the sequence of classified actions corresponding to the sequence of video images by:
identifying a normal region, such as a bed or a sofa in the sequence of video images;
determining a spatial relationship between each of the sequence of classified actions and the identified normal region; and
when a classified dangerous action among of the sequence of classified actions is determined to be within the identified normal region, reclassifying the classified dangerous action as a normal action.
16. The fall-detection system of claim 12 , wherein a fall is detected for the person when the state machine transitions into the alarm state.
17. An embedded fall-detection system, comprising:
one or more cameras configured to capture a sequence of video images including a person;
one or more processors;
a memory coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the system to:
receive the sequence of video images;
detect a pose of the person within each image in the sequence of video images;
classify, for each image in the sequence of video images, the pose of the person within the image into an action among a set of predetermined actions;
aggregate the sequence of classified actions for the person corresponding to the sequence of video images; and
process the aggregated sequence of classified actions to determine if a fall has occurred to the person using a state machine that includes a normal state representing normal activities, an alarm state representing a fall, and at least one intermediate state between the normal state and the alarm state, wherein using the state machine to determine if a fall has occurred further includes:
receiving a current state and an associated state score for the person in the state machine; and
sequentially processing the aggregated sequence of classified actions by:
for each classified action in the aggregated sequence of classified actions classified as a dangerous activity;
increasing the state score of the current state in the state machine to obtain an updated state score of the current state; and
if the updated state score of the current state exceeds a predetermined upper bound, causing the state machine to transition from the current state toward the alarm state; and
for each classified action in the aggregated sequence of classified actions classified as a normal activity;
decreasing the state score of the current state in the state machine to obtain an updated state score of the current state; and
if the updated state score of the current state drops below a predetermined lower bound, causing the state machine to transition from the current state toward the normal state.
18. The embedded fall-detection system of claim 17 , wherein a fall is detected for the person when the state machine transitions into the alarm state.