Systems and methods for detection of subject activity by processing video and other signals using artificial intelligence
Various embodiments of a system and associated method for detection of subject activity by processing video and other signals using artificial intelligence are shown herein. In particular, a subject monitoring system is shown that monitors subjects using a video-capable camera or other suitable video capture device to identify a subject's status of an individual by monitoring subject actions in real-time. The system further monitors other persons in the room with a subject to identify their actions and identities to ensure safety of the subject and facility while preventing confusion of the system as multiple individuals step in and out of frame over the course of the collected video feed.
1 . A system, comprising:
a processor in communication with a memory, the memory including instructions, which, when executed, cause the processor to:
receive a real-time video feed including a plurality of frames that capture a body;
extract a feature set for the body from the real-time video feed including a plurality of features descriptive of the body captured within an incoming frame of the plurality of frames of the video feed;
recognize, by a neural network of the processor, an action captured over the plurality of frames, the neural network configured to interpret one or more spatial characteristics of the body between each feature set of a plurality of feature sets, each feature set corresponding with a respective frame of the plurality of frames of the video feed; and
generate an alert based on the recognized action.
2 . The system of claim 1 , wherein the memory includes instructions, which, when executed, further cause the processor to:
extract skeleton joint data for the body within a current frame of a plurality of frames of the real-time video feed; and
extract the feature set for the body based on a geometry of the skeleton joint data.
3 . The system of claim 1 , wherein the memory includes instructions, which, when executed, further cause the processor to:
extract an ocular distance for the body from a first frame of the real-time video feed; and
identify the body within a second frame of the real-time video feed based on the extracted ocular distance for the body.
4 . The system of claim 2 , wherein the memory includes instructions, which, when executed, further cause the processor to:
apply one or more additional recognition tasks for the body over the plurality of frames using the extracted skeleton joint data; and
combine results of the one or more additional recognition tasks with action recognition data indicative of the recognized action, the results of the one or more additional recognition tasks being correctly attributed to a corresponding body captured within the real-time video feed based on one or more facial characteristics of the body.
5 . The system of claim 1 , wherein the memory includes instructions, which, when executed, further cause the processor to:
recognize the action being performed by the body using a current feature set and a plurality of prior feature sets from one or more previous frames of the real-time video feed with consideration for a variation in elapsed time between frames of the real-time video feed.
6 . The system of claim 5 , wherein the neural network is a recurrent neural network including a plurality of long short term memory (LSTM) units, each LSTM unit configured to store a current memory of a feature set of the plurality of feature sets and directly adjust the current memory stored therein with respect to an elapsed time relative to other feature sets of the plurality of feature sets associated with the real-time video feed.
7 . The system of claim 1 , wherein the memory includes instructions, which, when executed, further cause the processor to:
extract one or more facial characteristics of the body for correct attribution of the body across multiple frames of the real-time video feed.
8 . The system of claim 7 , wherein the one or more extracted facial characteristics include an ocular distance that defines a rectangle area around a face of the body captured within the real-time video feed to match the face from a first frame of the plurality of frames to a second frame of the plurality of frames of the real-time video feed.
9 . The system of claim 1 , further comprising:
a display in communication with the processor configured to display the alert based on the recognized action.
10 . The system of claim 1 , further comprising:
a mitigation module in association with the processor and configured to broadcast the alert to one or more additional devices.
11 . A method, comprising:
receiving, at a processor, a real-time video feed including a plurality of frames that capture a body;
extracting a feature set for the body from the real-time video feed including a plurality of features descriptive of the body captured within an incoming frame of the video feed;
recognizing, by a neural network of the processor, an action captured over the plurality of frames, the neural network configured to interpret one or more spatial characteristics of the body between each feature set of a plurality of feature sets; and
generating an alert based on the recognized action.
12 . The method of claim 11 , further comprising:
adjusting a memory stored within a long short term memory unit of the neural network with respect to an elapsed time relative to other feature sets of the plurality of feature sets associated with the real-time video feed.
13 . The method of claim 11 , further comprising:
broadcasting the alert to one or more additional devices in communication with the processor.
14 . The method of claim 11 , further comprising:
applying one or more additional recognition tasks for the body over the plurality of frames using extracted skeleton joint data; and
combining results of the one or more additional recognition tasks with action recognition data indicative of the recognized action, the results of the one or more additional recognition tasks being correctly attributed to a corresponding body captured within the real-time video feed based on one or more facial characteristics of the body.
15 . The method of claim 14 , further comprising:
generating the alert based on the results of the one or more additional recognition tasks.
16 . The method of claim 11 , further comprising:
extracting skeleton joint data for the body within a current frame of the plurality of frames of the real-time video feed; and
extracting the feature set for the body based on the skeleton joint data.
17 . A system, comprising:
a processor in communication with a memory, the memory including instructions, which, when executed, cause the processor to:
receive a real-time video feed including a plurality of frames that capture a body;
extract a feature set for the body from the real-time video feed including a plurality of features descriptive of the body captured within an incoming frame of the video feed;
recognize, by a neural network of the processor, an action captured over the plurality of frames, the neural network configured to interpret one or more spatial characteristics of the body between each feature set of a plurality of feature sets captured across the plurality of frames with respect to an elapsed time between each frame of the plurality of frames; and
generate an alert based on the recognized action.
18 . The system of claim 17 , wherein the neural network is a recurrent neural network including a plurality of long short term memory (LSTM) units, wherein each LSTM unit stores a current memory of a feature set of the plurality of feature sets.
19 . The system of claim 18 , wherein each LSTM unit of the plurality of LSTM units is configured to directly adjust the current memory stored therein with respect to an elapsed time relative to other feature sets of the plurality of feature sets associated with the real-time video feed.
20 . The system of claim 18 , wherein a feature set of the plurality of feature sets corresponds to a respective frame of the plurality of frames of the real-time video feed.
21 . The system of claim 18 , wherein the memory includes instructions, which, when executed, further cause the processor to:
extract skeleton joint data for the body within a current frame of the plurality of frames of the real-time video feed; and
extract the feature set for the body based on the skeleton joint data.