Multimodal video-language artificial intelligence framework for surveillance environments
Methods and systems monitor activity in a surveillance environment. A three-dimensional convolutional neural network is used to extract spatiotemporal features from video frames. A connector module projects the spatiotemporal feature embeddings into a multimodal latent space. A language model analyzes the multimodal embeddings to output structured data which indicates the characteristics of a detected event, and possibly also unstructured data (e.g., a natural language summary). The output may be stored in a database.
1 . A computer-implemented method for automated surveillance analysis, the computer-implemented method comprising:
training, by one or more processors, a language model using a controlled ratio of value-only annotated video training data samples to structured data-value annotated video training data samples;
receiving, by the one or more processors, a time series of video frames depicting an environment;
generating, by the one or more processors and using a three-dimensional (3D) convolutional neural network, a plurality of spatiotemporal feature embeddings of the time series of video frames;
generating, by the one or more processors, a plurality of multimodal embeddings at least in part by projecting the plurality of spatiotemporal feature embeddings into a shared multimodal latent space;
generating, by the one or more processors, an output by using the language model to analyze the plurality of multimodal embeddings, wherein the output includes structured data indicating one or more characteristics of an event; and
storing, by the one or more processors, the output in a database.
2 . The computer-implemented method of claim 1 ,
wherein the output includes a JavaScript Object Notation (JSON) object, wherein the JSON object is indicative of one or more of (i) context of the event, (ii) activity detected during the event, or (iii) temporal information associated with the event.
3 . The computer-implemented method of claim 2 ,
wherein the output further includes a free-form text description comprising one or more of (i) descriptions of the context of the event, (ii) activity detected during the event, or (iii) temporal information associated with the event.
4 . The computer-implemented method of claim 3 , wherein generating the output comprises:
generating, using the language model, the free-form text description based on the JSON object.
5 . The computer-implemented method of claim 1 , further comprising:
generating, by the one or more processors and based on user inputs entered via a video annotation application, annotated video training data; and
training, by the one or more processors, the language model using the annotated video training data.
6 . The computer-implemented method of claim 5 ,
wherein the video annotation application provides text entry fields for user-configurable video annotation schemas.
7 . The computer-implemented method of claim 1 , further comprising:
fine-tuning, by the one or more processors, the language model based on retail-focused question-answer pairs.
8 . The computer-implemented method of claim 1 ,
wherein the event includes (i) an item selection event, (ii) an item concealment event, (iii) a self-checkout event, (iv) a cashier checkout event, (v) an item restocking event, (vi) a customer assistance event, or (vii) a staff maintenance event.
9 . The computer-implemented method of claim 1 ,
wherein the one or more characteristics of the event include one or more (i) characteristics indicative of a concealment mechanism, (ii) characteristics indicative of clothing worn, (iii) characteristics indicative of gender, (iv) characteristics indicative of hair color, (v) characteristics of clothing, or (vi) characteristics of personal affects.
10 . A computing system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the computing system to:
train a language model using a controlled ratio of value-only annotated video training data samples to structured data-value annotated video training data samples;
receive a time series of video frames depicting an environment;
generate, using a three-dimensional (3D) convolutional neural network, a plurality of spatiotemporal feature embeddings of the time series of video frames;
generate a plurality of multimodal embeddings at least in part by projecting the plurality of spatiotemporal feature embeddings into a shared multimodal latent space;
generate an output by using the language model to analyze the plurality of multimodal embeddings, wherein the output includes structured data indicating one or more characteristics of an event; and
store the output in a database.
11 . The computing system of claim 10 ,
wherein the output includes a JavaScript Object Notation (JSON) object, wherein the JSON object is indicative of one or more of (i) context of the event, (ii) activity detected during the event, or (iii) temporal information associated with the event.
12 . The computing system of claim 10 , wherein the instructions further cause the computing system to:
generate, based on user inputs entered via a video annotation application, annotated video training data; and
train the language model using the annotated video training data.
13 . The computing system of claim 10 , wherein the instructions further cause the computing system to:
fine-tune the language model based on retail-focused question-answer pairs.
14 . A computer-readable non-transitory storage medium comprising executable instructions that, when executed by a computer system, cause the computer system to:
train a language model using a controlled ratio of value-only annotated video training data samples to structured data-value annotated video training data samples;
receive a time series of video frames depicting an environment;
generate, using a three-dimensional (3D) convolutional neural network, a plurality of spatiotemporal feature embeddings of the time series of video frames;
generate a plurality of multimodal embeddings at least in part by projecting the plurality of spatiotemporal feature embeddings into a shared multimodal latent space;
generate an output by using the language model to analyze the plurality of multimodal embeddings, wherein the output includes structured data indicating one or more characteristics of an event; and
store the output in a database.
15 . The computer-readable non-transitory storage medium of claim 14 ,
wherein the output includes a JavaScript Object Notation (JSON) object, wherein the JSON object is indicative of one or more of (i) context of the event, (ii) activity detected during the event, or (iii) temporal information associated with the event.
16 . The computer-readable non-transitory storage medium of claim 14 , wherein the executable instructions further cause the computer system to:
Generate, based on user inputs entered via a video annotation application, annotated video training data; and
train the language model using the annotated video training data.
17 . The computer-readable non-transitory storage medium of claim 14 , wherein the executable instructions further cause the computer system to:
fine-tune the language model based on retail-focused question-answer pairs.
18 . The computer-implemented method of claim 1 , further comprising:
causing, by the one or more processors and based on the output, an external device to perform an action by transmitting a control signal to the external device.
19 . The computing system of claim 10 , wherein the instructions further cause the computing system to:
cause, based on the output, an external device to perform an action by transmitting a control signal to the external device.
20 . The computer-readable non-transitory storage medium of claim 14 , wherein the executable instructions further cause the computer system to:
cause, based on the output, an external device to perform an action by transmitting a control signal to the external device.