Natural language processing for searching security video data

1 . A method, comprising:

obtaining surveillance video data using one or more image capture devices located at a user location;

generating frame embeddings for each frame of the surveillance video data as it is received, wherein each frame embedding is a vector representing a corresponding frame of the surveillance video data;

identifying an alarm condition based on the frame embeddings and one or more alarm embeddings, wherein the one or more alarm embeddings are text embeddings; and

generating an alert based on the alarm condition.

2 . The method of claim 1 , wherein the alarm condition is a user-defined alarm condition.

3 . The method of claim 2 , further comprising:

receiving a text description of the user-defined alarm condition; and

generating, using a machine learning model, a custom alarm embedding based on the text description, wherein the one or more alarm embeddings include the custom alarm embedding.

4 . The method of claim 3 , wherein identifying an alarm condition based on the frame embeddings and one or more alarm embeddings further comprises:

determining a similarity value between the custom alarm embedding and the frame embeddings; and

identifying the alarm condition based on the similarity value and a sensitivity value associated with the user-defined alarm condition.

5 . The method of claim 4 , wherein the sensitivity value is received with the text description of the user-defined alarm condition.

6 . The method of claim 4 , wherein the sensitivity value is predefined based on the text description of the user-defined alarm condition.

7 . The method of claim 1 , wherein generating an alert based on the alarm condition, further comprises:

activating one or more mitigation systems at the user location based on the alarm condition.

8 . The method of claim 1 , wherein identifying an alarm condition based on the frame embeddings and one or more alarm embeddings, further comprises:

identifying a new object in the surveillance video data;

generating an anomaly embedding for the new object;

identifying similar embeddings to the anomaly embedding are present in a vector database; and

triggering an anomaly detection.

9 . The method of claim 8 , wherein generating an alert based on the alarm condition, further comprises:

causing a user interface element to be displayed on a user interface, the user interface element prompting a user to label the anomaly; and

clearing the anomaly detection.

10 . A system, comprising:

a processor; and

a computer readable storage medium including instructions stored thereon which, when executed by the processor, cause the system to perform operations comprising:

obtaining surveillance video data using one or more image capture devices located at a user location;

generating frame embeddings for each frame of the surveillance video data as it is received, wherein each frame embedding is a vector representing a corresponding frame of the surveillance video data;

identifying an alarm condition based on the frame embeddings and one or more alarm embeddings, wherein the one or more alarm embeddings are text embeddings; and

generating an alert based on the alarm condition.

11 . The system of claim 10 , wherein the alarm condition is a user-defined alarm condition.

12 . The system of claim 11 , wherein the operations further comprise:

receiving a text description of the user-defined alarm condition; and

generating, using a machine learning model, a custom alarm embedding based on the text description, wherein the one or more alarm embeddings include the custom alarm embedding.

13 . The system of claim 12 , wherein the operation of identifying an alarm condition based on the frame embeddings and one or more alarm embeddings further comprises:

determining a similarity value between the custom alarm embedding and the frame embeddings; and

identifying the alarm condition based on the similarity value and a sensitivity value associated with the user-defined alarm condition.

14 . The system of claim 13 , wherein the sensitivity value is received with the text description of the user-defined alarm condition or is predefined based on the text description of the user-defined alarm condition.

15 . The system of claim 10 , wherein the operation of identifying an alarm condition based on the frame embeddings and one or more alarm embeddings, further comprises:

identifying a new object in the surveillance video data;

generating an anomaly embedding for the new object;

identifying similar embeddings to the anomaly embedding are present in a vector database; and

triggering an anomaly detection.

16 . The system of claim 15 , wherein the operation of generating an alert based on the alarm condition, further comprises:

causing a user interface element to be displayed on a user interface, the user interface element prompting a user to label the anomaly embedding; and

clearing the anomaly detection.

17 . A method, comprising:

receiving a request to search surveillance video data, the request including a text query;

obtaining a query embedding corresponding to the text query, wherein the query embedding is a vector representation of descriptors in the text query;

matching one or more frame embeddings corresponding to frames of the surveillance video data to the query embedding, wherein each of the one or more frame embeddings is a vector representing a corresponding frame of the surveillance video data; and

identifying at least one surveillance video corresponding to the one or more frame embeddings.

18 . The method of claim 17 , further comprising:

receiving the surveillance video data live from one or more image capture devices at a user location.

19 . The method of claim 17 , further comprising:

obtaining the surveillance video data from a surveillance video data archive.

20 . The method of claim 17 , wherein matching one or more frame embeddings corresponding to frames of the surveillance video data to the query embedding, further comprises:

determining a similarity value for the query embedding and each of the one or more frame embeddings, wherein a frame embedding includes one or more of an object embedding, a segment embedding, or a motion embedding, and wherein the similarity value is determined for each object embedding, segment embedding, and motion embedding; and

identifying the one or more matching frame embeddings based on the similarity values.