System and method for language-guided video analytics at the edge
Training a classifier using embeddings and building a latent space is disclosed. The embeddings may be based on weights in a trained machine learning model. Also, operation of the classifier to process video segments in real-time using the using the weights and the latent space is disclosed. The embeddings and the latent space allow the classification to be performed at an overall reduced dimensionality. The latent space is designed to efficiently scale with an increasing number of queries to permit fast search through the space. Embodiments permit real-time operation on video with dynamic features. The classifier reduces the bandwidth demand of video camera-equipped devices at a network edge by setting aside, accurately, non-informative video sequences rather than uploading video too many things over the network. Applications include security cameras, robots and augmented reality glasses.
1. A method of reducing, at a network edge, transmission bandwidth of video information of a video stream, the method comprising:
extracting video features of a first portion of the video stream, the first portion of the video stream comprising a first plurality of video frames and corresponding to a first time period;
forming, based on the video features, a first vector in a latent space;
computing a first index of a first match of the first vector in the latent space, wherein a first similarity score is associated with the first index and the first match corresponds to a second vector in the latent space;
determining if the first similarity score exceeds a similarity threshold; and
when the similarity threshold is exceeded, transmitting first data related to the first index;
wherein the second vector is pre-determined by:
extracting text features of a natural language query, and
forming, based on the text features, the second vector in the latent space.
2. The method of claim 1 , wherein the first data comprises a plurality of video frames corresponding to the first time period and the first index.
3. The method of claim 2 , wherein the transmitting comprises transmitting the first data to a cloud server.
4. The method of claim 2 , wherein the transmitting comprises transmitting the first data to an application.
5. The method of claim 4 , wherein the application is installed in a robot, the first data comprises the first index and the robot is configured to perform a task based on the first index.
6. The method of claim 1 , further comprising:
counting each similarity score of the first index exceeding a threshold during a voting time period as one vote of a plurality of votes; and
only performing the transmitting when a count of the plurality of votes exceeds a vote threshold.
7. The method of claim 1 , wherein the transmitting the first data related to the first index comprises transmitting the video stream and a text string corresponding to the first index as a portion of the transmission bandwidth.
8. An apparatus comprising a processor and memory, wherein the processor is positioned at a network edge for reducing transmission bandwidth of video information of a video stream, the memory comprising instructions, wherein the processor is configured to:
extract video features of a first portion of the video stream, the first portion of the video stream comprising a first plurality of video frames and corresponding to a first time period;
form, based on the video features, a first vector in a latent space;
compute a first index of a first match of the first vector in the latent space, wherein a first similarity score is associated with the first index and the first match corresponds to a second vector in the latent space;
determine if the first similarity score exceeds a similarity threshold; and
when the similarity threshold is exceeded, transmit first data related to the first index;
wherein the processor is further configured to pre-determine the second vector by:
extracting text features of a natural language query, and
forming, based on the text features, the second vector in the latent space.
9. The apparatus of claim 8 , wherein the first data comprises a plurality of video frames corresponding to the first time period and the first index.
10. The apparatus of claim 9 , wherein the processor is further configured to transmit the first data to a cloud server.
11. The apparatus of claim 9 , wherein the processor is further configured to transmit the first data to an application.
12. The apparatus of claim 11 , wherein the application is installed in a robot, the first data comprises the first index and the robot is configured to perform a task based on the first index.
13. The apparatus of claim 8 , wherein the processor is further configured to:
count each similarity score of the first index exceeding a threshold during a voting time period as one vote of a plurality of votes; and
only transmit the first data when a count of the plurality of votes exceeds a vote threshold.
14. The apparatus of claim 8 , wherein the processor is further configured to transmit the video stream and a text string corresponding to the first index as a portion of the transmission bandwidth.
15. A non-transitory computer readable medium comprising instructions, the instructions configured to cause one or more processors to:
extract video features of a first portion of a video stream, the first portion of the video stream comprising a first plurality of video frames and corresponding to a first time period;
form, based on the video features, a first vector in a latent space;
compute a first index of a first match of the first vector in the latent space, wherein a first similarity score is associated with the first index and the first match corresponds to a second vector in the latent space;
determine if the first similarity score exceeds a similarity threshold; and
when the similarity threshold is exceeded, transmit first data related to the first index;
wherein the second vector is pre-determined by:
extracting text features of a natural language query, and
forming, based on the text features, the second vector in the latent space.
16. The non-transitory computer readable medium of claim 15 , wherein the first data comprises a plurality of video frames corresponding to the first time period and the first index.
17. The non-transitory computer readable medium of claim 16 , wherein the instructions are further configured to cause one or more processors to transmit the first data to a cloud server.
18. The non-transitory computer readable medium of claim 16 , wherein the instructions are further configured to cause one or more processors to transmit the first data to an application.
19. The non-transitory computer readable medium of claim 18 , wherein the application is installed in a robot, the first data comprises the first index and the robot is configured to perform a task based on the first index.
20. The non-transitory computer readable medium of claim 18 , wherein the instructions are further configured to cause one or more processors to:
count each similarity score of the first index exceeding a threshold during a voting time period as one vote of a plurality of votes; and
only transmit the first data when a count of the plurality of votes exceeds a vote threshold.