IP Library Granted Patent US 11,594,043
Granted Patent B1
US 11,594,043 · App. 17/700,383 · Granted Feb 28, 2023

People and vehicle analytics on the edge

Inventors: Yi Xu (Belmont, CA); Mayank Gupta (Foster City, CA); Xia Yang (San Jose, CA); Yuanyuan Chen (San Mateo, CA); Zixiao (Shawn) Wang (San Mateo, CA); Qiang (Kevin) Fu (Sunnyvale, CA); Yunchao Gong (Los Altos, CA); Naresh Nagabushan (San Mateo, CA)
Assignee: Verkada Inc.
G06V20/58B60W40/08G06F16/284G06F16/287B60W2420/42
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,594,043
App. No.
17/700,383
Granted
Feb 28, 2023
Kind
B1
Abstract

A computer vision processor of a camera generates hyperzooms for persons or vehicles from image frames captured by the camera. The hyperzooms include a first hyperzoom associated with the persons or vehicles. The computer vision processor tracks traffic patterns of the persons or vehicles while obviating network usage by the camera by predicting positions of the persons or vehicles using a Kalman Filter from the first hyperzoom. The persons or vehicles are detected in the second hyperzoom. The positions of the persons or vehicles are updated based on detecting the persons or vehicles in the second hyperzoom. The first hyperzoom is removed from the camera. Tracks of the persons or vehicles are generated based on the updated positions. The second hyperzoom is removed from the camera. Track metadata is generated from the tracks for storing in a key-value database located on a non-transitory computer-readable storage medium of the camera.

Claims (74)

1. A method comprising:

generating, by a computer vision processor of a camera, hyperzooms for objects from image frames captured by the camera, the hyperzooms comprising a first hyperzoom and a second hyperzoom, the first hyperzoom associated with the objects;

tracking, by the computer vision processor, traffic patterns of the objects while obviating network usage by the camera by:

predicting positions of the objects using a Kalman Filter from the first hyperzoom;

updating the positions based on detecting the objects in the second hyperzoom, wherein the first hyperzoom is removed from the camera;

generating tracks of the objects based on the updated positions, wherein the second hyperzoom is removed from the camera; and

generating track metadata from the tracks for storing in a key-value database located on a memory card of the camera, wherein the tracks are removed from the camera;

receiving, by the camera, a query over a full-duplex communication channel, the query encoded as a key and a message, the message specifying a characteristic of the traffic patterns;

generating, by the computer vision processor, a response to the message by filtering the track metadata using the key and the characteristic of the traffic patterns;

transmitting, by the camera, the response over the full-duplex communication channel, the response for sending to a user device; and

displaying the response in a graphical representation on a screen of the user device.

2. The method of claim 1 , further comprising:

generating, by the computer vision processor, a visualization of the track; and

storing, by the camera, the tracks and the visualization in a NoSQL key-value structure on cloud storage, the visualization for sending to the user device.

3. The method of claim 1 , wherein the key-value database stores the tracks as byte arrays on the non-transitory computer-readable storage medium.

4. The method of claim 1 , wherein the key-value database stores the tracks in immutable row partitions in sorted order.

5. The method of claim 1 , wherein the key-value database stores the tracks in a relational database management system (RDBMS) on the non-transitory computer-readable storage medium.

6. The method of claim 1 , further comprising:

assigning, by the computer vision processor, identifiers to the objects; and

responsive to detecting the objects in the second hyperzoom:

propagating, by the computer vision processor, the identifiers in the second hyperzoom.

7. The method of claim 1 , further comprising:

generating, by the computer vision processor, a heatmap using the track metadata; and

transmitting, by the camera, the heatmap over the full-duplex communication channel, the heatmap for sending to the user device.

8. A camera comprising:

a non-transitory computer-readable storage medium; and

a computer vision processor communicably coupled to the non-transitory computer-readable storage medium and configured to:

track traffic patterns of objects while obviating network usage by the camera by performing steps to:

predict positions of the objects using a Kalman Filter from a first hyperzoom;

generate tracks of the objects based on updated positions from a second hyperzoom; and

generate track metadata from the tracks for storing in a database located on the non-transitory computer-readable storage medium, wherein the tracks are removed from the camera;

receive a query over a full-duplex communication channel, the query encoded as a key and a message, the message specifying a characteristic of the traffic patterns;

generate a response to the message by filtering the track metadata using the key and the characteristic of the traffic patterns;

transmit the response over the full-duplex communication channel, the response for sending to a user device; and

display the response in a graphical representation on a screen of the user device.

9. The camera of claim 8 , wherein the computer vision processor is further configured to:

generate a visualization of the track; and

store the tracks and the visualization in a NoSQL key-value structure on cloud storage, the visualization for sending to the user device.

10. The camera of claim 8 , wherein the computer vision processor is further configured to:

generate hyperzooms for the objects from image frames captured by the camera, the hyperzooms comprising the first hyperzoom and the second hyperzoom.

11. The camera of claim 8 , wherein the computer vision processor is further configured to:

detect the objects in the second hyperzoom; and

update the positions based on detecting the objects in the second hyperzoom, wherein the first and second hyperzooms are removed from the camera.

12. The camera of claim 8 , wherein the database stores the tracks in one of byte arrays on the non-transitory computer-readable storage medium, immutable row partitions in sorted order, or a relational database management system (RDBMS) on the non-transitory computer-readable storage medium.

13. The camera of claim 8 , wherein the computer vision processor is further configured to:

assign identifiers to the objects; and

responsive to detecting the persons or vehicles in the second hyperzoom:

persist the identifiers in the second hyperzoom.

14. The camera of claim 8 , wherein the computer vision processor is further configured to:

generate a heatmap using the track metadata; and

transmit the heatmap over the full-duplex communication channel, the heatmap for sending to the user device.

15. A system comprising:

one or more computer processors; and

a non-transitory computer-readable storage medium storing computer instructions, which when executed by the one or more computer processors cause the one or more computer processors to:

track traffic patterns of objects while obviating network usage by the camera by performing steps to:

predict positions of the objects using a Kalman Filter from a first hyperzoom;

generate tracks of the objects based on updated positions from a second hyperzoom; and

generate track metadata from the tracks for storing in a database located on the camera, wherein the tracks are removed from the camera;

receive a query over a full-duplex communication channel, the query encoded as a key and a message, the message specifying a characteristic of the traffic patterns;

generate a response to the message by filtering the track metadata using the key and the characteristic of the traffic patterns; and

transmit the response over the full-duplex communication channel, the response for sending to a user device; and

display the response in a graphical representation on a screen of the user device.

16. The system of claim 15 , wherein the computer instructions further cause the one or more computer processors to:

generate a visualization of the track; and

store the tracks and the visualization in a NoSQL key-value structure on cloud storage, the visualization for sending to the user device.

17. The system of claim 15 , wherein the computer instructions further cause the one or more computer processors to:

generate hyperzooms for the objects from image frames captured by the camera, the hyperzooms comprising the first hyperzoom and the second hyperzoom.

18. The system of claim 15 , wherein the computer instructions further cause the one or more computer processors to:

detect the objects in the second hyperzoom; and

update the positions based on detecting the persons or vehicles in the second hyperzoom, wherein the first and second hyperzooms are removed from the camera.

19. The system of claim 15 , wherein the database stores the tracks in one of byte arrays on the camera, immutable row partitions in sorted order, or a relational database management system (RDBMS) on the camera.

20. The system of claim 15 , wherein the computer instructions further cause the one or more computer processors to:

generate a heatmap using the track metadata; and

transmit the heatmap over the full-duplex communication channel, the heatmap for sending to the user device.

Assignments (2)
SECURITY INTEREST Recorded Jul 18, 2022
From: VERKADA INC.
To: SILICON VALLEY BANK, AS AGENT
Reel/Frame 060537/0838 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 27, 2022
From: XU, YI; GUPTA, MAYANK; YANG, XIA; CHEN, YUANYUAN; WANG, ZIXIAO (SHAWN); FU, QIANG (KEVIN); GONG, YUNCHAO; NAGABUSHAN, NARESH
To: VERKADA INC.
Reel/Frame 060044/0135 →
Cited By (4)
US 12,205,380 US 12,518,529 US 12,608,938 US 12,683,953