IP Library Granted Patent US 11,557,041
Granted Patent B2
US 11,557,041 · App. 17/350,830 · Granted Jan 17, 2023

Dynamic artificial intelligence camera model update

Inventors: Santle Camilus Kulandai Samy (Santa Clara, CA); Youngchoon Park (Milwaukee, WI)
Assignee: Sensormatic Electronics, LLC
G06T7/13G06T7/149G06T7/20G06T2207/10016G06T2207/20081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,557,041
App. No.
17/350,830
Granted
Jan 17, 2023
Kind
B2
Abstract

A system may be configured to dynamically update deployed machine learning models. In some aspects, the system may receive sampled video information, generate first object detection information based on a cloud model and the sampled video information, and generate second object detection information based on a first edge model and the sampled video information. Further, the system may select, based on the first object detection information and the second object detection information, a plurality of training images from the sampled video information, detect motion information corresponding to motion of one or more detected objects within the plurality of training images, generate a plurality of annotated images based at least in part on the first object detection information and the motion information, and generate a second edge model based upon training the first edge model using the plurality of annotated images.

Claims (75)

1. A method comprising:

receiving sampled video information captured by a video capture device;

generating first object detection information based on a cloud model and the sampled video information, the cloud model configured to detect objects within the sampled video information;

generating second object detection information based on a first edge model and the sampled video information, the first edge model configured to detect objects within the sampled video information and employed at an edge device coupled with the video capture device;

selecting, based on comparing the first object detection information to the second object detection information, a plurality of training images from the sampled video information;

detecting motion information corresponding to motion of one or more detected objects within the plurality of training images;

generating a plurality of annotated images based at least in part on comparing the first object detection information to the motion information; and

generating a second edge model based upon training the first edge model using the plurality of annotated images.

2. The method of claim 1 , further comprising sending the second edge model to the edge device.

3. The method of claim 1 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, and selecting the plurality of training images from the sampled video information comprises:

selecting the first image frame for the plurality of training images based on a count of the first plurality of bounding representations not equaling a count of the second plurality of bounding representations.

4. The method of claim 1 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, and selecting the plurality of training images from the sampled video information comprises:

discarding the first image frame based on a count of the first plurality of bounding representations equaling a count of the second plurality of bounding representations.

5. The method of claim 1 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, and the motion information includes a second plurality of bounding representations detected in the first image frame, and generating the plurality of annotated images comprises:

generating, based on a count of the first plurality of bounding representations equaling a count of the second plurality of bounding representations, a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

6. The method of claim 1 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and generating the plurality of annotated images comprises:

determining that one or more bounding representations of the first plurality of bounding representations do not match any of the third plurality of bounding representations;

determining that each of the one or more bounding representations of the first plurality of bounding representations match a bounding representation within the second object detection information; and

generating a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

7. The method of claim 1 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and generating the plurality of annotated images comprises:

determining that one or more bounding representations of the second plurality of bounding representations do not match any of the first plurality of bounding representations;

determining that the one or more bounding representations do not match any of the third plurality of bounding representations; and

generating a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

8. The method of claim 1 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and further comprising:

determining a review context based at least in part on two of the first plurality of bounding representations, the second plurality of bounding representations, or the third plurality of bounding representations;

receiving annotation information via a graphical user interface; and

generating a second annotated image of the plurality of annotated images including the annotation information.

9. The method of claim 1 , wherein generating the second edge model based upon training the first edge model using the plurality of annotated images comprises:

determining the second edge model based on at least a layer of the first edge model based on a transfer learning operation.

10. A system comprising:

a memory storing instructions thereon; and

at least one processor coupled with the memory and configured by the instructions to:

receive sampled video information captured by a video capture device;

generate first object detection information based on a cloud model and the sampled video information, the cloud model configured to detect objects within the sampled video information;

generate second object detection information based on a first edge model and the sampled video information, the first edge model configured to detect objects within the sampled video information and employed at an edge device coupled with the video capture device;

select, based on comparing the first object detection information to the second object detection information, a plurality of training images from the sampled video information;

detect motion information corresponding to motion of one or more detected objects within the plurality of training images;

generate a plurality of annotated images based at least in part on comparing the first object detection information to the motion information; and

generate a second edge model based upon training the first edge model using the plurality of annotated images.

11. The system of claim 10 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, and to select the plurality of training images from the sampled video information, the at least one processor is further configured by the instructions to:

select the first image frame for the plurality of training images based on a count of the first plurality of bounding representations not equaling a count of the second plurality of bounding representations.

12. The system of claim 10 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the motion information includes a second plurality of bounding representations detected in the first image frame, and generating the plurality of annotated images comprises, and to generate the plurality of annotated images, the at least one processor is further configured by the instructions to:

generate, based on a count of the first plurality of bounding representations equaling a count of the second plurality of bounding representations, a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

13. The system of claim 10 , the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and to generate the plurality of annotated images, the at least one processor is further configured by the instructions to:

determine that one or more bounding representations of the first plurality of bounding representations do not match any of the third plurality of bounding representations;

determine that each of the one or more bounding representations of first plurality of bounding representations matches a bounding representation within the second object detection information; and

generate a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

14. The system of claim 10 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and to generate the plurality of annotated images, the at least one processor is further configured by the instructions to:

determine that one or more bounding representations of the second plurality of bounding representations do not match any of the first plurality of bounding representations;

determine that the one or more bounding representations do not match any of the third plurality of bounding representations; and

generate a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

15. The system of claim 10 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, the at least one processor is further configured by the instructions to:

determine a review context based at least in part on two of the first plurality of bounding representations, the second plurality of bounding representations, or the third plurality of bounding representations;

receive annotation information via a graphical user interface; and

generate a second annotated image of the plurality of annotated images including the annotation information.

16. A non-transitory computer-readable device having instructions thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:

receiving sampled video information captured by a video capture device;

generating first object detection information based on a cloud model and the sampled video information, the cloud model configured to detect objects within the sampled video information;

generating second object detection information based on a first edge model and the sampled video information, the first edge model configured to detect objects within the sampled video information and employed at an edge device coupled with the video capture device;

selecting, based on comparing the first object detection information to the second object detection information, a plurality of training images from the sampled video information;

detecting motion information corresponding to motion of one or more detected objects within the plurality of training images;

generating a plurality of annotated images based at least in part on comparing the first object detection information to the motion information; and

generating a second edge model based upon training the first edge model using the plurality of annotated images, the second edge model to be employed at the video capture device or another video capture device.

17. The non-transitory computer-readable device of claim 16 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, and selecting the plurality of training images from the sampled video information comprises:

selecting the first image frame for the plurality of training images based on a count of the first plurality of bounding representations not equaling a count of the second plurality of bounding representations.

18. The non-transitory computer-readable device of claim 16 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, and the motion information includes a second plurality of bounding representations detected in the first image frame, and generating the plurality of annotated images comprises:

generating, based on a count of the first plurality of bounding representations equaling a count of the second plurality of bounding representations, a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

19. The non-transitory computer-readable device of claim 16 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and generating the plurality of annotated images comprises:

determining that one or more bounding representations of the first plurality of bounding representations do not match any of the third plurality of bounding representations;

determining that each of the one or more bounding representations of the first plurality of bounding representations match a bounding representation within the second object detection information; and

generating a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

20. The non-transitory computer-readable device of claim 16 , wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and generating the plurality of annotated images comprises:

determining that one or more bounding representations of the second plurality of bounding representations do not match any of the first plurality of bounding representations;

determining that the one or more bounding representations do not match any of the third plurality of bounding representations; and

generating a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 23, 2024
From: JOHNSON CONTROLS TYCO IP HOLDINGS LLP
To: TYCO FIRE & SECURITY GMBH
Reel/Frame 068494/0384 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 6, 2022
From: KULANDAI SAMY, SANTLE CAMILUS; PARK, YOUNGCHOON
To: SENSORMATIC ELECTRONICS, LLC
Reel/Frame 059516/0659 →