IP Library › Granted Patent US 12,094,133
Granted Patent B2
US 12,094,133 · App. 17/585,060 · Granted Sep 17, 2024

Augmenting training samples for motion detection systems using image additive operations

Inventors: Narayanan Ramanathan (Chantilly, VA); Gang Qian (McLean, VA); Sima Taheri (McLean, VA); Sung Chun Lee (Fairfax, VA); Sravanti Bondugula (Vienna, VA); Allison Beach (Leesburg, VA)
Assignee: ObjectVideo Labs, LLC
G06T7/254G06T2207/20081G06T2207/20216G06T2207/20224
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,094,133
App. No.
17/585,060
Granted
Sep 17, 2024
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an event detector. The methods, systems, and apparatus include actions of identifying a portion of a first interframe difference image that represents motion of an OI, determining that a second interframe difference image represents motion by a non-OI, combining the portion of the first interframe difference image and the second interframe difference image as a third interframe difference image labeled as motion of both an OI and a non-OI, and training an event detector with the third interframe difference image.

Claims (69)

1. A computer-implemented method comprising:

accessing a first portion of a first interframe difference image a) that represents first motion by a first object that is an object of interest and b) was generated using a first sequence of two or more images captured by a first camera, each first image of which depicts the first object;

accessing a second interframe difference image i) that represents second motion by a second object that is not an object of interest and b) was generated using a second sequence of two or more images captured by a second camera, each second image of which depicts the second object, the second object being a different object than the first object;

determining whether a first bounding box for the first object in the first interframe difference image overlaps with a second bounding box for the second object in the second interframe difference image;

generating, using a result of the determination whether the first bounding box for the first object in the first interframe difference image overlaps with the second bounding box for the second object in the second interframe difference image, a third interframe difference image a) that represents both the first motion by the first object that is an object of interest and the second motion by the second object that is not an object of interest and b) by combining the first portion of the first interframe difference image and at least a second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image;

training an event detector to detect motion caused by objects of interest by providing, as input to the event detector, the third interframe difference image; and

storing, in memory and for a monitoring system of a property, data for the event detector for detecting objects of interest at the property.

2. The method of claim 1 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image comprises:

determining, for one or more first pixel locations in the first interframe difference image and one or more corresponding second pixel locations in the second interframe difference image, an average value of the two pixel locations; and

generating the third interframe difference image to include, at one or more third pixel locations each of which correspond to a first location in the one or more first pixel locations and a second location in the one or more second pixel locations, the corresponding average values of the two pixel locations in the first interframe difference image and the second interframe difference image.

3. The method of claim 2 , comprising:

determining that the first bounding box for the first object in the first interframe difference image does not overlap with the second bounding box for the second object in the second interframe difference image, wherein:

generating the third interframe difference image is responsive to determining that the first bounding box for the first object in the first interframe difference image does not overlap with the second bounding box for the second object in the second interframe difference image.

4. The method of claim 1 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image comprises:

determining, for one or more first pixel locations in the first interframe difference image and one or more corresponding second pixel locations in the second interframe difference image, a weighted average value of the two pixel locations; and

generating the third interframe difference image to include, at one or more third pixel locations each of which correspond to a first location in the one or more first pixel locations and a second location in the one or more second pixel locations, the corresponding weighted average values of the two pixel locations in the first interframe difference image and the second interframe difference image.

5. The method of claim 1 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image comprises:

generating the third interframe difference image to include a first set of pixel locations with values that are from the first portion of the first interframe difference image that represents motion of the first object that is an object of interest and a second set of pixel locations with values that are from at least the second portion of the second interframe difference image.

6. The method of claim 5 , comprising:

determining that a bounding box for the first object that is an object of interest and a bounding box for the second object that is not an object of interest overlap; and

in response to determining that the bounding box for the first object and the bounding box for the second object overlap, generating the third interframe difference image to include the first set of pixel locations with the values that are from the first portion of the first interframe difference image that represents motion of the first object that is an object of interest and the second set of pixel locations with values that are from at least the portion of the second interframe difference image.

7. The method of claim 1 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image comprises:

determining, for each pixel location within the first portion of the first interframe difference image that represents motion of the first object that is an object of interest, an average value of the pixel location in the first interframe difference image and one or more corresponding second pixel locations in the second interframe difference image; and

generating the third interframe difference image to include (i), at one or more third pixel locations each of which are within a bounding box and correspond to a first location in the one or more first pixel locations, the corresponding average values of the pixel locations in the first interframe difference image and the second interframe difference image, and (ii), at one or more pixel locations outside the bounding box, a value from the second interframe difference image.

8. The method of claim 1 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image comprises:

generating the third interframe difference image to include a first set of pixel locations with values that are from pixel locations within the first portion of the first interframe difference image that represents motion of the first object that is an object of interest at pixel locations translated from the pixel locations within the portion of the first interframe difference image and a second set of pixel locations with values that are from at least the second portion of the second interframe difference image.

9. The method of claim 8 , comprising:

translating the pixel locations within the first portion of the first interframe difference image that represents motion of the first object that is an object of interest to the first set of pixel locations; and

determining that the first set of pixel locations and a bounding box around the second object do not overlap.

10. The method of claim 1 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image comprises:

generating the third interframe difference image to include a first set of pixel locations with values that are from pixel locations within the first portion of the first interframe difference image that represents motion of the first object that is an object of interest scaled by a factor and a second set of pixel locations with values that are from the second interframe difference image.

11. The method of claim 1 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image comprises:

generating the third interframe difference image to include a first set of pixel locations with values, that are from pixel locations within the first portion of the first interframe difference image that represents motion of the first object that is an object of interest scaled by a factor, at pixel locations translated from the pixel locations within a bounding box and a second set of pixel locations with values that are from at least the second portion of the second interframe difference image.

12. The method of claim 1 , wherein accessing a first portion of a first interframe difference image a) that represents first motion by a first object that is an object of interest and b) was generated using a first sequence of two or more images captured by a first camera, each first image of which depicts the first object comprises:

obtaining a first image and a second image from the first sequence of two or more images captured by the first camera;

determining a bounding box around the first object that is an object of interest from the second image; and

generating the first interframe difference image using a difference between values of pixels in the first image and the second image, wherein the first portion of the first interframe difference image that represents motion of the first object that is an object of interest corresponds to a location of the bounding box around the first object that was determined.

13. A system comprising:

one or more computers and one or more non-transitory storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

accessing a first portion of a first interframe difference image a) that represents first motion by a first object that is an object of interest and b) was generated using a first sequence of two or more images captured by a first camera, each first image of which depicts the first object;

accessing a second interframe difference image i) that represents second motion by a second object that is not an object of interest and b) was generated using a second sequence of two or more images captured by a second camera, each second image of which depicts the second object, the second object being a different object than the first object;

determining whether a first bounding box for the first object in the first interframe difference image overlaps with a second bounding box for the second object in the second interframe difference image;

generating, using a result of the determination whether the first bounding box for the first object in the first interframe difference image overlaps with the second bounding box for the second object in the second interframe difference image, a third interframe difference image a) that represents both the first motion by the first object that is an object of interest and the second motion by the second object that is not an object of interest and b) by combining the first portion of the first interframe difference image and at least a second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image;

training an event detector to detect motion caused by objects of interest by providing, as input to the event detector, the third interframe difference image; and

storing, in memory and for a monitoring system of a property, data for the event detector for detecting objects of interest at the property.

14. The system of claim 13 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image comprises:

determining, for one or more first pixel locations in the first interframe difference image and one or more corresponding second pixel locations in the second interframe difference image, an average value of the two pixel locations; and

generating the third interframe difference image to include, at one or more third pixel locations each of which correspond to a first location in the one or more first pixel locations and a second location in the one or more second pixel locations, the corresponding average values of the two pixel locations in the first interframe difference image and the second interframe difference image.

15. The system of claim 14 , the operations comprising:

determining that the first bounding box for the first object in the first interframe difference image does not overlap with the second bounding box for the second object in the second interframe difference image, wherein:

generating the third interframe difference image is responsive to determining that the first bounding box for the first object in the first interframe difference image does not overlap with the second bounding box for the second object in the second interframe difference image.

16. The system of claim 13 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image:

determining, for one or more first pixel locations in the first interframe difference image and one or more corresponding second pixel locations in the second interframe difference image, a weighted average value of the two pixel locations; and

generating the third interframe difference image to include, at one or more third pixel locations each of which correspond to a first location in the one or more first pixel locations and a second location in the one or more second pixel locations, the corresponding weighted average values of the two pixel locations in the first interframe difference image and the second interframe difference image.

17. The system of claim 13 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image comprises:

generating the third interframe difference image to include a first set of pixel locations with values that are from the first portion of the first interframe difference image that represents motion of the first object that is an object of interest and a second set of pixel locations with values that are from at least the second portion of the second interframe difference image.

18. The system of claim 17 , the operations comprising:

determining that a bounding box for the first object that is an object of interest and a bounding box for the second object that is not an object of interest overlap; and

in response to determining that the bounding box for the first object that is an object of interest and the bounding box for the second object overlap, generating the third interframe difference image to include the first set of pixel locations with the values that are from the first portion of the first interframe difference image that represents motion of the first object that is an object of interest and the second set of pixel locations with values that are from at least the second portion of the second interframe difference image.

19. The system of claim 13 , wherein generating the third interface difference image by combining the first portion of the first interframe difference image and at least the second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image comprises:

determining, for each pixel location within the first portion of the first interframe difference image that represents motion of the first object that is an object of interest an average value of the pixel location in the first interframe difference image and one or more corresponding second pixel locations in the second interframe difference image; and

generating the third interframe difference image to include (i), at one or more third pixel locations each of which within a bounding box and correspond to a first location in the one or more first pixel locations, the corresponding average values of the pixel locations in the first interframe difference image and the second interframe difference image, and (ii), at one or more pixel locations outside the bounding box, a value from the second interframe difference image.

20. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

accessing a first portion of a first interframe difference image a) that represents first motion by a first object that is an object of interest and b) was generated using a first sequence of two or more images captured by a first camera, each first image of which depicts the first object;

accessing a second interframe difference image i) that represents second motion by a second object that is not an object of interest and b) was generated using a second sequence of two or more images captured by a second camera, each second image of which depicts the second object, the second object being a different object than the first object;

determining whether a first bounding box for the first object in the first interframe difference image overlaps with a second bounding box for the second object in the second interframe difference image;

generating, using a result of the determination whether the first bounding box for the first object in the first interframe difference image overlaps with the second bounding box for the second object in the second interframe difference image, a third interframe difference image a) that represents both the first motion by the first object that is an object of interest and the second motion by the second object that is not an object of interest and b) by combining the first portion of the first interframe difference image and at least a second portion of the second interframe difference image using values of pixel locations from the first interframe difference image and the second interframe difference image;

training an event detector to detect motion caused by objects of interest by providing, as input to the event detector, the third interframe difference image; and

storing, in memory and for a monitoring system of a property, data for the event detector for detecting objects of interest at the property.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 23, 2022
From: RAMANATHAN, NARAYANAN; QIAN, GANG; TAHERI, SIMA; LEE, SUNG CHUN; BONDUGULA, SRAVANTI; BEACH, ALLISON
To: OBJECTVIDEO LABS, LLC
Reel/Frame 059079/0758 →
Continuity (2)
Provisional Application 63147870 · Feb 10, 2021
Related Publication 20220254037A1 · Aug 11, 2022
Cited By (1)
US 12,586,314