Granted Patent B2

US 12,749,221 · App. 17/967,409 · Granted Sep 29, 2026

Inventors: Gang Qian (McLean, VA); Allison Beach (Leesburg, VA)

Assignee: ObjectVideo Labs, LLC

G06T7/80G06T3/60G06T7/60G06T7/74G06V10/751G06V10/761G06T2207/20164

Abstract

Claims (94)

1 . A computer-implemented method comprising:

maintaining, in a virtual space and simulating an image capture by a camera, a projection of an image depicting a calibration object onto a field of view of the camera;

generating, using the projection of the image onto the field of view of the camera in the virtual space, two or more simulated views of the calibration object in the virtual space;

detecting, for each of the two or more simulated views in the virtual space, a plurality of interest points for the calibration object, at least some of the interest points for different simulated views from the two or more simulated views comprising different data;

aggregating, for each common interest point included in at least two of the plurality of interest points for the two or more simulated views in the virtual space and for the calibration object, data for the common interest point for the calibration object using the different data from the respective interest points that represent the common interest point;

selecting, from the plurality of interest points and using the aggregated data for the common interest points, a subset of interest points;

determining an estimated center location of the calibration object in the image using the subset of interest points, comprising:

generating candidate center locations using descriptors for multiple interest point matched pairs from the subset of interest points;

generating one or more clusters of the candidate center locations using a clustering algorithm; and

identifying, from the one or more clusters, a centroid of a largest cluster as the estimated center location of the calibration object in the image;

determining physical locations of the subset of interest points of the calibration object in a calibration object centered coordinate system;

determining pixel locations of the subset of interest points in the image of the calibration object captured by the camera;

determining, using (i) the estimated center location of the calibration object, (ii) the pixel locations, and (iii) the physical locations, a transformation from the calibration object centered coordinate system to a camera centered coordinate system;

determining, using the transformation, a camera tilt angle and a camera mount height of the camera for use in analyzing images captured by the camera; and

transmitting, to a monitoring system, a target image and data indicating the camera tilt angle and the camera mount height to cause the monitoring system to analyze the target image captured by the camera using the camera tilt angle and the camera mount height.

2 . The method of claim 1 , comprising:

generating a mapping by matching one or more of the subset of interest points in the calibrated object centered coordinate system with a corresponding interest point in the image of the calibrated object captured by the camera, wherein determining the transformation uses the mapping.

3 . The method of claim 2 , comprising:

generating, for at least some of the interest points, one or more descriptors that indicate distinguishable features of a region around the corresponding interest point, wherein:

matching the one or more of the subset of interest points comprises matching the one or more of the subset of interest points in the calibrated object centered coordinate system with the corresponding interest point in the image of the calibrated object using a similarity of the respective associated one or more descriptors that indicate the distinguishable features of the region around the corresponding interest point.

4 . The method of claim 1 , wherein determining the physical locations of the subset of interest points in the calibration object centered coordinate system comprises:

obtaining physical dimension of the calibrated object;

obtaining pixel locations of the subset of interest points in a reference image, wherein the reference image is captured with a second camera that is centered above the calibration object, wherein the calibration object fills an entirety of the reference image; and

determining the physical locations of the subset of interest points using the physical dimension of the calibrated object and the pixel locations of the subset of interest points in the reference image.

5 . The method of claim 1 , wherein determining the transformation from the calibration object centered coordinate system to the camera centered coordinate system comprises:

determining an initial solution using a linear transformation from the calibration object centered coordinate system to the camera centered coordinate system;

determining, using the pixel locations and the physical locations, a rotation matrix that indicates a rotation from the calibration object centered coordinate system to the camera centered coordinate system; and

determining, using the pixel locations and the physical locations, a location vector that indicates a translation from the calibration object centered coordinate system to the camera centered coordinate system, wherein determining the rotation matrix and the location vector comprises optimizing the initial solution using a non-linear least square method.

6 . The method of claim 1 , comprising:

analyzing the target image captured by the camera using the camera tilt angle and the camera mount height.

7 . The method of claim 6 , wherein analyzing the target image comprises at least one of estimating a distance between the camera and an object depicted in the target image and localizing a footprint of the object.

8 . A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

maintaining, in a virtual space and simulating an image capture by a camera, a projection of an image depicting a calibration object onto a field of view of the camera;

generating, using the projection of the image onto the field of view of the camera in the virtual space, two or more simulated views of the calibration object in the virtual space;

detecting, for each of the two or more simulated views in the virtual space, a plurality of interest points for the calibration object, at least some of the interest points for different simulated views from the two or more simulated views comprising different data;

aggregating, for each common interest point included in at least two of the plurality of interest points for the two or more simulated views in the virtual space and for the calibration object, data for the common interest point for the calibration object using the different data from the respective interest points that represent the common interest point;

selecting, from the plurality of interest points and using the aggregated data for the common interest points, a subset of interest points;

determining an estimated center location of the calibration object in the image using the subset of interest points, comprising:

generating candidate center locations using descriptors for multiple interest point matched pairs from the subset of interest points;

generating one or more clusters of the candidate center locations using a clustering algorithm; and

identifying, from the one or more clusters, a centroid of a largest cluster as the estimated center location of the calibration object in the image;

determining physical locations of the subset of interest points of the calibration object in a calibration object centered coordinate system;

determining pixel locations of the subset of interest points in the image of the calibration object captured by the camera;

determining, using (i) the estimated center location of the calibration object, (ii) the pixel locations, and (iii) the physical locations, a transformation from the calibration object centered coordinate system to a camera centered coordinate system;

determining, using the transformation, a camera tilt angle and a camera mount height of the camera for use in analyzing images captured by the camera; and

transmitting, to a monitoring system, a target image and data indicating the camera tilt angle and the camera mount height to cause the monitoring system to analyze the target image captured by the camera using the camera tilt angle and the camera mount height.

9 . The system of claim 8 , the operations comprise:

generating, for at least some of the interest points, one or more descriptors that indicate distinguishable features of a region around a corresponding interest point; and

generating a mapping by matching one or more of the subset of interest points in the calibrated object centered coordinate system with the corresponding interest point in the image of the calibrated object captured by the camera using a similarity of the respective associated one or more descriptors that indicate the distinguishable features of the region around the corresponding interest point.

10 . The system of claim 8 , wherein determining the physical locations of the subset of interest points in the calibration object centered coordinate system comprises:

obtaining physical dimension of the calibrated object;

obtaining pixel locations of the subset of interest points in a reference image, wherein the reference image is captured with a second camera that is centered above the calibration object, wherein the calibration object fills an entirety of the reference image; and

determining the physical locations of the subset of interest points using the physical dimension of the calibrated object and the pixel locations of the subset of interest points in the reference image.

11 . The system of claim 10 , wherein the reference image is captured with the second camera that is at a tilt angle of ninety degrees from horizon.

12 . The system of claim 8 , the operations comprise:

obtaining intrinsic camera parameters; and

determining the transformation using the pixel locations, the physical locations, and the intrinsic camera parameters.

13 . The system of claim 8 , wherein determining the transformation from the calibration object centered coordinate system to the camera centered coordinate system comprises:

determining an initial solution using a linear transformation from the calibration object centered coordinate system to the camera centered coordinate system;

determining, using the pixel locations and the physical locations, a rotation matrix that indicates a rotation from the calibration object centered coordinate system to the camera centered coordinate system; and

determining, using the pixel locations and the physical locations, a location vector that indicates a translation from the calibration object centered coordinate system to the camera centered coordinate system, wherein determining the rotation matrix and the location vector comprises optimizing the initial solution using a non-linear least square method.

14 . A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

maintaining, in a virtual space and simulating an image capture by a camera, a projection of an image depicting a calibration object onto a field of view of the camera;

generating, using the projection of the image onto the field of view of the camera in the virtual space, two or more simulated views of the calibration object in the virtual space;

detecting, for each of the two or more simulated views in the virtual space, a plurality of interest points for the calibration object, at least some of the interest points for different simulated views from the two or more simulated views comprising different data;

aggregating, for each common interest point included in at least two of the plurality of interest points for the two or more simulated views in the virtual space and for the calibration object, data for the common interest point for the calibration object using the different data from the respective interest points that represent the common interest point;

selecting, from the plurality of interest points and using the aggregated data for the common interest points, a subset of interest points;

determining an estimated center location of the calibration object in the image using the subset of interest points, comprising:

generating candidate center locations using descriptors for multiple interest point matched pairs from the subset of interest points;

generating one or more clusters of the candidate center locations using a clustering algorithm; and

identifying, from the one or more clusters, a centroid of a largest cluster as the estimated center location of the calibration object in the image;

determining physical locations of the subset of interest points of the calibration object in a calibration object centered coordinate system;

determining pixel locations of the subset of interest points in the image of the calibration object captured by the camera;

determining, using (i) the estimated center location of the calibration object, (ii) the pixel locations, and (iii) the physical locations, a transformation from the calibration object centered coordinate system to a camera centered coordinate system;

determining, using the transformation, a camera tilt angle and a camera mount height of the camera for use in analyzing images captured by the camera; and

transmitting, to a monitoring system, a target image and data indicating the camera tilt angle and the camera mount height to cause the monitoring system to analyze the target image captured by the camera using the camera tilt angle and the camera mount height.

15 . The method of claim 3 , wherein generating the one or more descriptors comprises, for at least some of the interest points:

generating, for at least some of the two or more simulated views, one or more first descriptors each of which is for a corresponding interest point from the plurality of interest points for the corresponding simulated view; and

generating the one or more descriptors by aggregating, for each common interest point, descriptors from the one or more first descriptors that are for the common interest point.

16 . The method of claim 1 , comprising:

generating an object representation of the calibration object by projecting at least some of the simulated views of the calibration object onto the image, wherein:

determining the physical locations of the subset of interest points of the calibration object uses the object representation of the calibration object.

17 . The method of claim 1 , wherein:

generating the two or more simulated views of the calibration object comprises generating one or more homographic augmented views of the calibration object; and

detecting the plurality of interest points for the calibration object comprises detecting, for each of the one or more homographic augmented views of the calibration object, the plurality of interest points for the calibration object.

18 . The method of claim 1 , wherein:

generating the two or more simulated views of the calibration object comprises generating one or more photometric augmented views of the calibration object; and

detecting the plurality of interest points for the calibration object comprises detecting, for each of the one or more photometric augmented views of the calibration object, the plurality of interest points for the calibration object.

19 . The method of claim 1 , comprising:

computing, using the plurality of interest points for the two or more simulated views, a probability that an interest point is in separate ones of the plurality of interest points for a corresponding simulated view, wherein:

selecting the subset of interest points uses the probability that an interest point is in separate ones of the plurality of interest points for a corresponding simulated view.

20 . The method of claim 1 , comprising:

computing, using the plurality of interest points for the two or more simulated views, a frequency that an interest point is in separate ones of the plurality of interest points for a corresponding simulated view, wherein:

selecting the subset of interest points uses the frequency that an interest point is in separate ones of the plurality of interest points for a corresponding simulated view.

Assignments (1)

Continuity (2)

References Cited (19)