IP Library Granted Patent US 11,042,991
Granted Patent B2
US 11,042,991 · App. 16/149,691 · Granted Jun 22, 2021

Determining multiple camera positions from multiple videos

Inventors: Joan Lasenby (Cambridge, GB); Stuart Bennett (Cambridge, GB); Sasi Inguva (Mountain View, CA); Damien Kelly (San Francisco, CA); Andrew Crawford (San Francisco, CA); Hugh Denman (San Francisco, CA); Anil Kokaram (Sunnyvale, CA)
Assignee: Google LLC
G06T7/246G06T7/73H04N17/002
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,042,991
App. No.
16/149,691
Granted
Jun 22, 2021
Kind
B2
Abstract

The technology disclosed herein includes a method for determining the position of multiple cameras relative to each other. In one example, the method may include: receiving, by a processor, a first video recording of a first camera and a second video recording of a second camera; selecting a set of frames of the first video recording; determining a blurriness measure for multiple frames of the set; identifying feature points in multiple frames of the set; selecting a frame from the set of frames based on the blurriness measure and the identified feature points; and determining a position of a first camera relative to a second camera by comparing the selected frame with a frame of the second video recording.

Claims (73)

1. A method comprising:

receiving, by a processor, a video recording of a camera;

determining blurriness measures for a set of frames of the video recording;

identifying feature points in multiple frames of the set;

selecting a plurality of frames from the set based on the blurriness measures and the identified feature points, wherein the plurality of frames of the video recording comprise a first frame before a rotation of the camera and a second frame after the rotation of the camera, wherein the first frame and the second frame each include a particular image feature;

identifying an intersection of a normal corresponding to the particular image feature of the first frame and a normal corresponding to the particular image feature of the second frame, wherein the intersection corresponds to a location of the camera;

calculating a focal parameter of the camera based on the intersection; and

determining, based on the calculated focal parameter, a position of the camera relative to an object associated with the particular image feature.

2. The method of claim 1 , wherein determining the blurriness measures comprises determining how much of a high frequency component of a video signal in a particular frame of the video recording compares to high frequency components of video signals in neighboring frames of the video recording.

3. The method of claim 2 , further comprising selecting the neighboring frames according to a pre-set time difference between the particular frame and the neighboring frames.

4. The method of claim 3 , further comprising selecting at least one of the neighboring frames from frames having low blurriness measures, high feature point counts, and non-zero rotation with respect to the particular frame.

5. The method of claim 1 , wherein determining the position of the camera further comprises processing a frame of the video recording and a frame of a video recording of a second camera to estimate the location and an orientation of the camera relative to the second camera, wherein the processing, comprises:

feature mapping;

camera internal calibration; and

camera external calibration.

6. The method of claim 5 , wherein the feature mapping comprises:

identifying key feature points that do not vary in scale and orientation beyond a threshold amount in multiple frames of the set of frames;

determining orientations of a plurality of the identified key feature points;

comparing a frame of the set of frames of the video recording to multiple frames of a video recording of a second camera;

determining a matching frame by identifying a neighboring frame having one or more of the same key feature points;

selecting matched frames having similar scale and orientation;

sorting the selected matched frames into bins; and

selecting frames from bins having a highest count of sorted frames.

7. The method of claim 1 , further comprising:

identifying a frame pair from the video recording that comprises a two-dimensional displacement of one or more key feature points, wherein the frame pair comprises the first frame and the second frame;

determining the normals from image planes of the frame pair; and

calculating a focal length based on a distance between a point of the intersection and a point on one of the image planes.

8. The method of claim 7 , wherein identifying the frame pair comprises:

selecting the first frame from the plurality of frames in the video recording based on a number of key features that match with a frame in a video recording of a second camera; and

selecting the second frame from the plurality of frames based on one or more of the blurriness measures.

9. The method of claim 7 , wherein the external camera calibration comprises:

estimating real-world image coordinates; and

providing reconstructed estimates of world points.

10. A system comprising:

a processor; and

a computer-readable storage medium having instructions that when executed by the processor, cause the processor to:

determine blurriness measures for a set of frames of a video recording of a camera;

identify feature points in multiple frames of the set;

select a plurality of frames from the set based on the blurriness measures and the identified feature points, wherein the plurality of frames of the video recording comprise a first frame before a rotation of the camera and a second frame after the rotation of the camera, wherein the first frame and the second frame each include a particular image feature;

identify an intersection of a normal corresponding to the particular image feature of the first frame and a normal corresponding to the particular image feature of the second frame, wherein the intersection corresponds to a location of the camera;

calculate a focal parameter of the camera based on the intersection; and

determine, based on the calculated focal parameter, a position of the camera relative to an object associated with the particular image feature.

11. The system of claim 10 , wherein to determine the blurriness measures, the processor determines how a high frequency component of a video signal in a particular frame of the video recording compares to high frequency components of video signals in neighboring frames of the video recording.

12. The system of claim 11 , wherein the processor selects at least one of the neighboring frames according to a pre-set time difference between the particular frame and the at least one neighboring frame.

13. The system of claim 12 , wherein the processor selects a neighboring frame from the video recording that has a low blurriness measure, a high feature point count, and a non-zero rotation with respect to the particular frame.

14. The system of claim 10 , wherein the processor further executes feature mapping, and wherein the feature mapping causes the processor to:

identify key feature points that do not vary in scale and orientation beyond a threshold amount in multiple frames of the video recording;

determine orientations of a plurality of the identified key feature points;

compare a frame of the video recording to multiple frames of a second video recording of a second camera;

determine a matching frame by identifying a corresponding frame from the second video recording that has one or more of the key feature points;

select a plurality of matching frames that have similar scale and orientation;

sort the plurality of matching frames into bins; and

select final frames from bins having a highest count of sorted frames.

15. The system of claim 10 , wherein the processor is further to:

identify a frame pair from the video recording that comprises two-dimensional displacement of one or more key feature points, wherein the frame pair comprises the first frame and the second frame;

determine the normals from image planes of the frame pair; and

calculate a focal length based on a distance between a point of the intersection and a point on one of the image planes.

16. A non-transitory machine-readable storage medium comprising instructions that cause a processing device to:

receive a video recording of a camera;

identify feature points in multiple frames of the video recording;

determine blurriness measures for multiple frames of the video recording;

select a plurality of frames of the video recording based on the blurriness measures and the identified feature points, wherein the plurality of frames comprise a first frame before a rotation of the camera and a second frame after the rotation of the camera, wherein the first frame and the second frame each include a particular image feature;

identify an intersection of a normal corresponding to the particular image feature of the first frame and a normal corresponding to the particular image feature of the second frame, wherein the intersection corresponds to a location of the camera;

calculate a focal parameter of the camera based on the intersection; and

determine, using the calculated focal parameter, a position of the camera relative to an object associated with the particular image feature.

17. The non-transitory machine-readable storage medium of claim 16 , wherein the processing device is to select a neighboring frame according a pre-set time difference between a particular frame and the neighboring frame.

18. The non-transitory machine-readable storage medium of claim 17 , wherein the processing device selects the neighboring frame that has a low blurriness, a high feature point count, and a non-zero rotation with respect to the particular frame.

19. The non-transitory machine-readable storage medium of claim 16 , wherein the processing device is to:

select the first frame from the plurality of frames of the video recording based on a number of key features that match with a frame in a video recording of a second camera; and

select the second frame from the plurality of frames based on one or more of the blurriness measures.

20. The non-transitory machine-readable storage medium of claim 16 , wherein the processing device is to:

estimate real-world image coordinates; and

provide reconstructed estimates of world points.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 3, 2018
From: LASENBY, JOAN; BENNETT, STUART; INGUVA, SASI; KELLY, DAMIEN; CRAWFORD, ANDREW; DENMAN, HUGH; KOKARAM, ANIL
To: GOOGLE INC.
Reel/Frame 047190/0222 →
CHANGE OF NAME Recorded Oct 3, 2018
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 047190/0227 →
Continuity (2)
Continuation 14092413 · Nov 27, 2013
Related Publication 20190035090A1 · Jan 31, 2019