IP Library Granted Patent US 11,790,606
Granted Patent B2
US 11,790,606 · App. 17/368,477 · Granted Oct 17, 2023

Determining camera rotations based on known translations

Inventor: Luc Robert (Valbonne, FR)
Assignee: Bentley Systems, Incorporated
G06T17/20G06T7/579G06T7/70G06T2210/56
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,790,606
App. No.
17/368,477
Granted
Oct 17, 2023
Kind
B2
Abstract

In example embodiments, techniques are provided for calculating camera rotation using translations between sensor-derived camera positions (e.g., from GPS) and pairwise information, producing a sensor-derived camera pose that may be integrated in an early stage of SfM reconstruction. A software process of a photogrammetry application may obtain metadata including sensor-derived camera positions for a plurality of cameras for a set of images and determine optical centers based thereupon. The software process may estimate unit vectors along epipoles from a given camera of the plurality of cameras to two or more other cameras. The software process then may determine a camera rotation that best maps unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles. The determined camera rotation and the sensor-derived camera position form a sensor-derived camera pose that may be returned and used.

Claims (52)

1. A method for calculating camera rotation using translations between sensor-derived camera positions, in a structure-from-motion (SfM) reconstruction of a scene from a set of images, comprising:

obtaining metadata including sensor-derived camera positions for a plurality of cameras for the set of images;

prior to performing pairwise calculations of the SfM reconstruction of the scene, determining camera rotation from the sensor-derived camera positions to form a sensor-derived camera pose by

determining optical centers of the plurality of cameras based on the sensor-derived camera positions,

estimating, by a software process of a photogrammetry application executing on one or more computing devices, unit vectors along epipoles from a given camera of the plurality of cameras to two or more other cameras, and

determining, by the software process, a camera rotation that best maps unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles, the determined camera rotation and the sensor-derived camera position forming the sensor-derived camera pose; and

using the previously determined sensor-derived camera pose of the given camera in the pairwise calculations of the SfM reconstruction of the scene.

2. The method of claim 1 , wherein the sensor-derived camera positions are global positioning system (GPS) positions.

3. The method of claim 1 , wherein the estimating unit vectors along epipoles comprises selecting cameras that observe a common portion of the scene and determining an Essential Matrix or a Fundamental Matrix based on the selected cameras.

4. The method of claim 1 , wherein the determining the camera rotation that best maps the unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles comprises calculating an approximate solution using a singular value decomposition.

5. The method of claim 1 , wherein the determining the camera rotation that best maps the unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles comprises calculating an approximate solution using unit quaternions.

6. The method of claim 1 , further comprising:

producing a sparse three-dimensional (3D) point cloud that represents the 3D structure of the scene;

intensifying the sparse 3D point cloud to create a dense 3D point cloud of the scene; and

reconstructing a 3D mesh of the scene from the dense 3D point cloud.

7. The method of claim 6 , further comprising:

at least one of displaying the 3D mesh on a display screen of the one or more computing devices, storing the 3D mesh to memory or storage of the one or more computing devices, or providing the 3D mesh to another software application executing on the one or more computing devices.

8. A computing device comprising:

a processor; and

a memory coupled to the processor and configured to store a photogrammetry application and a set of images of a scene that include metadata indicating sensor-derived camera positions, the photogrammetry application configured to reconstruct the scene from the set of images using, in part, a camera rotation calculation subprocess, the camera rotation calculation subprocess configured to:

determine optical centers of a plurality of cameras for the set of images based on the sensor-derived camera positions,

estimate unit vectors along epipoles from a given camera of the plurality of cameras to two or more other cameras,

determine a camera rotation from the sensor-derived camera positions to form a sensor-derived camera pose by determining a camera rotation that best maps unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles, the determined camera rotation and the sensor-derived camera position forming the sensor-derived camera pose, and

return the sensor-derived camera pose of the given camera; and

the photogrammetry application configured to

use the returned sensor-derived camera pose of the given camera in pairwise calculations to reconstruct the scene.

9. The computing device of claim 8 , wherein the sensor-derived camera positions are global positioning system (GPS) positions.

10. The computing device of claim 8 , wherein camera rotation calculation subprocess is further configured to estimate unit vectors along epipoles by selecting cameras that observe a common portion of the scene and determining an Essential Matrix or a Fundamental Matrix based on the selected cameras.

11. The computing device of claim 8 , wherein camera rotation calculation subprocess is further configured to determine the camera rotation that best maps the unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles by calculating an approximate solution using a singular value decomposition.

12. The computing device of claim 8 , wherein camera rotation calculation subprocess is configured to determine the camera rotation that best maps the unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles by calculating an approximate solution using unit quaternions.

13. The computing device of claim 8 , wherein the photogrammetry application is further configured to:

produce a sparse three-dimensional (3D) point cloud that represents the 3D structure of the scene;

intensify the sparse 3D point cloud to create a dense 3D point cloud of the scene; and

reconstruct a 3D mesh of the scene from the dense 3D point cloud.

14. The computing device of claim 13 , wherein the photogrammetry application when executed is further operable to:

display the 3D mesh on a display screen of the one or more computing devices, store the 3D mesh to memory or storage of the one or more computing devices, or provide the 3D mesh to another software application executing on the computing device.

15. A non-transitory computing device readable medium having instructions stored thereon, the instructions when executed by one or more computing devices operable to:

obtain metadata including sensor-derived camera positions for a plurality of cameras for a set of images of a scene;

prior to performing pairwise calculations of a structure-from-motion (SfM) reconstruction of a scene from a set of images,

determine optical centers of the plurality of cameras based on sensor-derived camera positions,

estimate unit vectors along epipoles from a given camera of the plurality of cameras to two or more other cameras, and

determine a camera rotation from the sensor-derived camera positions to form a sensor-derived camera post by determining a camera rotation that best maps unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles, the determined camera rotation and the sensor-derived camera position forming sensor-derived camera pose of the given camera; and

use the previously determined sensor-derived camera pose of the given camera in the pairwise calculations of the SfM reconstruction of the scene.

16. The non-transitory computing device readable medium of claim 15 , wherein the sensor-derived camera positions are global positioning system (GPS) positions.

17. The non-transitory computing device readable medium of claim 15 , wherein the instructions that when executed are operable to estimate unit vectors along epipoles comprise instructions that when executed are operable to select cameras that observe a common portion of the scene and determine an Essential Matrix or a Fundamental Matrix based on the selected cameras.

18. The non-transitory computing device readable medium of claim 15 , wherein the instructions that when executed are operable to determine the camera rotation that best maps the unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles comprise instructions that when executed are operable to calculate an approximate solution using a singular value decomposition.

19. The non-transitory computing device readable medium of claim 15 , wherein the instructions that when executed are operable to determine the camera rotation that best maps the unit vectors defined based on differences in the optical centers to the unit vectors along the epipoles comprise instructions that when executed are operable to calculate an approximate solution using unit quaternions.

20. The non-transitory computing device readable medium of claim 15 , wherein the instructions when executed by one or more computing devices are further operable to:

produce a sparse three-dimensional (3D) point cloud that represents the 3D structure of the scene;

intensify the sparse 3D point cloud to create a dense 3D point cloud of the scene;

reconstruct a 3D mesh of the scene from the dense 3D point cloud; and

at least one of display the 3D mesh on a display screen of the one or more computing devices, store the 3D mesh to memory or storage of the one or more computing devices, or provide the 3D mesh to another software application executing on the one or more computing devices.

Assignments (2)
SECURITY INTEREST Recorded Oct 25, 2024
From: BENTLEY SYSTEMS, INCORPORATED
To: PNC BANK, NATIONAL ASSOCIATION
Reel/Frame 069268/0042 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 6, 2021
From: ROBERT, LUC
To: BENTLEY SYSTEMS, INCORPORATED
Reel/Frame 056766/0268 →