Image-based 3D panorama

1. A method for generating a 3-dimensional (3D) panorama, the method performed by a data processing system and comprising:

receiving, by the data processing system, images of a 3D scene;

reconstructing, by the data processing system, geometry of a plurality of 3D bubble-views from the images, wherein reconstructing includes:

using a structure from motion framework for camera localization;

generating a 3D surface mesh model of the scene using multi-view stereo via cylindrical surface sweeping for each bubble-view, wherein the cylindrical surface sweeping quantizes the scene with multiple depth surfaces with respect to a bubble view center and hypothesizes a depth of each light ray to be intersecting with one of the depth surfaces, wherein an intersecting point of each light ray is projected on each depth surface, and thereafter the cylindrical surface sweeping performs forward projection to find correspondences across multiple cameras; and

registering multiple 3D bubble-views in a common coordinate system, wherein registering multiple 3D bubble-views in a common coordinate system comprises registering partial images from different bubble-views to form a new coordinate system, and estimating a relative pose from each bubble-view to map images from each bubble-view to the new coordinate system; and

displaying the surface mesh models.

2. The method of claim 1 , wherein reconstructing further comprises: refining the 3D surface mesh model of each bubble-view using a depth optimization technique that utilizes a smoothness constraint to resolve issues caused by textureless regions of the scene.

3. The method of claim 1 , wherein the camera localization is used for registering a point cloud acquired from a synchronized depth sensor to form a fused point cloud in a global coordinate system.

4. The method of claim 3 , wherein the fused point cloud is utilized for refining the 3D surface mesh model with a depth optimization technique.

5. The method of claim 1 , wherein incremental bundle adjustment is utilized to register the multiple 3D bubble-views.

6. A data processing system for generating a 3-dimensional (3D) panorama comprising:

a processor; and

an accessible memory, the data processing system particularly configured to:

receive images of a 3D scene;

reconstruct geometry of a plurality of 3D bubble-views from the images,

wherein reconstructing includes:

using a structure from motion framework for camera localization;

generating a 3D surface mesh model of the scene using multi-view stereo via cylindrical surface sweeping for each bubble-view, wherein the cylindrical surface sweeping quantizes the scene with multiple depth surfaces with respect to a bubble view center and hypothesizes a depth of each light ray to be intersecting with one of the depth surfaces, wherein an intersecting point of each light ray is projected on each depth surface, and thereafter the cylindrical surface sweeping performs forward projection to find correspondences across multiple cameras; and

registering multiple 3D bubble-views in a common coordinate system, wherein registering multiple 3D bubble-views in a common coordinate system comprises registering partial images from different bubble-views to form a new coordinate system, and estimating a relative pose from each bubble-view to map images from each bubble-view to the new coordinate system; and

display the surface mesh models.

7. The data processing system of claim 6 , wherein the data processing system is further configured to:

refine the 3D surface mesh model of each bubble-view using a depth optimization technique that utilizes a smoothness constraint to resolve issues caused by textureless regions of the scene.

8. The data processing system of claim 6 , wherein the camera localization is used for registering a point cloud acquired from a synchronized depth sensor to form a fused point cloud in a global coordinate system.

9. The data processing system of claim 8 , wherein the fused point cloud is utilized for refining the 3D surface mesh model with a depth optimization technique.

10. The data processing system of claim 6 , wherein incremental bundle adjustment is utilized to register the multiple 3D bubble-views.

11. The data processing system of claim 6 , wherein the images of the 3D scene are acquired using a short-to-medium range image acquisition protocol.

12. The data processing system of claim 6 , wherein the images of the 3D scene are acquired using a very short range image acquisition protocol.

13. A non-transitory computer-readable medium encoded with executable instructions that, when executed, cause one or more data processing systems to:

receive images of a 3D scene;

reconstruct geometry of a plurality of 3D bubble-views from the images,

wherein reconstructing includes:

using a structure from motion framework for camera localization;

generating a 3D surface mesh model of the scene using multi-view stereo via cylindrical surface sweeping for each bubble-view, wherein the cylindrical surface sweeping quantizes the scene with multiple depth surfaces with respect to a bubble view center and hypothesizes a depth of each light ray to be intersecting with one of the depth surfaces, wherein an intersecting point of each light ray is projected on each depth surface, and thereafter the cylindrical surface sweeping performs forward projection to find correspondences across multiple cameras; and

registering multiple 3D bubble-views in a common coordinate system, wherein registering multiple 3D bubble-views in a common coordinate system comprises registering partial images from different bubble-views to form a new coordinate system, and estimating a relative pose from each bubble-view to map images from each bubble-view to the new coordinate system; and

display the surface mesh model.

14. The computer-readable medium of claim 13 , further encoded with computer-executable instructions that, when executed, cause the data processing system to:

refine the 3D surface mesh model of each bubble-view using a depth optimization technique that utilizes a smoothness constraint to resolve issues caused by textureless regions of the scene.

15. The computer-readable medium of claim 13 , wherein the camera localization is used for registering a point cloud acquired from a synchronized depth sensor to form a fused point cloud in a global coordinate system.

16. The computer-readable medium of claim 15 , wherein the fused point cloud is utilized for refining the 3D surface mesh model with a depth optimization technique.

17. The computer-readable medium of claim 13 , wherein incremental bundle adjustment is utilized to register the multiple 3D bubble-views.

18. A method for generating a 3-dimensional (3D) panorama, the method performed by a data processing system and comprising:

receiving, by the data processing system, images of a 3D scene;

reconstructing, by the data processing system, geometry of a plurality of 3D bubble-views from the images, wherein reconstructing includes:

using a structure from motion framework for camera localization;

generating a 3D surface mesh model of the scene using multi-view stereo via cylindrical surface sweeping for each bubble-view, wherein the cylindrical surface sweeping quantizes the scene with multiple depth surfaces with respect to a bubble view center and hypothesizes a depth of each light ray to be intersecting with one of the depth surfaces, wherein an intersecting point of each light ray is projected on each depth surface, and thereafter the cylindrical surface sweeping performs forward projection to find correspondences across multiple cameras; and

registering multiple 3D bubble-views in a common coordinate system, wherein registering multiple 3D bubble-views in a common coordinate system comprises jointly estimating a camera pose from different bubble-views while keeping the camera pose intact for one bubble-view that serves as the reference bubble-view; and

displaying the surface mesh models.

19. A data processing system for generating a 3-dimensional (3D) panorama comprising:

a processor; and

an accessible memory, the data processing system particularly configured to:

receive images of a 3D scene;

reconstruct geometry of a plurality of 3D bubble-views from the images, wherein reconstructing includes:

using a structure from motion framework for camera localization;

generating a 3D surface mesh model of the scene using multi-view stereo via cylindrical surface sweeping for each bubble-view, wherein the cylindrical surface sweeping quantizes the scene with multiple depth surfaces with respect to a bubble view center and hypothesizes a depth of each light ray to be intersecting with one of the depth surfaces, wherein an intersecting point of each light ray is projected on each depth surface, and thereafter the cylindrical surface sweeping performs forward projection to find correspondences across multiple cameras; and

registering multiple 3D bubble-views in a common coordinate system, wherein registering multiple 3D bubble-views in a common coordinate system comprises jointly estimating a camera pose from different bubble-views while keeping the camera pose intact for one bubble-view that serves as the reference bubble-view; and

display the surface mesh models.

20. A non-transitory computer-readable medium encoded with executable instructions that, when executed, cause one or more data processing systems to:

receive images of a 3D scene;

reconstruct geometry of a plurality of 3D bubble-views from the images, wherein reconstructing includes:

using a structure from motion framework for camera localization;

generating a 3D surface mesh model of the scene using multi-view stereo via cylindrical surface sweeping for each bubble-view, wherein the cylindrical surface sweeping quantizes the scene with multiple depth surfaces with respect to a bubble view center and hypothesizes a depth of each light ray to be intersecting with one of the depth surfaces, wherein an intersecting point of each light ray is projected on each depth surface, and thereafter the cylindrical surface sweeping performs forward projection to find correspondences across multiple cameras; and

registering multiple 3D bubble-views in a common coordinate system, wherein registering multiple 3D bubble-views in a common coordinate system comprises jointly estimating a camera pose from different bubble-views while keeping the camera pose intact for one bubble-view that serves as the reference bubble-view; and

display the surface mesh model.