Method and apparatus for producing 3D model of an environment
The present invention provides a system (method and apparatus) for creating photorealistic 3D models of environments and/or objects from a plurality of stereo images obtained from a mobile stereo camera and optional monocular cameras. The cameras may be handheld, mounted on a mobile platform, manipulator or a positioning device. The system automatically detects and tracks features in image sequences and self-references the stereo camera in 6 degrees of freedom by matching the features to a database to track the camera motion, while building the database simultaneously. A motion estimate may be also provided from external sensors and fused with the motion computed from the images. Individual stereo pairs are processed to compute dense 3D data representing the scene and are transformed, using the estimated camera motion, into a common reference and fused together. The resulting 3D data is represented as point clouds, surfaces, or volumes. The present invention also provides a system (method and apparatus) for enhancing 3D models of environments or objects by registering information from additional sensors to improve model fidelity or to augment it with supplementary information by using a light pattern projector. The present invention also provides a system (method and apparatus) for generating photo-realistic 3D models of underground environments such as tunnels, mines, voids and caves, including automatic registration of the 3D models with pre-existing underground maps.
1. A method of producing a three-dimensional (3D) model of an environment, comprising the steps of:
a) acquiring a first and second plurality of successive stereo images of an environment from at least one of a plurality of stereo cameras which are moving with respect to the environment, the at least one stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;
b) for each image of said first plurality of successive stereo images, detecting features in the stereo images and computing a 3D position and descriptor for each detected feature, and storing said 3D position and said descriptor as a 3D feature in a database;
c) for each image of said first plurality of successive stereo images, computing relative motion of the at least one stereo camera with respect to the environment by matching the detected features in the stereo images with said 3D features stored in the database using descriptors of said 3D features;
d) for each image of said second plurality of successive stereo images, computing dense 3D data sets, representative of the environment from the at least one or another of said plurality of stereo cameras;
e) for each image of said second plurality of successive stereo images, transforming the computed dense 3D data from step d) into a selected coordinate frame of reference using the computed relative motion from step c) to give transformed dense 3D data in the selected coordinate frame of reference; and
f) storing the transformed dense 3D data for each image of said second plurality of successive stereo images and producing a 3D model of the environment from the stored transformed dense 3D data.
2. The method according to claim 1 wherein said at least one stereo includes first and second stereo cameras, wherein said first plurality of stereo images is acquired by said first stereo camera, wherein said second plurality of stereo images is acquired by said second stereo camera.
3. The method according to claim 2 wherein each successive image of said first plurality of stereo images at least partially overlaps with a preceding image in said first plurality of stereo images.
4. The method according to claim 3 wherein said first stereo camera is in a fixed spatial relationship to said second stereo camera.
5. The method according to claim 4 wherein said acquiring step comprises calculating a time stamp for each image of said first and second plurality of stereo images, and wherein said transforming step comprises comparing time stamps from said first plurality of stereo images to time stamps from said second plurality of stereo images.
6. The method according to claim 4 wherein said first and second stereo cameras acquire said stereo images in a synchronized manner such that each stereo image in said first plurality of stereo images has a corresponding stereo image in said second plurality of stereo images, and wherein said transforming step comprises transforming the computed dense 3D data from said corresponding stereo image from said second plurality of stereo images in step d) into a selected coordinate frame of reference using the computed relative motion from said stereo image from said first plurality of stereo images in step c) to give transformed dense 3D data in the selected coordinate frame of reference.
7. The method according to claim 1 wherein said second plurality of stereo images comprises fewer stereo images than said first plurality of stereo images.
8. The method according to claim 1 further comprising the step of projecting a pattern of dots onto said environment prior to acquiring each image of said second plurality of stereo images.
9. The method according to claim 1 further comprising the steps of acquiring additional data from an additional sensor, and augmenting said 3D model with said additional data.
10. The method according to claim 9 wherein said 3D model is one or combination of surface mesh, point cloud and 3D shapes.
11. The method according to claim 10 wherein combining said 3D model with said additional data comprises texturing or linking said 3D model with said additional data.
12. The method according to claim 11 wherein said additional data is selected from the group consisting of images obtained in the visible and outside visible parts of the light spectrum.
13. An apparatus for producing a three-dimensional (3D) model of an environment, comprising:
a) at least one stereo camera of a plurality of stereo cameras for acquiring a first and second plurality of successive stereo images of an environment which are moving with respect to the environment, the at least one stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;
b) processing means programmed with instructions for
i) detecting, for each image of said first plurality of successive stereo images, features in the stereo images and computing a 3D position and descriptor for each detected feature, and storing said 3D position and said descriptor as a 3D feature in a database;
ii) for each image of said first plurality of successive stereo images, computing relative motion of the at least one stereo camera with respect to the environment by matching the detected features in the stereo images with said 3D features stored in the database using descriptors of said 3D features;
iii) for each image of said second plurality of successive stereo images, computing dense 3D data sets, representative of the environment from the at least one or another of said plurality of stereo cameras;
iv) for each image of said second plurality of successive stereo images, transforming the computed dense 3D data from step b iii) into a selected coordinate frame of reference using the computed relative motion from step b ii) to give transformed dense 3D data in the selected coordinate frame of reference; and
v) storing the transformed dense 3D data for each image of said second plurality of successive stereo images and producing a 3D model of the environment from the stored transformed dense 3D data.
14. The apparatus according to claim 13 wherein said at least one stereo includes first and second stereo cameras, wherein said first plurality of stereo images is acquired by said first stereo camera, wherein said second plurality of stereo images is acquired by said second stereo camera.
15. The apparatus according to claim 14 wherein said first and second stereo cameras are configured such that each successive image of said first plurality of stereo images at least partially overlaps with a preceding image in said first plurality of stereo images.
16. The apparatus according to claim 15 wherein said first stereo camera is in a fixed spatial relationship to said second stereo camera.
17. The apparatus according to claim 16 wherein said processing means is programmed with instructions for calculating a time stamp for each image of said first and second plurality of stereo images, and wherein said processing means is programmed with instructions for comparing time stamps from said first plurality of stereo images to time stamps from said second plurality of stereo images.
18. The apparatus according to claim 16 wherein said first and second stereo cameras are configured to acquire said stereo images in a synchronized manner such that each stereo image in said first plurality of stereo images has a corresponding stereo image in said second plurality of stereo images, and wherein said processing means is programmed to transform the computed dense 3D data from said corresponding stereo image from said second plurality of stereo images in iv) into a selected coordinate frame of reference using the computed relative motion from said stereo image from said first plurality of stereo images in ii) to give transformed dense 3D data in the selected coordinate frame of reference.
19. The apparatus according to claim 13 further including a light pattern projector for the step of projecting a pattern of dots onto said environment prior to acquiring each image of said second plurality of stereo images.
20. The apparatus according to claim 13 further including an additional sensor for acquiring additional data from an additional sensor, and wherein the processing means is programmed with instructions to augment said 3D model with said additional data.
21. The apparatus according to claim 20 wherein said 3D model is one or combination of surface mesh, point cloud and 3D shapes.
22. The apparatus according to claim 21 wherein combining said 3D model with said additional data comprises texturing or linking said 3D model with said additional data.
23. The apparatus according to claim 22 wherein said additional data is selected from the group consisting of images obtained in the visible and outside visible parts of the light spectrum.
24. A method of producing a three-dimensional (3D) model of an underground environment, comprising the steps of:
a) acquiring 3D information of an underground environment from at least one range sensing device;
b) producing a 3D surface model of the underground environment from the acquired 3D information suitable for visualization, analysis or post-processing;
c) acquiring an image of said underground environment from a camera;
d) texturing the 3D surface model with said image;
e) locating the range sensing device by back-sighting to at least two existing survey stations located in the underground environment using a second range sensing device in a fixed relationship to said first range sensing device; and
f) based on the position information from step e, transforming the 3D model of the underground mine to a map of the underground environment in a coordinate frame of reference of the underground environment.
25. The method according to claim 24 further including the step of annotating the 3D surface model.
26. The method according to claim 25 wherein annotating the 3D surface model comprises annotating for ore grading and geological assessment.
27. The method according to claim 24 wherein said camera and said first range sensing device are combined in a single device.
28. The method according to claim 27 wherein said single device is a stereo camera.
29. An apparatus for producing a three-dimensional (3D) model of an underground environment, comprising:
a) at least a first range sensing device for acquiring 3D information of an underground environment; and
b) processing means programmed with instructions for
i) producing a 3D surface model of the underground environment from the acquired 3D information suitable for visualization, analysis or post-processing;
ii) acquiring an image of said underground environment from a camera;
iii) texturing the 3D surface model with said image;
iv) locating the range sensing device by back-sighting to at least two existing survey stations located in the underground environment using a second range sensing device in a fixed relationship to said first range sensing device; and
v) based on the position information from iv), transforming the 3D model of the underground mine to a map of the underground environment in a coordinate frame of reference of the underground environment.
30. The apparatus according to claim 29 wherein the processing means is programmed with instructions for annotating the 3D surface model.
31. The apparatus according to claim 30 wherein annotating the 3D surface model comprises annotating for ore grading and geological assessment.
32. The apparatus according to claim 29 wherein said camera and said first range sensing device are combined in a single device.
33. The apparatus according to claim 32 wherein said single device is a stereo camera.