Three dimensional acquisition and rendering
View Patent ↗A method and system of using multiple image cameras or multiple image and depth cameras to capture a target object. Geometry and texture are reconstructed using captured images and depth images. New images are rendered using geometry based rendering methods or image based rendering methods.
1. A method of rendering new images, comprising:
acquiring a set of images using one or more image cameras;
acquiring a set of depth images using one or more depth cameras;
converting the set of depth images to geometric models representing scene objects;
converting the set of images acquired from the one or more image cameras into texture maps; and
rendering new images from the geometric models and texture maps using a geometry-based rendering technique;
wherein converting the depth images to the geometric models comprises:
detecting points in the set of depth images under viewpoint and lighting variations;
generating a descriptor for each point based on its local neighborhood, wherein the descriptor comprises surface normal and specular high light information based on non-lambertian surface assumption applied to the set of depth images;
detecting correspondences across the set of depth images by using the descriptors;
refining, based on the correspondences, camera intrinsic and extrinsic parameters for each image of the set of depth images using bundle-adjustment; and
converting the depth images to the geometric models by using a patch-based multi-view stereo for dense point cloud and a multi-view reconstruction,
wherein converting the set of images acquired from the one or more image cameras into texture maps comprises:
parameterizing the geometric model to divide it into a set of sub-models; and
re-projecting the set of images back to the geometric model to create a new texture map for each of the set of sub-models.
2. The method of claim 1 , wherein the geometry-based rendering technique is rasterization.
3. The method of claim 1 , wherein the geometry-based rendering technique is ray-tracing.
4. The method of claim 1 , wherein the scene object is static and one model and set of textures is converted for that scene object.
5. The method of claim 1 , wherein the scene object is animated.
6. The method of claim 5 , wherein every frame of the animated scene has a separate geometric model and texture map.
7. The method of claim 5 , wherein the animated scene has a single texture map and every frame of the animated scene has a separate geometric model.
8. A system for creating a rendering of images comprising:
one or more processors; and
a memory coupled with the one or more processors, the memory configured to store instructions that when executed by the one or more processors cause the one or more processors to:
acquire a set of images using one or more image cameras;
acquire a set of depth images using one or more depth cameras;
convert the set of depth images to geometric models representing scene objects;
convert the set of images acquired from the one or more image cameras into texture maps; and
render new images from the geometric models and texture maps using a geometry-based rendering technique;
wherein converting the depth images to the geometric models comprises:
detecting points in the set of depth images under viewpoint and lighting variations;
generating a descriptor for each point based on its local neighborhood, wherein the descriptor comprises surface normal and specular high light information based on non-lambertian surface assumption applied to the set of depth images;
detecting correspondences across the set of depth images by using the descriptors;
refining, based on the correspondences, camera intrinsic and extrinsic parameters for each image of the set of depth images using bundle-adjustment; and
converting the depth images to the geometric models by using a patch-based multi-view stereo for dense point cloud and a multi-view reconstruction,
wherein converting the set of images acquired from the one or more image cameras into texture maps comprises:
parameterizing the geometric model to divide it into a set of sub-models; and
re-projecting the set of images back to the geometric model to create a new texture map for each of the set of sub-models.
9. The system of claim 8 , wherein the geometry-based rendering technique is rasterization.
10. The system of claim 8 , wherein the geometry-based rendering technique is ray-tracing.
11. The system of claim 8 , wherein the scene object is static and one model and set of textures is converted for that scene object.
12. The system of claim 8 , wherein the scene object is animated.
13. The system of claim 12 , wherein the scene is animated and every frame of the animated scene has a separate geometric model and texture map.
14. The system of claim 12 , wherein the scene is animated and the animated scene has a single texture map and every frame of the animated scene has a separate geometric model.
15. A non-transitory computer-readable storage medium having stored thereon instructions for causing at least one computer system to create a rendering of images comprising, the instructions comprising:
acquiring a set of images using one or more image cameras;
acquiring a set of depth images using one or more depth cameras;
converting the set of depth images to geometric models representing scene objects;
converting the set of images acquired from the one or more image cameras into texture maps; and
rendering new images from the geometric models and texture maps using a geometry-based rendering technique;
wherein converting the depth images to the geometric models comprises:
detecting points in the set of depth images under viewpoint and lighting variations;
generating a descriptor for each point based on its local neighborhood, wherein the descriptor comprises surface normal and specular high light information based on non-lambertian surface assumption applied to the set of depth images;
detecting correspondences across the set of depth images by using the descriptors;
refining, based on the correspondences, camera intrinsic and extrinsic parameters for each image of the set of depth images using bundle-adjustment; and
converting the depth images to the geometric models by using a patch-based multi-view stereo for dense point cloud and a multi-view reconstruction,
wherein converting the set of images acquired from the one or more image cameras into texture maps comprises:
parameterizing the geometric model to divide it into a set of sub-models; and
re-projecting the set of images back to the geometric model to create a new texture map for each of the set of sub-models.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the geometry-based rendering technique is rasterization.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the geometry-based rendering technique is ray-tracing.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the scene object is static and one model and set of textures is converted for that scene object.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the scene object is animated.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the scene is animated and every frame of the animated scene has a separate geometric model and texture map.