Scan color restoration
Techniques are described to generate a 3D scene by mapping a point cloud with a 2D image, and colorize portions of the 3D scene synthetically. An input is received to select, from the 3D scene, a portion to be colorized synthetically. The colorizing includes generating a reflectance image based on an intensity image of the point cloud. The colorizing further includes generating an occlusion mask that identifies the selected portion in the reflectance image. The colorizing further includes estimating, using a trained machine learning model, a color for each of the one or more points in the selected portion based on the reflectance image, the occlusion mask, and the 2D image. The 3D scene is updated by using the estimated colors from the trained machine learning model to colorize the selected portion.
1. A system comprising:
a three-dimensional (3D) scanner;
a camera with a viewpoint that is different from a viewpoint of the 3D scanner; and
at least one processor coupled with the 3D scanner and the camera, the at least one processor configured to:
access a point cloud captured by the 3D scanner, the point cloud comprises depth values of points in a surrounding environment;
access a 2D image captured by the camera, the 2D image comprises a plurality of pixels representing color information of the points in the surrounding environment;
generate a 3D scene by mapping the point cloud with the 2D image;
receive an input that selects, from the 3D scene, a portion to be colorized synthetically;
colorize the one or more points in the selected portion in the 3D scene, the colorizing comprising:
generating a reflectance image based on an intensity image of the point cloud;
generating an occlusion mask that identifies the selected portion in the reflectance image; and
estimate, using a trained machine learning model, a color for each of the one or more points in the selected portion based on the reflectance image, the occlusion mask, and the 2D image; and
update the 3D scene by using the estimated colors from the trained machine learning model to colorize the selected portion.
2. The system of claim 1 , wherein the trained machine learning model is a generative adversarial neural network.
3. The system of claim 1 , wherein the 2D image is a panoramic image.
4. The system of claim 1 , wherein the 3D scene comprises a plurality of occluded regions, and each occluded region is colorized using a sliding window.
5. The system of claim 1 , wherein colorizing the occluded portion of the 3D scene comprises transforming the 2D image into cube map representation.
6. The system of claim 1 , wherein the selected portion is an occluded portion of the surrounding environment that is visible from the 3D scanner, and is not visible from the camera.
7. The system of claim 1 , wherein the selected portion is an artifact that is to be removed from the 3D scene.
8. The system of claim 1 , wherein the trained machine learning model is trained using synthetically generated data from previously captured 3D images.
9. A computer-implemented method comprising:
accessing a point cloud captured by a 3D scanner, the point cloud comprises depth values of points in a surrounding environment;
accessing a 2D image captured by a camera, the 2D image comprises a plurality of pixels representing color information of the points in the surrounding environment;
generating a 3D scene by mapping the point cloud with the 2D image;
receiving an input that selects, from the 3D scene, a portion to be colorized synthetically;
colorizing the one or more points in the selected portion in the 3D scene, the colorizing comprising:
generating a reflectance image based on an intensity image of the point cloud;
generating an occlusion mask that identifies the selected portion in the reflectance image; and
estimating, using a trained machine learning model, a color for each of the one or more points in the selected portion based on the reflectance image, the occlusion mask, and the 2D image; and
updating the 3D scene by using the estimated colors from the trained machine learning model to colorize the selected portion.
10. The computer-implemented method of claim 9 , wherein the 2D image is a panoramic image.
11. The computer-implemented method of claim 9 , wherein the 3D scene comprises a plurality of occluded regions, and each occluded region is colorized using a sliding window.
12. The computer-implemented method of claim 9 , wherein colorizing the occluded portion of the 3D scene comprises transforming the 2D image into cube map representation.
13. The computer-implemented method of claim 9 , wherein the selected portion is an occluded portion of the surrounding environment that is visible from the 3D scanner, and is not visible from the camera.
14. The computer-implemented method of claim 9 , wherein the selected portion is an artifact that is to be removed from the 3D scene.
15. The computer-implemented method of claim 9 , wherein the trained machine learning model is trained using synthetically generated data from previously captured 3D images.