Method and system for motion adaptive fusion of optical images and depth maps acquired by cameras and depth sensors
View Patent ↗A method and system for fusing sensed measurements includes a depth sensor to acquire depth measurements of a scene as a sequence of frames, and a camera configured to acquire intensity measurements of the scene as a sequence of images, wherein a resolution of the depth sensor is less than a resolution of the camera. A processor searches for similar patches in multiple temporally adjacent frames of the depth measurements, groups the similar patches into blocks using the intensity measurements, increases a resolution of the blocks using prior constraints to obtain increased resolution blocks, and then constructs a depth image with a resolution greater than the resolution of sensor from the increased resolution blocks.
1. A method for fusing sensed measurements, comprising steps:
acquiring depth measurements of a scene as a sequence of frames with a depth sensor;
acquiring intensity measurements of the scene as a sequence of images with a camera, wherein a resolution of the depth sensor is less than a resolution of the camera, and further comprising computerized steps;
searching for similar patches in multiple temporally adjacent frames of the depth measurements;
grouping the similar patches into three-dimensional (3D) blocks using the intensity measurements;
increasing a depth resolution of the three-dimensional (3D) blocks by performing a matrix completion of each 3D block using rank regularization to obtain increased resolution blocks, wherein the rank regularization is only enforced on the depth measurements of the patches;
constructing a depth image with a resolution greater than the resolution of the depth sensor from the increased resolution blocks; and
repeating the computerized steps until a termination condition is reached.
2. The method of claim 1 , wherein the depth sensor and the camera are calibrated.
3. The method of claim 1 , wherein the depth sensor is a light radar (LIDAR) sensor.
4. The method of claim 1 , wherein the camera is a video camera.
5. The method of claim 1 , wherein the rank regularization controls a similarity of the patches of images grouped in the same block.
6. The method of claim 5 , wherein the rank regularization penalizes a rank of the block.
7. The method of claim 5 , wherein the cost function is minimized using an augmented-Lagrangian method.
8. The method of claim 7 , the augmented-Lagrangian uses an alternating direction method of multipliers.
9. The method of claim 1 , wherein the depth image is motion adaptive by combining measurements from multiple views of the scene.
10. The method of claim 1 , wherein the searching and grouping uses a search area centered at a reference patch, and by considering overlapping patches in each frame.
11. The method of claim 1 , wherein the rank regularization uses a v-shrinkage operator.
12. The method of claim 1 , wherein the increasing the resolution minimizes a cost function that combines a quadratic data-fidelity term.
13. The method of claim 1 , wherein the rank regularization reduces the rank of the block.
14. The method of claim 1 , wherein the rank regularization reduces a nuclear norm of the block.
15. The method of claim 1 wherein the rank regularization applies a soft-thresholding function.
16. The method of claim 1 , wherein the rank regularization uses a nonconvex regularizer.
17. The method of claim 1 , wherein the rank regularization uses an iterative optimization.
18. The method of claim 1 , wherein the rank regularization enforces a data fidelity criterion.
19. The method of claim 18 , wherein an enforcement of the data fidelity criterion is decoupled from an enforcement of the rank regularization.
20. The method of claim 18 , wherein the rank regularization is nonuniform.
21. A system for fusing sensed measurements, comprising:
a depth sensor to acquire depth measurements of a scene as a sequence of frames;
a camera to acquire intensity measurements of the scene as a sequence of images, wherein a resolution of the depth sensor is less than a resolution of the camera; and
a processor configured to search for similar patches in multiple temporally adjacent frames of the depth measurements, to group the similar patches into three-dimensional (3D) blocks using the intensity measurements, to increase a depth resolution of the three-dimensional (3D) blocks by performing a matrix completion of each 3D block using rank regularization to obtain increased resolution blocks, wherein the rank regularization is only enforced on the depth measurements of the patches; to construct a depth image with a resolution greater than the resolution of the depth sensor from the increased resolution blocks.
22. A system for fusing sensed measurements, comprising:
a depth sensor to acquire depth measurements of a scene as a sequence of frames;
a camera to acquire intensity measurements of the scene as a sequence of images, wherein a resolution of the depth sensor is less than a resolution of the camera; and
a processor configured to search for similar patches in multiple temporally adjacent frames of the depth measurements, to group the similar patches into three-dimensional (3D) blocks using the intensity measurements, to increase a depth resolution of the three-dimensional (3D) blocks by performing a matrix completion of each 3D block using rank regularization to obtain increased resolution blocks, to construct a depth image with a resolution greater than the resolution of the depth sensor from the increased resolution blocks, until a termination condition is reached, wherein the rank regularization is only enforced on the depth measurements of the patches.
23. The method of claim 22 , wherein the rank regularization controls a similarity of the patches of images grouped in the same block and penalizes a rank of the block, and cost function is minimized using an augmented-Lagrangian method, such that the augmented-Lagrangian uses an alternating direction method of multipliers.
24. A method for fusing sensed measurements, comprising steps:
acquiring depth measurements of a scene as a sequence of frames with a depth sensor;
acquiring corresponding intensity measurements of the scene as a sequence of images with a camera, wherein a resolution of the depth sensor is less than a resolution of the camera;
searching for similar patches in multiple temporally adjacent frames of the intensity measurements;
grouping patches of the depth measurements corresponding to the similar patches of the intensity measurements into three-dimensional (3D) blocks of the depth measurements;
performing a matrix completion of each 3D block using rank regularization to obtain increased resolution blocks, wherein the rank regularization is only enforced on the depth measurements of the patches; and
constructing a depth image with a resolution greater than the resolution of the depth sensor from the increased resolution blocks.