Depth completion method of sparse depth map and system thereof

1 . A depth completion method of sparse depth map, comprising:

acquiring a grayscale image and a sparse depth map corresponding to the grayscale image;

obtaining a nearest neighbor interpolation (NNI) image and a Euclidean distance transform (EDT) image based on the sparse depth map;

inputting the grayscale image, the NNI image, and the EDT image into a neural network model, thereby outputting a predicted residual map; and

generating a predicted dense depth map according to the predicted residual map and the NNI image;

wherein the neural network model extracts features of the grayscale image, the NNI image, and the EDT image by adopting an encoder-decoder fashion based on a UNet network architecture;

wherein the encoder-decoder fashion is composed by an EfficientNetV2-Lite model as an encoder part and Conv2D layers with nearest neighbor up-sampling as a decoder part;

wherein the EfficientNetV2-Lite model only contains Add, Pad, Conv2D, DepthwiseConv2D, ReLU6 operations, and the EfficientNetV2-Lite model is built with Conv2D and DepthwiseConv2D followed by ReLU6;

wherein the decoder part is implemented with only Conv2D followed by ReLU.

2 . The depth completion method of claim 1 , wherein the predicted dense depth map is generated by adopting a pixel-level addition method according to the predicted residual map and the NNI image, wherein the predicted residual map includes residual information of the NNI image.

3 . The depth completion method of claim 1 , wherein the grayscale image and the sparse depth map are acquired by using a time-of-flight (ToF) sensor.

4 . The depth completion method of claim 1 , further comprising:

performing a down-sampling process on the grayscale image, the NNI image, and the EDT image before the grayscale image, the NNI image, and the EDT image are inputted into the neural network model; and

performing an up-sampling process on the predicted dense depth map;

wherein the down-sampling process and the up-sampling process are performed by bilinear interpolation with antialiasing.

5 . The depth completion method of claim 1 , further comprising:

performing a model pruning operation on the neural network model to compress the neural network model.

6 . The depth completion method of claim 5 , wherein the model pruning operation is merely performed on plural target layers of the neural network model, wherein a number of weights of each of the target layers is larger than a threshold.

7 . The depth completion method of claim 5 , further comprising:

performing a model clustering operation on the neural network model to further compress the neural network model after the model pruning operation is performed.

8 . The depth completion method of claim 7 , wherein the model clustering operation is merely performed on plural target layers of the neural network model, wherein a number of weights of each of the target layers is larger than a threshold.

9 . The depth completion method of claim 1 , further comprising:

quantizing the neural network model from a floating-point number model to an integer model.

10 . A system for depth completion of sparse depth map, comprising:

a time-of-flight (ToF) sensor configured to acquire a grayscale image and a sparse depth map corresponding to the grayscale image; and

a processor configured to:

receive the grayscale image and the sparse depth map from the ToF sensor;

obtain a nearest neighbor interpolation (NNI) image and a Euclidean distance transform (EDT) image based on the sparse depth map;

input the grayscale image, the NNI image, and the EDT image into a neural network model, thereby outputting a predicted residual map; and

generate a predicted dense depth map according to the predicted residual map and the NNI image;

wherein the neural network model extracts features of the grayscale image, the NNI image, and the EDT image by adopting an encoder-decoder fashion based on a UNet network architecture;

wherein the encoder-decoder fashion is composed by an EfficientNetV2-Lite model as an encoder part and Conv2D layers with nearest neighbor up-sampling as a decoder part;

wherein the EfficientNetV2-Lite model only contains Add, Pad, Conv2D, DepthwiseConv2D, ReLU6 operations, and the EfficientNetV2-Lite model is built with Conv2D and DepthwiseConv2D followed by ReLU6;

wherein the decoder part is implemented with only Conv2D followed by ReLU.

11 . The system of claim 10 , wherein the processor generates the predicted dense depth map according to the predicted residual map and the NNI image by adopting a pixel-level addition method, wherein the predicted residual map includes residual information of the NNI image.

12 . The system of claim 11 , wherein the processor is further configured to:

perform a down-sampling process on the grayscale image, the NNI image, and the EDT image before the grayscale image, the NNI image, and the EDT image are inputted into the neural network model; and

perform an up-sampling process on the predicted dense depth map;

wherein the down-sampling process and the up-sampling process are performed by bilinear interpolation with antialiasing.

13 . The system of claim 10 , wherein the processor is further configured to:

perform a model pruning operation on the neural network model to compress the neural network model.

14 . The system of claim 13 , wherein the model pruning operation is merely performed on plural target layers of the neural network model, wherein a number of weights of each of the target layers is larger than a threshold.

15 . The system of claim 14 , wherein the processor is further configured to:

perform a model clustering operation on the neural network model to further compress the neural network model after the model pruning operation is performed.

16 . The system of claim 15 , wherein the model clustering operation is merely performed on plural target layers of the neural network model, wherein a number of weights of each of the target layers is larger than a threshold.

17 . The system of claim 10 , wherein the processor is further configured to:

quantize the neural network model from a floating-point number model to an integer model.