Cross-modal dense semantic label generation by diffusion and iterative inpainting

1 . A method comprising:

constructing, by a computing system, an image graph where nodes of the image graph represent pixels of a camera image;

performing a first diffusion of labels on the image graph using sparse labels from a point cloud sensor to generate propagated labels;

applying inpainting to one or more regions of the camera image to generate inpainted labels;

performing a second diffusion of labels on the image graph to update the propagated labels;

fusing the propagated labels and the inpainted labels to generate dense semantic labels for the camera image, wherein fusing the propagated labels and the inpainted labels comprises updating segmentation masks for the pixels using the propagated labels and the inpainted labels; and

repeating the inpainting and the second diffusion until a convergence of labels is achieved, wherein the convergence is determined based at least in part on a similarity between consecutive segmentation masks.

2 . The method of claim 1 , wherein constructing the image graph further comprises assigning proximity-based weights to the pixels of the image graph.

3 . The method of claim 1 , wherein constructing the image graph further comprises assigning visual similarity-based weights to the pixels of the image graph.

4 . The method of claim 1 , wherein performing the first diffusion and the second diffusion comprises propagating labels of objects in the camera image through edges of the image graph to generate the propagated labels.

5 . The method of claim 1 , wherein performing the first diffusion and the second diffusion comprises estimating confidence values for the propagated labels.

6 . The method of claim 5 , wherein applying inpainting to the one or more regions further comprises applying inpainting to the one or more regions using labels with confidence values over a predetermined threshold value.

7 . The method of claim 1 , wherein the one or more regions comprise at least one of missing labels and unreliable labels.

8 . The method of claim 1 , further comprising sending the dense semantic labels to an advanced driver assistance system (ADAS) for operation of a vehicle.

9 . The method of claim 8 , wherein the camera image and the sparse labels from a light detection and ranging (LIDAR) point cloud sensor represent a real-world scene in proximity to the vehicle.

10 . The method of claim 1 , further comprising performing object detection based at least in part on the dense semantic labels.

11 . An apparatus comprising:

at least one memory; and

processing circuitry in communication with the at least one memory, the processing circuitry configured to:

construct an image graph where nodes of the image graph represent pixels of a camera image;

perform a first diffusion of labels on the image graph using sparse labels from a point cloud sensor to generate propagated labels;

apply inpainting to one or more regions of the camera image to generate inpainted labels;

perform a second diffusion of labels on the image graph to update the propagated labels;

fuse the propagated labels and the inpainted labels to generate dense semantic labels for the camera image, wherein the processing circuitry configured to fuse the propagated labels and the inpainted labels is further configured to update segmentation masks for the pixels using the propagated labels and the inpainted labels; and

repeat the inpainting and the second diffusion until a convergence of labels is achieved, wherein the processing circuitry configured to determine convergence is further configured to determine convergence based at least in part on a similarity between consecutive segmentation masks.

12 . The apparatus of claim 11 , wherein instructions to construct the image graph further comprise instructions to assign proximity-based weights to the pixels of the image graph.

13 . The apparatus of claim 11 , wherein instructions to construct the image graph further comprise instructions assigning visual similarity-based weights to the pixels of the image graph.

14 . A non-transitory computer-readable medium storing instructions, that when executed by processing circuitry, cause the processing circuitry to:

construct an image graph where nodes of the image graph represent pixels of a camera image;

perform a first diffusion of labels on the image graph using sparse labels from a point cloud sensor to generate propagated labels;

apply inpainting to one or more regions of the camera image to generate inpainted labels;

perform a second diffusion of labels on the image graph to update the propagated labels;

fuse the propagated labels and the inpainted labels to generate dense semantic labels for the camera image, wherein the instructions to fuse the propagated labels and the inpainted labels further cause the processing circuitry to update segmentation masks for the pixels using the propagated labels and the inpainted labels; and

repeat the inpainting and the second diffusion until a convergence of labels is achieved, wherein the instructions to determine convergence further cause the processing circuitry to determine convergence based at least in part on a similarity between consecutive segmentation masks.