System and method for upsampling of sparse point cloud for 3D registration

1. A method for three-dimensional point cloud registration, the method comprising:

generating, with a processor, a three-dimensional point cloud of a scanned object using data received from a three-dimensional imaging sensor and storing the three-dimensional point cloud in a first memory location;

generating, with the processor, a two-dimensional image of the scanned object using data received from a two-dimensional imaging sensor and storing the two-dimensional image in a second memory location;

comparing the three-dimensional point cloud and the two-dimensional image, with the processor, and aligning the three-dimensional point cloud with the two-dimensional image according to at least three common points that are common to both the three-dimensional point cloud and the two-dimensional image;

generating, with the processor, a first upsampled three-dimensional point cloud by

identifying, with the processor, at least one missing point in the three-dimensional point cloud,

determining, with the processor, an intensity of neighboring pixels in the two-dimensional image neighboring the at least one missing point in the three-dimensional point cloud, and

filling, with the processor, the at least one missing point in the three-dimensional point cloud with a filler point using depth information from depth values in the three-dimensional point cloud that correspond with the neighboring pixels in the two-dimensional image where the neighboring pixels have intensities that correspond with an intensity of a point in the two-dimensional image associated with a respective missing point in the three-dimensional point cloud;

generating, with the processor, a second upsampled three-dimensional point cloud by

determining, with the processor, at least one local area of the first upsampled three-dimensional point cloud,

determining, with the processor, entropies of pixels in the two-dimensional image that correspond with the at least one local area of the first upsampled three-dimensional point cloud, and

adding, with the processor, at least one point to the at least one local area of the first upsampled three-dimensional point cloud based on the entropies of pixels in the two-dimensional image that correspond with the at least one local area and a scaled entropy threshold; and

registering, with the processor, the second upsampled three-dimensional point cloud with a predetermined three-dimensional model of the scanned object.

2. The method of claim 1 , wherein the at least one missing point is filled with the filler point when the intensities of the neighboring pixels and the intensity of the point in the two-dimensional image associated with the respective missing point in the three-dimensional point cloud match within a predetermined threshold.

3. The method of claim 1 , wherein filling the at least one missing point in the three-dimensional point cloud with the filler point further comprises assigning a weight for the neighboring pixels in the two-dimensional image.

4. The method of claim 3 , further comprising determining the weight for a respective neighboring pixel based on a distance between the at least one missing point and the respective neighboring pixel and a difference in intensity between the at least one missing point and the respective neighboring pixel.

5. The method of claim 1 , wherein generating the first upsampled three-dimensional point cloud further comprises:

assigning depth values to points in the two-dimensional image based on corresponding points in the three-dimensional point cloud; and

interpolating depth values for pixels of the two-dimensional image that do not have corresponding points in the three-dimensional point cloud.

6. The method of claim 5 , wherein interpolating the depth values for pixels of the two dimensional point cloud occurs in a predefined upsampling window around a point in the two-dimensional image having a known depth.

7. The method of claim 1 , wherein the at least one point added to the at least one local area of the first upsampled three-dimensional point cloud is located diagonally to pixels having a known depth.

8. The method of claim 1 , wherein the scaled entropy threshold includes:

a first threshold where upsampling does not occur if the entropies of pixels in the two-dimensional image that correspond with the at least one local area exceeds the first threshold;

a second threshold where a mean upsampling occurs if the entropies of pixels in the two-dimensional image that correspond with the at least one local area are less than the second threshold; and

a weighted upsampling occurs if the entropies of pixels in the two-dimensional image that correspond with the at least one local area is between the first and second thresholds.

9. An apparatus for three-dimensional point cloud registration, the apparatus comprising:

a three-dimensional imaging sensor;

a two-dimensional imaging sensor; and

a processor connected to both the three-dimensional sensor and the two-dimensional sensor, the processor being configured to:

generate a three-dimensional point cloud of a scanned object using data received from the three-dimensional imaging sensor and store the three-dimensional point cloud in a first memory location;

generate a two-dimensional image of the scanned object using data received from the two-dimensional imaging sensor and store the two-dimensional image in a second memory location;

compare the three-dimensional point cloud and the two-dimensional image and align the three-dimensional point cloud with the two-dimensional image according to at least one common point that is common to both the three-dimensional point cloud and the two-dimensional image;

generate a first upsampled three-dimensional point cloud by

identifying at least one missing point in the three-dimensional point cloud,

determining an intensity of neighboring pixels in the two-dimensional image neighboring the at least one missing point in the three-dimensional point cloud, and

filling the at least one missing point in the three-dimensional point cloud with a filler point using depth information from depth values in the three-dimensional point cloud that correspond with the neighboring pixels in the two-dimensional image where the neighboring pixels have intensities that correspond with an intensity of a point in the two-dimensional image associated with a respective missing point in the three-dimensional point cloud;

generate a second upsampled three-dimensional point cloud by

determining at least one local area of the first upsampled three-dimensional point cloud,

determining entropies of pixels in the two-dimensional image that correspond with the at least one local area of the first upsampled three-dimensional point cloud, and

adding at least one point to the at least one local area of the first upsampled three-dimensional point cloud based on the entropies of pixels in the two-dimensional image that correspond with the at least one local area and a scaled entropy threshold; and

registering the second upsampled three-dimensional point cloud with a predetermined three-dimensional model of the scanned object.

10. The apparatus of claim 9 , wherein the processor is configured to fill the at least one missing point with the filler point when the intensities of the neighboring pixels and the intensity of the point in the two-dimensional image associated with the respective missing point in the three-dimensional point cloud match within a predetermined threshold.

11. The apparatus of claim 9 , wherein filling the at least one missing point in the three-dimensional point cloud with the filler point further comprises assigning a weight for the neighboring pixels in the two-dimensional image.

12. The apparatus of claim 9 , wherein the processor configured to, when generating the first upsampled three-dimensional point cloud,

assign depth values to points in the two-dimensional image based on corresponding points in the three-dimensional point cloud; and

interpolate depth values for pixels of the two-dimensional image that do not have corresponding points in the three-dimensional point cloud.

13. The apparatus of claim 12 , wherein the processor is configured to interpolate the depth values for pixels of the two-dimensional image in a predefined upsampling window around a point in the two-dimensional image having a known depth.

14. The apparatus of claim 9 , wherein the scaled entropy threshold includes:

a first threshold where upsampling does not occur if the entropies of pixels in the two-dimensional image that correspond with the at least one local area exceeds the first threshold;

a second threshold where a mean upsampling occurs if the entropies of pixels in the two-dimensional image that correspond with the at least one local area are less than the second threshold; and

a weighted upsampling occurs if the entropies of pixels in the two-dimensional image that correspond with the at least one local area is between the first and second thresholds.

15. A non-transitory computer readable medium with instructions stored thereon, that when executed by a processor, are operable for performing a method for three-dimensional point cloud registration, the method comprising:

generating a three-dimensional point cloud of a scanned object using data received from a three-dimensional imaging sensor and storing the three-dimensional point cloud in a first memory location;

generating a two-dimensional image of the scanned object using data received from a two-dimensional imaging sensor and storing the two-dimensional image in a second memory location;

comparing the three-dimensional point cloud and the two-dimensional image and aligning the three-dimensional point cloud with the two-dimensional image according to at least one common point that is common to both the three-dimensional point cloud and the two-dimensional image;

generating a first upsampled three-dimensional point cloud by

identifying, with the processor, at least one missing point in the three-dimensional point cloud,

determining, with the processor, an intensity of neighboring pixels in the two-dimensional image neighboring the at least one missing point in the three-dimensional point cloud, and

filling, with the processor, the at least one missing point in the three-dimensional point cloud with a filler point using depth information from depth values in the three-dimensional point cloud that correspond with the neighboring pixels in the two-dimensional image where the neighboring pixels have intensities that correspond with an intensity of a point in the two-dimensional image associated with a respective missing point in the three-dimensional point cloud;

generating a second upsampled three-dimensional point cloud by

determining, with the processor, at least one local area of the first upsampled three-dimensional point cloud,

determining, with the processor, entropies of pixels in the two-dimensional image that correspond with the at least one local area of the first upsampled three-dimensional point cloud, and

adding, with the processor, at least one point to the at least one local area of the first upsampled three-dimensional point cloud based on the entropies of pixels in the two-dimensional image that correspond with the at least one local area and a scaled entropy threshold; and

registering the second upsampled three-dimensional point cloud with a predetermined three-dimensional model of the scanned object.

16. The non-transitory computer readable medium of claim 15 , wherein the at least one missing point is filled with the filler point when the intensities of the neighboring pixels and the intensity of the point in the two-dimensional image associated with the respective missing point in the three-dimensional point cloud match within a predetermined threshold.

17. The non-transitory computer readable medium of claim 15 , wherein filling the at least one missing point in the three-dimensional point cloud with the filler point further comprises assigning a weight for the neighboring pixels in the two-dimensional image.

18. The non-transitory computer readable medium of claim 15 , wherein generating the first upsampled three-dimensional point cloud further comprises:

assigning depth values to points in the two-dimensional image based on corresponding points in the three-dimensional point cloud; and

interpolating depth values for pixels of the two-dimensional image that do not have corresponding points in the three-dimensional point cloud.

19. The non-transitory computer readable medium of claim 18 , wherein interpolating the depth values for pixels of the two dimensional point cloud occurs in a predefined upsampling window around a point in the two-dimensional image having a known depth.

20. The non-transitory computer readable medium of claim 15 , wherein the scaled entropy threshold includes:

a first threshold where upsampling does not occur if the entropies of pixels in the two-dimensional image that correspond with the at least one local area exceeds the first threshold;

a second threshold where a mean upsampling occurs if the entropies of pixels in the two-dimensional image that correspond with the at least one local area are less than the second threshold; and

a weighted upsampling occurs if the entropies of pixels in the two-dimensional image that correspond with the at least one local area is between the first and second thresholds.