Granted Patent B2

US 12,277,741 · App. 17/496,078 · Granted Apr 15, 2025

Inventor: Kazuchika Iwami (Tokyo, JP)

Assignee: FUJIFILM Corporation

G06V10/22G06F18/21G06T7/30G06T17/00G06V20/00G06V20/64G06T2207/10028G06T2207/20081

Abstract

Claims (65)

1. A point cloud data processing apparatus comprising:

a processor that is configured to

acquire image data of an object,

acquire point cloud data having a corresponding positional relationship with the image data and representing pieces of three-dimensional information of a large number of points on an outer surface of the object,

recognize the object on the basis of the image data, and acquire a region of the object and attribute information for identifying the object,

select, from the point cloud data, point cloud data that belongs to the region of the object, and assign the acquired attribute information to the selected point cloud data,

recognize a missing part of the point cloud data that belongs to the region of the object on the basis of the region of the object and the attribute information,

calculate point cloud data of the missing part by linear interpolation using point cloud data in a neighborhood or by spline interpolation, and

determine whether the missing part exists or not in the point cloud data on the basis of luminance of the image data,

wherein the processor is further configured to

project three-dimensional coordinates of each point onto a specific plane to generate the image data having pixel values in accordance with reflection intensity or distance of each point of the point cloud data, and

assign the attribute information based on the generated image data,

wherein the processor is configured to

group the point cloud data for each partial region of the object by

recognizing, in a case where the image data includes image data of the object having a plurality of partial regions, a region of each partial region among the plurality of partial regions, and acquiring the region of the each partial region and attribute information for identifying the each partial region, and

selecting, from the point cloud data, point cloud data that belongs to the region of each partial region among the partial regions on a per partial region basis, and assigning the attribute information of the partial region to the point cloud data selected on a per partial region basis, and

wherein the processor is configured to identify each partial region of the object by image processing for extracting contours of the object to acquire the attribute information of each partial region.

2. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to

acquire, in a case where the image data includes pieces of image data of a plurality of objects, a region of each object among the objects and attribute information for identifying the object on a per object basis, and

select, from the point cloud data, point cloud data that belongs to the region of each object among the objects on a per object basis, and assign the attribute information of the object to the point cloud data selected on a per object basis.

3. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to

assign the attribute information to the point cloud data of the missing part.

4. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to acquire a captured image acquired by image capturing of the object as the image data.

5. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to

generate the image data on the basis of the point cloud data, and

acquire the generated image data.

6. The point cloud data processing apparatus according to claim 1 , wherein the processor is formed of a recognizer subjected to machine learning or deep learning.

7. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to generate a three-dimensional model of the object on the basis of the point cloud data that is assigned the attribute information.

8. The point cloud data processing apparatus according to claim 1 , wherein the image data and the point cloud data are respectively acquired by devices having a common optical axis.

9. The point cloud data processing apparatus according to claim 1 , wherein the image data and the point cloud data are respectively acquired by devices for which a positional relationship between the devices is known.

10. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to acquire data obtained by a laser scanner.

11. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to acquire the point cloud data by using a Time-of-Flight (ToF) camera or a stereo camera.

12. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to identify each partial region of the object by a recognizer constructed through machine learning or deep learning to acquire the attribute information of each partial region.

13. A point cloud data processing method comprising:

acquiring image data of an object;

acquiring point cloud data having a corresponding positional relationship with the image data and representing pieces of three-dimensional information of a large number of points on an outer surface of the object;

recognizing the object on the basis of the image data, and acquiring a region of the object and attribute information for identifying the object;

selecting, from the point cloud data, point cloud data that belongs to the region of the object, and assigning the acquired attribute information to the selected point cloud data;

recognizing a missing part of the point cloud data that belongs to the region of the object on the basis of the region of the object and the attribute information for identifying the object;

calculating point cloud data of the missing part by linear interpolation using point cloud data in a neighborhood or by spline interpolation; and

determining whether the missing part exists or not in the point cloud data on the basis of luminance of the image data,

wherein the method further comprises:

projecting three-dimensional coordinates of each point onto a specific plane to generate the image data having pixel values in accordance with reflection intensity or distance of each point of the point cloud data, and

assigning the attribute information based on the generated image data,

wherein the method further comprises:

grouping the point cloud data for each partial region of the object by

recognizing, in a case where the image data includes image data of the object having a plurality of partial regions, a region of each partial region among the plurality of partial regions, and acquiring the region of the each partial region and attribute information for identifying the each partial region, and

selecting, from the point cloud data, point cloud data that belongs to the region of each partial region among the partial regions on a per partial region basis, and assigning the attribute information of the partial region to the point cloud data selected on a per partial region basis, and

wherein the method further comprises identifying each partial region of the object by image processing for extracting contours of the object to acquire the attribute information of each partial region.

14. A non-transitory computer readable recording medium storing a program for causing a computer to perform a point cloud data process comprising:

acquiring image data of an object;

acquiring point cloud data having a corresponding positional relationship with the image data and representing pieces of three-dimensional information of a large number of points on an outer surface of the object;

recognizing the object on the basis of the image data, and acquiring a region of the object and attribute information for identifying the object;

selecting, from the point cloud data, point cloud data that belongs to the region of the object, and assigning the acquired attribute information to the selected point cloud data;

recognizing a missing part of the point cloud data that belongs to the region of the object on the basis of the region of the object and the attribute information for identifying the object;

calculating point cloud data of the missing part by linear interpolation using point cloud data in a neighborhood or by spline interpolation; and

determining whether the missing part exists or not in the point cloud data on the basis of luminance of the image data,

wherein the point cloud data process further comprises:

projecting three-dimensional coordinates of each point onto a specific plane to generate the image data having pixel values in accordance with reflection intensity or distance of each point of the point cloud data, and

assigning the attribute information based on the generated image data,

wherein the point cloud data process further comprises:

grouping the point cloud data for each partial region of the object by

recognizing, in a case where the image data includes image data of the object having a plurality of partial regions, a region of each partial region among the plurality of partial regions, and acquiring the region of the each partial region and attribute information for identifying the each partial region, and

selecting, from the point cloud data, point cloud data that belongs to the region of each partial region among the partial regions on a per partial region basis, and assigning the attribute information of the partial region to the point cloud data selected on a per partial region basis, and

wherein the point cloud data process further comprises identifying each partial region of the object by image processing for extracting contours of the object to acquire the attribute information of each partial region.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (31)