Granted Patent B2

US 12,633,100 · App. 18/286,312 · Granted May 19, 2026

Inventor: Heejin Lee (Incheon, KR)

Assignee: HYUNDAI DOOSAN INFRACORE CO., LTD.

G06V10/803G01S17/86G06T7/593G06V10/25G06V20/58E02F9/261G06V10/12

Abstract

Claims (60)

1 . A sensor fusion system, comprising:

a stereo camera having two lenses and configured to generate image data and a first point cloud data by capturing a three-dimensional image;

two lidar sensors configured to generate a second point cloud data and a third point cloud data, respectively; and

at least one processor configured to detect object information and a point cloud data with respect to surroundings based on the image data, the first point cloud data, the second point cloud data, and the third point cloud data obtained from the stereo camera and the two lidar sensors,

wherein the stereo camera generates the image data and the first point cloud data with respect to a first measurement region,

wherein among the two lidar sensors, a first lidar sensor generates the second point cloud data with respect to a second measurement region by sensing the second measurement region directed upward by a first predetermined angle based on a horizontal surface, and wherein a second lidar sensor generates the third point cloud data with respect to a third measurement region by sensing the third measurement region directed downward by a second predetermined angle based on the horizontal surface.

2 . The sensor fusion system of claim 1 , wherein the at least one processor is configured to:

convert the first point cloud data, the second point cloud data, and the third point cloud data, each of which having a different coordinate system, into data of one unified coordinate system,

obtain a fourth point cloud data with respect to a fourth measurement region formed by integrating the second measurement region and the third measurement region by integrating the converted second point cloud data and the third point cloud data,

obtain a fifth point cloud data with respect to a fifth measurement region including the first measurement region and the fourth measurement region by integrating the converted first point cloud data and the fourth point cloud data, and

obtain a sixth point cloud data having uniform spacing and density by performing gridding at the uniform spacing meet to a grid required by a construction machine with respect to the fifth measurement region to process the fifth point cloud data.

3 . The sensor fusion system of claim 2 , wherein the at least one processor is configured to obtain the fourth point cloud data by interpolating the second point cloud data and the third point cloud data in an overlapping region among the second measurement region and the third measurement region.

4 . The sensor fusion system of claim 3 , wherein the at least one processor is configured to:

set the fifth measurement region having a rectangular shape based on the first measurement region and the fourth measurement region,

obtain the fifth point cloud data with respect to an overlapping region of the fifth measurement region which overlaps the first measurement region and the fourth measurement region by interpolating the first point cloud data and the fourth point cloud data of the overlapping region,

obtain the fifth point cloud data with respect to a region of the fifth measurement region which is included only to the first measurement region by using the first point cloud data of the region,

obtain the fifth point cloud data with respect to a region of the fifth measurement region which is included only to the fourth measurement region by using the fourth point cloud data of the region, and

obtain the fifth point cloud data with respect to a region of the fifth measurement region which is not included to both the first measurement region and the fourth measurement region by extrapolating the first point cloud data and the fourth point cloud data of a periphery of the region.

5 . The sensor fusion system of claim 4 ,

wherein the extrapolation is performed by applying a machine learning method.

6 . The sensor fusion system of claim 1 ,

wherein the at least one processor is configured to:

preprocess the image data,

detect and classify an object from the preprocessed image data, and

obtain a location coordinate and distance information with respect to the detected object by using a sixth point cloud data which have been obtained based on the image data, the first point cloud data, the second point cloud data, and the third point cloud data which have been obtained from the stereo camera and the two lidar sensors.

7 . The sensor fusion system of claim 6 , wherein the at least one processor is configured to:

detect and classify an object from the image data which has been preprocessed by applying a machine learning method, and

provide image information of obstacles including a dump bed shape of a dump truck, a human being, and a vehicle as training data to train an artificial intelligence performing the machine learning.

8 . A sensing method for detecting an object and generating a point cloud data with respect to a measurement region of a sensor fusion system having a stereo camera outputting an image data and a first point cloud data, and two lidar sensors outputting a second point cloud data and a third point cloud data, comprising:

obtaining the image data, the first point cloud data, the second point cloud data, and the third point cloud data;

obtaining a sixth point cloud data with respect to the measurement region based on the first point cloud data, the second point cloud data, and the third point cloud data;

preprocessing the image data;

detecting and classifying an object from the preprocessed image data; and

obtaining a location coordinate and distance information of the detected object by using the sixth point cloud data,

wherein the image data and the first point cloud data are obtained with respect to a first measurement region,

wherein the second point cloud data is obtained by sensing, with a first lidar sensor, a second measurement region directed upward by a first predetermined angle based on a horizontal surface, and

wherein the third point cloud data is obtained by sensing, with a second lidar sensor, a third measurement region directed downward by a second predetermined measurement region based on the horizontal surface.

9 . The sensing method of claim 8 ,

wherein the obtaining a sixth point cloud data with respect to the measurement region comprises:

converting the first point cloud data, the second point cloud data, and the third point cloud data, each of which having a different coordinate system, into data of one unified coordinate system;

obtaining a fourth point cloud data with respect to a fourth measurement region formed by integrating the second measurement region and the third measurement region, by integrating the converted second point cloud data and the third point cloud data;

obtaining a fifth point cloud data with respect to a fifth measurement region including the first measurement region and the fourth measurement region by integrating the converted first point cloud data and the fourth point cloud data; and

obtaining a sixth point cloud data having uniform spacing and density by performing gridding at the uniform spacing meet to a grid required by a construction machine with respect to the fifth measurement region to process the fifth point cloud data.

10 . The sensing method of claim 9 ,

wherein the obtaining the fourth point cloud data comprises:

obtaining the fourth point cloud data by interpolating the second point cloud data and the third point cloud data in an overlapping region among the second measurement region and the third measurement region.

11 . The sensing method of claim 10 ,

wherein the obtaining the fifth point cloud data comprises:

setting the fifth measurement region having a rectangular shape based on the first measurement region and the fourth measurement region;

obtaining the fifth point cloud data with respect to an overlapping region of the fifth measurement region which overlaps the first measurement region and the fourth measurement region by interpolating the first point cloud data and the fourth point cloud data of the overlapping region;

obtaining the fifth point cloud data with respect to a region of the fifth measurement region which is included only to the first measurement region by using the first point cloud data of the region;

obtaining the fifth point cloud data with respect to a region of the fifth measurement region which is included only to the fourth measurement region by using the fourth point cloud data of the region; and

obtaining the fifth point cloud data with respect to a region of the fifth measurement region which is not included to both the first measurement region and the fourth measurement region by extrapolating the first point cloud data and the fourth point cloud data of a periphery of the region.

12 . The sensing method of claim 11 ,

wherein the extrapolation is performed by applying a machine learning method.

13 . The sensing method of claim 8 ,

wherein the detecting and classifying an object from the preprocessed image data comprises:

detecting and classifying an object from the image data which has been preprocessed by applying a machine learning method, and

providing image information of obstacles including a dump bed shape of a dump truck, a human being, and a vehicle as training data to train an artificial intelligence performing the machine learning.

14 . A non-transitory computer-readable storage medium having stored thereon a computer program to execute an operation according to a method of claim 8 , when executed by a processor.

Assignments (3)

Priority Claims (1)

Continuity (1)

References Cited (18)